as suggested in #10901

upstream: https://cgit.freebsd.org/src/commit/lib/libfetch?id=ce700f78f7fb28a252978382a1d0a66d08b6469a

Apply upstream fix for error messages from FreeBSD:
https://cgit.freebsd.org/src/commit/lib/libfetch?id=631b82aca0fd41c8e0d48eebdb9c4e38b7306472

fixes #10776

implement it also for connecting to hosts

fixes #10869

fixes #10650

This is not actually needed, and it results in both static and
shared versions of zlib and openssl linked into libapk otherwise,
which is bad.

Jan Hendrik Farr authored 3 years ago and Timo Teräs committed 3 years ago

Currently, special characters in the username or password are not
handled correctly (when set in $http_proxy and $https_proxy). They
should be percent encoded in the environment variables then decoded
by libfetch and reencoded using base64. This implementation is mainly
taken from the current FreeBSD source and adapted to the apk-tools
version of libfetch.

fixes #10775

Ariadne Conill authored 3 years ago and Timo Teräs committed 3 years ago

on mac, openssl is usually provided by Homebrew or some other third-party
package management system, which means pkg-config is needed to find it.

we already use pkg-config to find openssl when building apk itself.

ref #10794

ref #10794

Unbreak handling of base 16 in fetch_parseuint(). It is used
only in http chunked mode handling.

Fixes: "libfetch: fix range checking for http/ftp protocol parsing"

Various parsing of numeric strings were not having adequate range
checking causing information leak or potential crash.

CVE-2021-36159
fixes #10749

Co-authored-by: Ariadne Conill <ariadne@dereferenced.org>
Reported-by: Samanta Navarro <ferivoz@riseup.net>

removes some code duplication

fixes #10738

Martin Vahlensieck authored 4 years ago and Timo Teräs committed 4 years ago

If server redirects from http to https, libfetch detects this, but
wrongly uses the old url scheme to determine the port. This subsequently
leads to the following OpenSSL error:

139741541575496:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:331:

Using the new scheme fixes this.  This error message comes from trying
to connect to port 80 with TLS, it can also be observed by issuing
  $ openssl s_client -connect alpinelinux.org:80

This bug was introduced in commit:
7158474f libfetch: keep http auth only if redirect is for the same host

Rosen Penev authored 4 years ago and Timo Teräs committed 4 years ago

(De)initialization is deprecated under OpenSSL 1.0 and above.

[TT: Some simplifications, and additional edits.]

Signed-off-by: Rosen Penev <rosenp@gmail.com>

Treat URLs with too long individual components as malformed instead
of silently truncating that field. There might be unexpected results
if hostname, username or password field gets truncated.

The connection pooling was broken in two ways:

 1. The original URL was always used as the connection pool URL,
    resulting in duplicate connections to the proxy for http URLs
    (each http URL would get separate proxy connection)

 2. The cache_url stored was always the socket level connect URL.
    In case of HTTPS, the lookup was done done with the real URL,
    but the proxy URL was stored as the "cache URL". Thus HTTPS
    CONNECT connections were never re-used.

This fixes the code with following logic:

 1. The cache key url is the real URL when no-proxy, or when HTTPS
    with proxy (the socket is connected to proxy, but logically it
    is connected to the real URL due to HTTP CONNECT request).
    And for HTTP with proxy, it's the proxy URL so same proxy
    connection can be reused for all requests going through it.

 2. fetch_connect() now gets cache key URL separately, and it always
    gets the same value as the fetch_cache_get() calls.

fixes #10734

Conny Seifert authored 4 years ago and Timo Teräs committed 4 years ago

Instead of skipping just one line, properly parse the response headers.

[TT: reworded commit message]

This enables matching numeric hosts against CIDR style subnet
matches in no_proxy environment variable.

Based on work by Thermi.

Co-authored-by: Noel Kuntze <noel.kuntze@thermi.consulting>

Mike Detwiler authored 4 years ago and Timo Teräs committed 4 years ago

Signed-off-by: Mike Detwiler <det@shift5.io>

fixes #10688

Ariadne Conill authored 4 years ago and Timo Teräs committed 4 years ago

If getservbyname() fails, libfetch will attempt to connect to port 0.

This creates main help like:
--
usage: apk [<OPTIONS>...] COMMAND [<ARGUMENTS>...]

Package installation and removal:
  add        Add packages to WORLD and commit changes
  del        Remove packages from WORLD and commit changes

System maintenance:
  fix        Check WORLD against the system and ensure consistency
  update     Update repository indexes
  upgrade    Install upgrades available from repositories
  cache      Commands related to the management of an offline package cache

Querying package information:
  info       Give detailed information about packages or repositories
  list       List packages matching a pattern or other criteria
  dot        Generate graphviz graphs
  policy     Show repository policy for packages

Repository maintenance:
  index      Create repository index file from packages
  fetch      Download packages from global repositories to a local directory
  manifest   Show checksums of package contents
  verify     Verify package integrity and signature

Miscellaneous:
  audit      Audit directories for changes
  stats      Show statistics about repositories and installations
  version    Compare package versions or perform tests on version strings

This apk has coffee making abilities.
--

And applet specific help like:
--
usage: apk add [<OPTIONS>...] PACKAGES...

Description:
  apk add adds the requested packages to WORLD and installs (or upgrades)
  them if not already present, ensuring all dependencies are met.

Options:
  --initdb         Initialize a new package database
  -l, --latest     Disables normal heuristics for choosing which repository to install a
  -u, --upgrade    When adding packages which are already installed, upgrade them rather
  -t, --virtual NAME
                   Instead of adding the specified packages to WORLD, create a new
  --no-chown       Do not change file owner or group
--

Fix comparing of the hostname portion that matches exactly.
The no_proxy matching is pretty rudimentary though and probably
could go through a bit of additional rework.

Fixes #10681

- split the code to a helper function
- do not set sockets to corked state when putting back to
  cache so socket state is always deterministic
- cork/uncork also when sending CONNECT to a proxy, this
  can reduce a little bit the latency how fast the packet
  gets sent out
- also pair corking with uncorking in http_request to make
  it more obvious pairing

Alex Wauck authored 4 years ago and Timo Teräs committed 4 years ago

The recent TCP_CORK change missed this bit of code.  This change
should improve performance a bit when making HTTP requests by calling
http_cmd only once instead of three times.

Unfortunately libfetch operates on raw sockets and is sending
each HTTP request line using separate syscall which causes the
HTTP request to be sent as multiple packets over the wire in most
configurations. This is not good for performance, but can also
cause subtle breakage if there's DPI firewall that does not get
the Host header.

Incidentally, it seems that on BSDs libfetch already sets
TCP_NOPUSH optimize the packetization. This commit adds same
logic for using TCP_CORK if available. When using TCP_CORK
there is no requirement to set TCP_NODELAY as uncorking will
also cause immediate send. Keep TCP_NODELAY in the fallback
codepaths.

Long term, it might make sense to replace or rewrite libfetch
to use application level buffering.

error: 'strncpy' specified bound 4096 equals destination size [-Werror=stringop-truncation]

Based on patch by Elan Ruusamäe <glen@delfi.ee>

This is unwanted information disclosure.

Reported-by: Max Justicz <max@justi.cz>

TLS_client_method is a LibreSSL extension.
SSLv23_client_method is generic, and doesn't mean SSL v2/v3 only.

ref #8161

fixes #8160

OpenSSL allows passing zero-length to indicate "use strlen".
LibreSSL requires using the real length always, so pass the length.

X509_check_host() is introduced in libressl-2.5.0 and openssl-1.0.2
which are not yet universally available. Add support for building
against the older versions.

loosely based on the freebsd implementation, implement https
connection settings to override CA, and use client certificate.

new files supported in /etc/apk/:
  ca.pem   - if exists, it contains CAs acceptable for https
	     (otherwise system wide CAs are used)
  crl.pem  - if ca.pem is used, this is the (optional) CRL for it
  cert.pem - used as client authentication certificate (+ key)
  cert.key - used as client key (can be also inside cert.pem)