return native error codes and map them in more detail in apk

fixes #10668

As discussed in #10749, I suggested that we remove FTP support to reduce possible attack surface
as there are no Alpine mirrors which use FTP.

There are also no known mirrors of any other apk-using distribution which use FTP, so I believe
we are safe to remove support for FTP-based mirrors in apk-tools 3.

Signed-off-by: Ariadne Conill <ariadne@dereferenced.org>

Apply upstream fix for error messages from FreeBSD:
https://cgit.freebsd.org/src/commit/lib/libfetch?id=631b82aca0fd41c8e0d48eebdb9c4e38b7306472

implement it also for connecting to hosts

fixes #10869

fixes #10650

Unbreak handling of base 16 in fetch_parseuint(). It is used
only in http chunked mode handling.

Fixes: "libfetch: fix range checking for http/ftp protocol parsing"

Various parsing of numeric strings were not having adequate range
checking causing information leak or potential crash.

CVE-2021-36159
fixes #10749

Co-authored-by: Ariadne Conill <ariadne@dereferenced.org>
Reported-by: Samanta Navarro <ferivoz@riseup.net>

Rosen Penev authored 4 years ago and Timo Teräs committed 4 years ago

(De)initialization is deprecated under OpenSSL 1.0 and above.

[TT: Some simplifications, and additional edits.]

Signed-off-by: Rosen Penev <rosenp@gmail.com>

The connection pooling was broken in two ways:

 1. The original URL was always used as the connection pool URL,
    resulting in duplicate connections to the proxy for http URLs
    (each http URL would get separate proxy connection)

 2. The cache_url stored was always the socket level connect URL.
    In case of HTTPS, the lookup was done done with the real URL,
    but the proxy URL was stored as the "cache URL". Thus HTTPS
    CONNECT connections were never re-used.

This fixes the code with following logic:

 1. The cache key url is the real URL when no-proxy, or when HTTPS
    with proxy (the socket is connected to proxy, but logically it
    is connected to the real URL due to HTTP CONNECT request).
    And for HTTP with proxy, it's the proxy URL so same proxy
    connection can be reused for all requests going through it.

 2. fetch_connect() now gets cache key URL separately, and it always
    gets the same value as the fetch_cache_get() calls.

fixes #10734

This enables matching numeric hosts against CIDR style subnet
matches in no_proxy environment variable.

Based on work by Thermi.

Co-authored-by: Noel Kuntze <noel.kuntze@thermi.consulting>

Ariadne Conill authored 4 years ago and Timo Teräs committed 4 years ago

If getservbyname() fails, libfetch will attempt to connect to port 0.

Fix comparing of the hostname portion that matches exactly.
The no_proxy matching is pretty rudimentary though and probably
could go through a bit of additional rework.

Fixes #10681

TLS_client_method is a LibreSSL extension.
SSLv23_client_method is generic, and doesn't mean SSL v2/v3 only.

OpenSSL allows passing zero-length to indicate "use strlen".
LibreSSL requires using the real length always, so pass the length.

loosely based on the freebsd implementation, implement https
connection settings to override CA, and use client certificate.

new files supported in /etc/apk/:
  ca.pem   - if exists, it contains CAs acceptable for https
	     (otherwise system wide CAs are used)
  crl.pem  - if ca.pem is used, this is the (optional) CRL for it
  cert.pem - used as client authentication certificate (+ key)
  cert.key - used as client key (can be also inside cert.pem)

from freebsd

ftp://ftp.fu-berlin.de/unix/NetBSD/packages/current-src/pkgsrc/net/libfetch/files

libfetch comes (at least) in netbsd and freebsd flavors which
differing functionality. Alpine and Arch package netbsd one,
but it's not widely packaged across other distributions.

We need NetBSD version as it does not use funopen(3) which is not
supported in musl, and supports connection pooling.

FreeBSD seems to be the orignal and better maintained version
with support for SSL CAs, client certificate authentication,
proxy authentication, and improved http redirect handling.

So this imports NetBSD version, and future commits will pick up
the needed improvements from FreeBSD tree.

Incidentally, this also fixes #7857 and likes for good.