RFC2616 §3.3.1 requires compliant client to accept the two
obsolete formats in addition to the Internet standard format.

Based on patch by John Hallam <sw@j.hallam.dk>

Jiri Kastner authored 1 month ago and Timo Teräs committed 1 month ago

Signed-off-by: Jiri Kastner <cz172638@gmail.com>

return native error codes and map them in more detail in apk

fixes #10668

Lot of complexity for very little value.

As discussed in #10749, I suggested that we remove FTP support to reduce possible attack surface
as there are no Alpine mirrors which use FTP.

There are also no known mirrors of any other apk-using distribution which use FTP, so I believe
we are safe to remove support for FTP-based mirrors in apk-tools 3.

Signed-off-by: Ariadne Conill <ariadne@dereferenced.org>

upstream: https://cgit.freebsd.org/src/commit/lib/libfetch?id=ce700f78f7fb28a252978382a1d0a66d08b6469a

fixes #10776

ref #10794

Various parsing of numeric strings were not having adequate range
checking causing information leak or potential crash.

CVE-2021-36159
fixes #10749

Co-authored-by: Ariadne Conill <ariadne@dereferenced.org>
Reported-by: Samanta Navarro <ferivoz@riseup.net>

removes some code duplication

fixes #10738

Martin Vahlensieck authored 4 years ago and Timo Teräs committed 4 years ago

If server redirects from http to https, libfetch detects this, but
wrongly uses the old url scheme to determine the port. This subsequently
leads to the following OpenSSL error:

139741541575496:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:331:

Using the new scheme fixes this.  This error message comes from trying
to connect to port 80 with TLS, it can also be observed by issuing
  $ openssl s_client -connect alpinelinux.org:80

This bug was introduced in commit:
7158474f libfetch: keep http auth only if redirect is for the same host

The connection pooling was broken in two ways:

 1. The original URL was always used as the connection pool URL,
    resulting in duplicate connections to the proxy for http URLs
    (each http URL would get separate proxy connection)

 2. The cache_url stored was always the socket level connect URL.
    In case of HTTPS, the lookup was done done with the real URL,
    but the proxy URL was stored as the "cache URL". Thus HTTPS
    CONNECT connections were never re-used.

This fixes the code with following logic:

 1. The cache key url is the real URL when no-proxy, or when HTTPS
    with proxy (the socket is connected to proxy, but logically it
    is connected to the real URL due to HTTP CONNECT request).
    And for HTTP with proxy, it's the proxy URL so same proxy
    connection can be reused for all requests going through it.

 2. fetch_connect() now gets cache key URL separately, and it always
    gets the same value as the fetch_cache_get() calls.

Conny Seifert authored 4 years ago and Timo Teräs committed 4 years ago

Instead of skipping just one line, properly parse the response headers.

[TT: reworded commit message]

fixes #10688

- split the code to a helper function
- do not set sockets to corked state when putting back to
  cache so socket state is always deterministic
- cork/uncork also when sending CONNECT to a proxy, this
  can reduce a little bit the latency how fast the packet
  gets sent out
- also pair corking with uncorking in http_request to make
  it more obvious pairing

Alex Wauck authored 5 years ago and Timo Teräs committed 5 years ago

The recent TCP_CORK change missed this bit of code.  This change
should improve performance a bit when making HTTP requests by calling
http_cmd only once instead of three times.

Unfortunately libfetch operates on raw sockets and is sending
each HTTP request line using separate syscall which causes the
HTTP request to be sent as multiple packets over the wire in most
configurations. This is not good for performance, but can also
cause subtle breakage if there's DPI firewall that does not get
the Host header.

Incidentally, it seems that on BSDs libfetch already sets
TCP_NOPUSH optimize the packetization. This commit adds same
logic for using TCP_CORK if available. When using TCP_CORK
there is no requirement to set TCP_NODELAY as uncorking will
also cause immediate send. Keep TCP_NODELAY in the fallback
codepaths.

Long term, it might make sense to replace or rewrite libfetch
to use application level buffering.

error: 'strncpy' specified bound 4096 equals destination size [-Werror=stringop-truncation]

Based on patch by Elan Ruusamäe <glen@delfi.ee>

ref #8161

fixes #8160

ftp://ftp.fu-berlin.de/unix/NetBSD/packages/current-src/pkgsrc/net/libfetch/files

libfetch comes (at least) in netbsd and freebsd flavors which
differing functionality. Alpine and Arch package netbsd one,
but it's not widely packaged across other distributions.

We need NetBSD version as it does not use funopen(3) which is not
supported in musl, and supports connection pooling.

FreeBSD seems to be the orignal and better maintained version
with support for SSL CAs, client certificate authentication,
proxy authentication, and improved http redirect handling.

So this imports NetBSD version, and future commits will pick up
the needed improvements from FreeBSD tree.

Incidentally, this also fixes #7857 and likes for good.