Performance degradation in Python 3.8 compared to the one from docker-library (python3.8:alpine3.12)

Migrating from docker-library the new python3 package bundled in Alpine3.12, an automated performance test shows a clear degradation.

The test consists in serializing complex JSON messages with orjson and sending them over plain TCP. The receiver side is outside of the container and always the same.

Performance degrades from ~200,000 messages per second with python3.7:alpine3.10 (or python3.8:alpine3.12) to ~180,000 with alpine:3.12 + apk add python3.

Profiling with py-spy all flame charts seem very similar. The hot spots in the code are:

1. ~37% on TCP socket.sendall() (non-asyncio) => ~38% on the faster version
2. ~30% on JSON serialization (with orjson) => 32% on the faster version

In both cases I used the same orjson 3.3.1 binary from manylinux wheel, as well as building different versions from scratch with same Rust version and commands.

Since the profile shapes are so similar in both cases, I am wondering if there is a general performance drop in the interpreter itself...

added lang:python type:bug labels

changed title from Boot sequence performance degradation in Python 3.8 compared to the one from docker-library (python3.8:alpine3.12) to Boot sequence% performance degradation in Python 3.8 compared to the one from docker-library (python3.8:alpine3.12)

Funny: writing ~10 expands to Boot sequence label.

@gatopeich reword to remove that combination of the tilde. "Around 10%" perhaps?

changed title from ~10% performance degradation in Python 3.8 compared to the one from docker-library (python3.8:alpine3.12) to Performance degradation in Python 3.8 compared to the one from docker-library (python3.8:alpine3.12)

Python3 in Alpine is built with "-Os" to optimise for size and the APKBUILD file has a comment referring to this:

-Os overwrites --enable-optimizations

whereas the docker-library Python doesn't specify this.

So I guess its a case of size versus performance tradeoff.

See also this thread on the mailing list:

https://lists.alpinelinux.org/~alpine/devel/%3C1593625212.dirkptm3b0.none%40localhost%3E#%3Cdcb0ce15-c5dc-3b38-39d8-a0b907e96c7a@postmarketos.org%3E

See !6945 (merged) and #11129 (closed)

@gatopeich can you confirm that !6945 (merged) enhanced the performance? Maybe something in the build infrastructure screwed this up again.

Will try Asap

I could not find ready-made packages to test this.

@J0WI or @dbradley could you provide binaries or quick instructions?

3462e07c is in 3.12 an later. So you can compare the binaries in 3.11 and 3.12 to see if it has an effect. It was fixed for me when I built the binary locally (!6945 (comment 83258)).

But 3.12 is already too slow for me. I thought that change was an improvement over that.

That's why I wonder if the patch has any effect (e.g. compared to 3.11) or if something on the build server is overriding the prefs. E.g. here

3462e07c had a possitive effect in performance. I posted details at !6945 (merged)

As python involved a lot in build stack it could use -O2 same as gcc.

Should it wait for 3.13 release?

I would say so. 3.13 is already being built, so a large change like that can better wait until after 3.13.

What about the idea of adding a python3 subpackage python3-optimised so that the usual python3 package is compiled with "-Os" (and installed by default via alpine-base) and python3-optimised is compiled with "-O2" and can be installed to replace python3 if wished?

mentioned in merge request !15340 (merged)

Has anybody checked the alleged size savings of -Os vs -O2 or -O3?

It might turn out small or even negligible...

!15340 (merged) shows ~10-15% increase in size for O2

I gonna check perf after digging ¡6945

Has anybody checked the alleged size savings of -Os vs -O2 or -O3?

It might turn out small or even negligible...

I just built locally the Python3 package from Edge with the only change being replacing "-Os" with "-O2".

ls -l --block-size=MB
-rw-r--r-- 1 dermot users  13MB Dec  6 18:04 python3-3.8.6-r1.apk
-rw-r--r-- 1 dermot users   6MB Dec  6 18:03 python3-dbg-3.8.6-r1.apk
-rw-r--r-- 1 dermot users  25MB Dec  6 18:03 python3-dev-3.8.6-r1.apk
-rw-r--r-- 1 dermot users   1MB Dec  6 18:03 python3-doc-3.8.6-r1.apk
-rw-r--r-- 1 dermot users  17MB Dec  6 18:04 python3-tests-3.8.6-r1.apk
-rw-r--r-- 1 dermot users   2MB Dec  6 18:04 python3-wininst-3.8.6-r1.apk

$ ls -l
-rw-r--r-- 1 dermot users  12939721 Dec  6 18:04 python3-3.8.6-r1.apk
-rw-r--r-- 1 dermot users   5207834 Dec  6 18:03 python3-dbg-3.8.6-r1.apk
-rw-r--r-- 1 dermot users  24894471 Dec  6 18:03 python3-dev-3.8.6-r1.apk
-rw-r--r-- 1 dermot users     13074 Dec  6 18:03 python3-doc-3.8.6-r1.apk
-rw-r--r-- 1 dermot users  16930975 Dec  6 18:04 python3-tests-3.8.6-r1.apk
-rw-r--r-- 1 dermot users   1024512 Dec  6 18:04 python3-wininst-3.8.6-r1.apk

and after unpacking the APK locally to get installed size:

$ du -c -BKB
46502kB	total

So main python3 APK file is 13MB / 46.5MB unpacked versus the current package figures of 12.59MB / 44.84MB shown here. So its not much of a size increase.

Note: I haven't tested the built packages, only built them. Also by making the "-Os" to "-O2" change and nothing else then existing "--enable-optimizations" configure flag kicked back into life:

checking for --enable-optimizations... yes

which meant that python was built twice - a full set of tests are run for profiling and then the 2nd build uses the profiling information. However it appears the tests run can be tweaked.

In my case on a 8 core laptop the package build took approx. 20 minutes.

As a side note regarding the "-fno-semantic-interposition" CFLAG option that was added by @J0WI in !6945 (merged), I was reading Fedora's discussion from when they decided to use this and noticed:

CFLAGS="-fno-semantic-interposition"
LDFLAGS="-fno-semantic-interposition"

When Archlinux started using it they also added it to LDFLAGS as well as CFLAGS.

The MR from @J0WI only added it to CFLAGS. Wondering if adding it also to LDFLAGS is necessary/unnecessary for Alpine/musl...

Docker is not using -fno-semantic-interposition.

See also

• #11129 (comment 81527)
• https://github.com/docker-library/python/issues/501#issuecomment-653275704

BTW having worse Python performance than Ubuntu might be a hard reason for many projects to drop Alpine.

Lower performance means higher server cost and a few MB cannot compensate for that!

I think that's not relevant for low end or iot devices where a small footprint is still more important than performance

Especially in embedded systems like IOT, CPU power is more expensive than memory. It is often the limiting factor. I speak from 20 years of professional experience.

nodejs is also affected. The build system tries to use -O3 in some cases.

Some has been fixed in 53dd8b58

Has PYTHONOPTIMIZE an effect when building/installing individual Python packages?

this issue has turned into a bit of a mess. i don't think this should be a catch-all for "optimization flags", instead each change should be made as a separate issue/MR. since i think the original issue of python -Os vs -O2/-O3 has been resolved, i'm closing this now.

But nothing has been fixed here and it has been proved that Python should be optimized at least with "-O2".

closed

mentioned in merge request !17674 (merged)

mentioned in merge request !17675 (closed)

mentioned in merge request !17926 (merged)

mentioned in merge request !17676 (merged)