NSS libraries are stripped which inhibits FIPS mode
The checksums for the NSS libraries libfreebl3.so, libnssdbm3.so and libsoftokn3.so seems to be incorrect. When I run docker container on the RHEL with FIPS mode enabled the NSS libraries fail to load from the JDK8u252.
From what I can see in the NSS build logs for NSS 3.60, I can see that the checksums are computed before the libraries are stripped.
Here we see the signing step:
Library File: ../../../nss/lib/freebl/Linux5.4_x86_64_gcc_glibc_PTH_64_OPT.OBJ/Linux_SINGLE_SHLIB/libfreebl3.so 800064 bytes
Check File: ../../../nss/lib/freebl/Linux5.4_x86_64_gcc_glibc_PTH_64_OPT.OBJ/Linux_SINGLE_SHLIB/libfreebl3.chk
Link: libfreebl3.chk
hash: 32 bytes
47 4f 5c b7 13 91 16 32 cb 9f
22 fd 19 84 10 e8 df fa 6c f8
29 73 fb cb 6c 9d 17 38 aa 5d
e1 2e
signature: 64 bytes
8c cb db 64 b2 3e 89 98 11 e7
57 71 42 6e 5a e8 ff 73 f0 98
43 69 66 ae 25 83 54 6d 06 91
00 17 59 e2 d0 a4 25 9a 42 c9
63 e3 1d e4 10 e0 3b 73 15 db
31 a3 c7 b8 f1 75 c1 6b 70 51
8e 60 17 66
Note the size of library is 800064 bytes.
But when the library is installed we get:
$ docker run -it alpine:3.12.3 sh
/ # apk add file nss
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/5) Installing libmagic (5.38-r0)
(2/5) Installing file (5.38-r0)
(3/5) Installing nspr (4.27-r0)
(4/5) Installing sqlite-libs (3.32.1-r0)
(5/5) Installing nss (3.60-r0)
Executing busybox-1.31.1-r19.trigger
OK: 16 MiB in 19 packages
/ # ls -l /usr/lib/libfreebl3.so.60
-rwxr-xr-x 1 root root 747400 Jan 6 21:15 /usr/lib/libfreebl3.so.60
/ # file /usr/lib/libfreebl3.so.60
/usr/lib/libfreebl3.so.60: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), statically linked, stripped
/ #
Further if I install nss-tools package and sign the libraries with shlibsign I get different checksums. Also after resigning libraries the JDK is able to load the libraries. The key thing here is that when FIPS mode is enabled the NSS libraries verify themselves against the checksums.