alpine network delay when operating HBASE via phoenix (#1) · Issues · alpine / infra / docker / ansible

alpine network delay when operating HBASE via phoenix

System environment

NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.11.5
PRETTY_NAME="Alpine Linux v3.11"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
~ $ uname -r
3.10.0-1062.18.1.el7.x86_64
~ $ apk list | grep musl
musl-1.1.24-r2 x86_64 {musl} (MIT) [installed]
musl-utils-1.1.24-r2 x86_64 {musl} (MIT BSD GPL2+) [installed]
musl-dev-1.1.24-r2 x86_64 {musl} (MIT) [installed]
~ $

Run test script in alpine

~ $ time python test.py 
1587781799.9128911
23.95860481262207
33667
0.6693758964538574
351
real    0m 25.33s
user    0m 24.46s
sys     0m 0.47s
~ $

Run test script on CentOS

(base) [root@bigdata-dev03 ~]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

(base) [root@bigdata-dev03 ~]# uname -r            
3.10.0-1062.18.1.el7.x86_64
(base) [root@bigdata-dev03 ~]# time python test.py      
1587781936.8903282
2.4388413429260254
33667
0.6046943664550781
351

real    0m3.930s
user    0m3.003s
sys     0m0.244s
(base) [root@bigdata-dev03 ~]#

Run the test script in alpine after upgrading the kernel

~ $ cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.11.5
PRETTY_NAME="Alpine Linux v3.11"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
~ $ uname -r
5.6.7-1.el7.elrepo.x86_64
~ $ time python test.py
1587781294.1356337
10.101239442825317
33667
0.367321252822876
351
real    0m 10.86s
user    0m 10.28s
sys     0m 0.09s
~ $

tcpdump intercepts packets

2273	21.438452	192.168.1.72	172.16.0.199	TCP	7006	[TCP Out-Of-Order] 80 → 33258 [ACK] Seq=7098353 Ack=5443 Win=50304 Len=6940 TSval=991811317 TSecr=979000685
2274	21.438487	192.168.1.72	172.16.0.199	TCP	4230	[TCP Out-Of-Order] 80 → 33258 [ACK] Seq=7105293 Ack=5443 Win=50304 Len=4164 TSval=991811317 TSecr=979000685
2275	21.438632	172.16.0.199	192.168.1.72	TCP	78	[TCP Window Update] 33258 → 80 [ACK] Seq=5443 Ack=7098353 Win=332672 Len=0 TSval=979000685 TSecr=991811316 SLE=7109457 SRE=7116397
2276	21.438701	172.16.0.199	192.168.1.72	TCP	78	[TCP Dup ACK 2270#1] 33258 → 80 [ACK] Seq=5443 Ack=7098353 Win=332672 Len=0 TSval=979000685 TSecr=991811316 SLE=7109457 SRE=7123337

I suspect it was caused by musl libc ......

System environment
```~ $ cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.11.5
PRETTY_NAME="Alpine Linux v3.11"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
~ $ uname -r
3.10.0-1062.18.1.el7.x86_64
~ $ apk list | grep musl
musl-1.1.24-r2 x86_64 {musl} (MIT) [installed]
musl-utils-1.1.24-r2 x86_64 {musl} (MIT BSD GPL2+) [installed]
musl-dev-1.1.24-r2 x86_64 {musl} (MIT) [installed]
~ $
```
Run test script in alpine
```
~ $ time python test.py 
1587781799.9128911
23.95860481262207
33667
0.6693758964538574
351
real    0m 25.33s
user    0m 24.46s
sys     0m 0.47s
~ $ 
```
Run test script on CentOS
```
(base) [root@bigdata-dev03 ~]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

(base) [root@bigdata-dev03 ~]# uname -r            
3.10.0-1062.18.1.el7.x86_64
(base) [root@bigdata-dev03 ~]# time python test.py      
1587781936.8903282
2.4388413429260254
33667
0.6046943664550781
351

real    0m3.930s
user    0m3.003s
sys     0m0.244s
(base) [root@bigdata-dev03 ~]# 
```

Run the test script in alpine after upgrading the kernel
```
~ $ cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.11.5
PRETTY_NAME="Alpine Linux v3.11"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
~ $ uname -r
5.6.7-1.el7.elrepo.x86_64
~ $ time python test.py
1587781294.1356337
10.101239442825317
33667
0.367321252822876
351
real    0m 10.86s
user    0m 10.28s
sys     0m 0.09s
~ $ 
```

tcpdump intercepts packets
```
2273	21.438452	192.168.1.72	172.16.0.199	TCP	7006	[TCP Out-Of-Order] 80 → 33258 [ACK] Seq=7098353 Ack=5443 Win=50304 Len=6940 TSval=991811317 TSecr=979000685
2274	21.438487	192.168.1.72	172.16.0.199	TCP	4230	[TCP Out-Of-Order] 80 → 33258 [ACK] Seq=7105293 Ack=5443 Win=50304 Len=4164 TSval=991811317 TSecr=979000685
2275	21.438632	172.16.0.199	192.168.1.72	TCP	78	[TCP Window Update] 33258 → 80 [ACK] Seq=5443 Ack=7098353 Win=332672 Len=0 TSval=979000685 TSecr=991811316 SLE=7109457 SRE=7116397
2276	21.438701	172.16.0.199	192.168.1.72	TCP	78	[TCP Dup ACK 2270#1] 33258 → 80 [ACK] Seq=5443 Ack=7098353 Win=332672 Len=0 TSval=979000685 TSecr=991811316 SLE=7109457 SRE=7123337
```

I suspect it was caused by musl libc ......

Discussion
Designs