rsyslog segfaults when configured with TCP listener when a log entry arrives from remote source
I believe this is a musl threading issue. I was able to reproduce the
problem exactly on three different systems, using alpine 3.2, 3.3 and
edge.
I was also able to reproduce the problem in a development chroot alpine
environment.
Specific steps to reproduce the problem:
Install rsyslog and rsyslog-tls
Add a configuration entry in the rsyslog.conf file to turn on the TCP
message receiver, using the gtls network stream driver, and another
entry in the config file to listen on a chosen TCP port.
Run rsyslogd from the command line in tty-attached and debug mode with
“rsyslogd -nd”
In a different container or machine:
Install rsyslog
Add a configuration entry in this rsyslog.conf file to turn on remote
message passing, and set it to pass all log entries to a remote machine,
for example: . @@172.2.0.1:514
Run rsyslogd from the commandline.
The client (second container) will start to send some logs to the
rsyslog receiver (first machine) and as soon as it receives one, it will
segfault.
EXAMPLE SEGFAULT:
#0 SanitizeMsg (pMsg=pMsg@entry=0x5555558351c0) at parser.c:425
#1 0x000055555556ec05 in ParseMsg (pMsg=0x5555558351c0) at
parser.c:660
#2 0x0000555555588621 in ratelimitMsg (ratelimit=0x5555557f9880,
pMsg=pMsg@entry=0x5555558351c0, ppRepMsg=0x7ffff7ea2b10) at
ratelimit.c:209
#3 0x00005555555888a1 in ratelimitAddMsg (ratelimit=,
pMultiSub=pMultiSub@entry=0x7ffff7ea2b88, pMsg=0x5555558351c0) at
ratelimit.c:261
#4 0x00007ffff6728ba8 in defaultDoSubmitMessage
(pThis=pThis@entry=0x555555833140, stTime=stTime@entry=0x7ffff7ea2b98,
ttGenTime=ttGenTime@entry=1461801559,
pMultiSub=pMultiSub@entry=0x7ffff7ea2b88) at tcps_sess.c:265
#5 0x00007ffff6728df1 in processDataRcvd (pMultiSub=0x7ffff7ea2b88,
ttGenTime=1461801559, stTime=0x7ffff7ea2b98, c=10 ‘\n’,
pThis=0x555555833140) at tcps_sess.c:421
#6 DataRcvd (pThis=0x555555833140, pData=0x7ffff7ea4c94 “<38>Apr
27 23:59:19 b625498cbc9f sshd[14]: Server listening on 0.0.0.0 port
22.\n<38>Apr 27 23:59:19 b625498cbc9f sshd[14]: Server
listening on :: port 22.\n”, iLen=)
at tcps_sess.c:488
#7 0x00007ffff6729ab6 in doReceive (pThis=pThis@entry=0x5555557f9540,
ppSess=ppSess@entry=0x7ffff7ec4c58, pPoll=pPoll@entry=0x0) at
tcpsrv.c:579
#8 0x00007ffff6729f3f in processWorksetItem
(pThis=pThis@entry=0x5555557f9540, pPoll=pPoll@entry=0x0,
idx=, pUsr=0x555555833140) at tcpsrv.c:625
#9 0x00007ffff672a194 in processWorkset
(pThis=pThis@entry=0x5555557f9540, pPoll=pPoll@entry=0x0,
numEntries=numEntries@entry=1, workset=workset@entry=0x7ffff7ec4d50) at
tcpsrv.c:711
#10 0x00007ffff672a8b0 in RunSelect (sizeWorkset=128,
workset=0x7ffff7ec4d50, pThis=0x5555557f9540) at tcpsrv.c:823
#11 Run (pThis=0x5555557f9540) at tcpsrv.c:885
#12 0x000055555558dd57 in thrdStarter (arg=0x55555582a700) at
../threads.c:212
#13 0x00007ffff7dc7152 in ?? () from /lib/ld-musl-x86_64.so.1
#14 0x0000000000000000 in ?? ()
There are actually two different segfaults that I have seen when
testing.
When running the receiving instance in debug mode (with -d) it will
segfault in the DbgPrint function, specifically when attempting to write
a debugging message to stdout after receiving the log entry.
When running in non-debug mode, it segfaults in SanatiseMsg (as above),
specifically when stripping off trailing zeros and LFs from the end of
the message.
This segfault occurs when the TCP driver is in both mode 1 (TLS
encryption enabled) and mode 0 (plain TCP communication).
This bug does not occur when using the UDP receiver for communication.
Only with the TCP one.
I built an identical build of this version of rsyslog in Ubuntu Linux
(with glibc) and the segfault does not occur.
If there is any further specific information I can provide I am happy to do so.
(from redmine: issue id 5504, created on 2016-04-28)