libvirt getting locked while forking when creating a domain external snapshot on Alpine 3.12 x86_64
This issue is related to 11602. While creating a domain external snapshot libvirt process gets locked while forking. Unfortunately the issue happens randomly so it's not that easy to reproduce but when it happens the only way to move on is to terminate the child process created.
Please find below gbd output:
GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-alpine-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 2229
Reading symbols from /usr/sbin/libvirtd...
(No debugging symbols found in /usr/sbin/libvirtd)
Reading symbols from /usr/lib/libvirt-lxc.so.0...
(No debugging symbols found in /usr/lib/libvirt-lxc.so.0)
Reading symbols from /usr/lib/libvirt-qemu.so.0...
(No debugging symbols found in /usr/lib/libvirt-qemu.so.0)
--Type <RET> for more, q to quit, c to continue without paging--c
Reading symbols from /usr/lib/libvirt.so.0...
(No debugging symbols found in /usr/lib/libvirt.so.0)
Reading symbols from /usr/lib/libtirpc.so.3...
(No debugging symbols found in /usr/lib/libtirpc.so.3)
Reading symbols from /usr/lib/libdbus-1.so.3...
(No debugging symbols found in /usr/lib/libdbus-1.so.3)
Reading symbols from /usr/lib/libgobject-2.0.so.0...
(No debugging symbols found in /usr/lib/libgobject-2.0.so.0)
Reading symbols from /usr/lib/libglib-2.0.so.0...
(No debugging symbols found in /usr/lib/libglib-2.0.so.0)
Reading symbols from /usr/lib/libintl.so.8...
(No debugging symbols found in /usr/lib/libintl.so.8)
Reading symbols from /usr/lib/libgcc_s.so.1...
(No debugging symbols found in /usr/lib/libgcc_s.so.1)
Reading symbols from /lib/ld-musl-x86_64.so.1...
Reading symbols from /usr/lib/debug//lib/ld-musl-x86_64.so.1.debug...
Reading symbols from /usr/lib/libcap-ng.so.0...
(No debugging symbols found in /usr/lib/libcap-ng.so.0)
Reading symbols from /usr/lib/libyajl.so.2...
(No debugging symbols found in /usr/lib/libyajl.so.2)
Reading symbols from /usr/lib/libnl-3.so.200...
(No debugging symbols found in /usr/lib/libnl-3.so.200)
Reading symbols from /usr/lib/libxml2.so.2...
(No debugging symbols found in /usr/lib/libxml2.so.2)
Reading symbols from /usr/lib/libgio-2.0.so.0...
(No debugging symbols found in /usr/lib/libgio-2.0.so.0)
Reading symbols from /usr/lib/libsasl2.so.3...
(No debugging symbols found in /usr/lib/libsasl2.so.3)
Reading symbols from /usr/lib/libgnutls.so.30...
(No debugging symbols found in /usr/lib/libgnutls.so.30)
Reading symbols from /usr/lib/libcurl.so.4...
(No debugging symbols found in /usr/lib/libcurl.so.4)
Reading symbols from /usr/lib/libgssapi_krb5.so.2...
(No debugging symbols found in /usr/lib/libgssapi_krb5.so.2)
Reading symbols from /usr/lib/libffi.so.7...
(No debugging symbols found in /usr/lib/libffi.so.7)
Reading symbols from /usr/lib/libpcre.so.1...
(No debugging symbols found in /usr/lib/libpcre.so.1)
Reading symbols from /lib/libz.so.1...
(No debugging symbols found in /lib/libz.so.1)
Reading symbols from /usr/lib/liblzma.so.5...
(No debugging symbols found in /usr/lib/liblzma.so.5)
Reading symbols from /usr/lib/libgmodule-2.0.so.0...
(No debugging symbols found in /usr/lib/libgmodule-2.0.so.0)
Reading symbols from /lib/libmount.so.1...
(No debugging symbols found in /lib/libmount.so.1)
Reading symbols from /usr/lib/libp11-kit.so.0...
(No debugging symbols found in /usr/lib/libp11-kit.so.0)
Reading symbols from /usr/lib/libunistring.so.2...
(No debugging symbols found in /usr/lib/libunistring.so.2)
Reading symbols from /usr/lib/libtasn1.so.6...
(No debugging symbols found in /usr/lib/libtasn1.so.6)
Reading symbols from /usr/lib/libnettle.so.7...
(No debugging symbols found in /usr/lib/libnettle.so.7)
Reading symbols from /usr/lib/libhogweed.so.5...
(No debugging symbols found in /usr/lib/libhogweed.so.5)
Reading symbols from /usr/lib/libgmp.so.10...
(No debugging symbols found in /usr/lib/libgmp.so.10)
Reading symbols from /usr/lib/libnghttp2.so.14...
(No debugging symbols found in /usr/lib/libnghttp2.so.14)
Reading symbols from /lib/libssl.so.1.1...
(No debugging symbols found in /lib/libssl.so.1.1)
Reading symbols from /lib/libcrypto.so.1.1...
(No debugging symbols found in /lib/libcrypto.so.1.1)
Reading symbols from /usr/lib/libkrb5.so.3...
(No debugging symbols found in /usr/lib/libkrb5.so.3)
Reading symbols from /usr/lib/libk5crypto.so.3...
(No debugging symbols found in /usr/lib/libk5crypto.so.3)
Reading symbols from /lib/libcom_err.so.2...
(No debugging symbols found in /lib/libcom_err.so.2)
Reading symbols from /usr/lib/libkrb5support.so.0...
(No debugging symbols found in /usr/lib/libkrb5support.so.0)
Reading symbols from /lib/libblkid.so.1...
(No debugging symbols found in /lib/libblkid.so.1)
Reading symbols from /usr/lib/libkeyutils.so.1...
(No debugging symbols found in /usr/lib/libkeyutils.so.1)
Reading symbols from /usr/lib/libvirt/connection-driver/libvirt_driver_network.so...
(No debugging symbols found in /usr/lib/libvirt/connection-driver/libvirt_driver_network.so)
Reading symbols from /usr/lib/libvirt/connection-driver/libvirt_driver_interface.so...
(No debugging symbols found in /usr/lib/libvirt/connection-driver/libvirt_driver_interface.so)
Reading symbols from /usr/lib/libnetcf.so.1...
(No debugging symbols found in /usr/lib/libnetcf.so.1)
Reading symbols from /lib/libudev.so.1...
(No debugging symbols found in /lib/libudev.so.1)
Reading symbols from /usr/lib/libaugeas.so.0...
(No debugging symbols found in /usr/lib/libaugeas.so.0)
Reading symbols from /usr/lib/libexslt.so.0...
(No debugging symbols found in /usr/lib/libexslt.so.0)
Reading symbols from /usr/lib/libxslt.so.1...
(No debugging symbols found in /usr/lib/libxslt.so.1)
Reading symbols from /usr/lib/libnl-route-3.so.200...
(No debugging symbols found in /usr/lib/libnl-route-3.so.200)
Reading symbols from /usr/lib/libfa.so.1...
(No debugging symbols found in /usr/lib/libfa.so.1)
Reading symbols from /usr/lib/libgcrypt.so.20...
(No debugging symbols found in /usr/lib/libgcrypt.so.20)
Reading symbols from /usr/lib/libgpg-error.so.0...
(No debugging symbols found in /usr/lib/libgpg-error.so.0)
Reading symbols from /usr/lib/libvirt/connection-driver/libvirt_driver_secret.so...
(No debugging symbols found in /usr/lib/libvirt/connection-driver/libvirt_driver_secret.so)
Reading symbols from /usr/lib/libvirt/connection-driver/libvirt_driver_storage.so...
(No debugging symbols found in /usr/lib/libvirt/connection-driver/libvirt_driver_storage.so)
Reading symbols from /usr/lib/libvirt/storage-backend/libvirt_storage_backend_fs.so...
(No debugging symbols found in /usr/lib/libvirt/storage-backend/libvirt_storage_backend_fs.so)
Reading symbols from /usr/lib/libvirt/storage-backend/libvirt_storage_backend_logical.so...
(No debugging symbols found in /usr/lib/libvirt/storage-backend/libvirt_storage_backend_logical.so)
Reading symbols from /usr/lib/libvirt/storage-backend/libvirt_storage_backend_scsi.so...
(No debugging symbols found in /usr/lib/libvirt/storage-backend/libvirt_storage_backend_scsi.so)
Reading symbols from /usr/lib/libvirt/storage-backend/libvirt_storage_backend_mpath.so...
(No debugging symbols found in /usr/lib/libvirt/storage-backend/libvirt_storage_backend_mpath.so)
Reading symbols from /lib/libdevmapper.so.1.02...
(No debugging symbols found in /lib/libdevmapper.so.1.02)
Reading symbols from /usr/lib/libvirt/storage-backend/libvirt_storage_backend_disk.so...
(No debugging symbols found in /usr/lib/libvirt/storage-backend/libvirt_storage_backend_disk.so)
Reading symbols from /usr/lib/libvirt/storage-backend/libvirt_storage_backend_zfs.so...
(No debugging symbols found in /usr/lib/libvirt/storage-backend/libvirt_storage_backend_zfs.so)
Reading symbols from /usr/lib/libvirt/connection-driver/libvirt_driver_nodedev.so...
(No debugging symbols found in /usr/lib/libvirt/connection-driver/libvirt_driver_nodedev.so)
Reading symbols from /usr/lib/libpciaccess.so.0...
(No debugging symbols found in /usr/lib/libpciaccess.so.0)
Reading symbols from /usr/lib/libvirt/connection-driver/libvirt_driver_nwfilter.so...
(No debugging symbols found in /usr/lib/libvirt/connection-driver/libvirt_driver_nwfilter.so)
Reading symbols from /usr/lib/libpcap.so.1...
(No debugging symbols found in /usr/lib/libpcap.so.1)
Reading symbols from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so...
(No debugging symbols found in /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so)
Reading symbols from /usr/lib/libvirt/storage-file/libvirt_storage_file_fs.so...
(No debugging symbols found in /usr/lib/libvirt/storage-file/libvirt_storage_file_fs.so)
__wait (addr=addr@entry=0x7fe200bfbaa0 <mal+32>, waiters=waiters@entry=0x7fe200bfbaa4 <mal+36>, val=val@entry=1, priv=128, priv@entry=1) at ./arch/x86_64/syscall_arch.h:40
40 ./arch/x86_64/syscall_arch.h: No such file or directory.
(gdb) backtrace
#0 __wait (addr=addr@entry=0x7fe200bfbaa0 <mal+32>, waiters=waiters@entry=0x7fe200bfbaa4 <mal+36>, val=val@entry=1, priv=128, priv@entry=1)
at ./arch/x86_64/syscall_arch.h:40
#1 0x00007fe200b8c3ca in lock (lk=0x7fe200bfbaa0 <mal+32>) at src/malloc/malloc.c:31
#2 lock_bin (i=1) at src/malloc/malloc.c:46
#3 malloc (n=<optimized out>, n@entry=24) at src/malloc/malloc.c:320
#4 0x00007fe200b8c584 in calloc (m=<optimized out>, n=24) at src/malloc/malloc.c:361
#5 0x00007fe20062212e in g_malloc0 () from /usr/lib/libglib-2.0.so.0
#6 0x00007fe20082c71f in virAlloc () from /usr/lib/libvirt.so.0
#7 0x00007fe200932517 in ?? () from /usr/lib/libvirt.so.0
#8 0x00007fe20092e281 in ?? () from /usr/lib/libvirt.so.0
#9 0x00007fe200882704 in virProcessRunInFork () from /usr/lib/libvirt.so.0
#10 0x00007fe20092e20e in ?? () from /usr/lib/libvirt.so.0
#11 0x00007fe20093104e in virSecurityManagerMoveImageMetadata () from /usr/lib/libvirt.so.0
#12 0x00007fe20092d6fb in ?? () from /usr/lib/libvirt.so.0
#13 0x00007fe20093104e in virSecurityManagerMoveImageMetadata () from /usr/lib/libvirt.so.0
#14 0x00007fe1ff1a8b54 in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so
#15 0x00007fe200a16732 in virDomainSnapshotCreateXML () from /usr/lib/libvirt.so.0
#16 0x0000563c90480abc in ?? ()
#17 0x00007fe200951554 in virNetServerProgramDispatch () from /usr/lib/libvirt.so.0
#18 0x00007fe200955a41 in ?? () from /usr/lib/libvirt.so.0
#19 0x00007fe20089b74c in ?? () from /usr/lib/libvirt.so.0
#20 0x00007fe20089b0cb in ?? () from /usr/lib/libvirt.so.0
--Type <RET> for more, q to quit, c to continue without paging--c
#21 0x00007fe200bba7b7 in start (p=0x7fe1ff6cf858) at src/thread/pthread_create.c:195
#22 0x00007fe200bbc8f0 in __clone () at src/thread/x86_64/clone.s:22
Backtrace stopped: frame did not save the PC
(gdb)
Thanks.