musl readdir() Cannot Be Used Reliably on NFS/SMB Shares if Contents of Directory are Changing (e.g. during "rm -rf")
Summary
We are running Alpine Linux 3.10 with musl version 1.1.22-r3.
It appears that musl's readdir()
implementation either has a bug or is missing the necessary logic to cope with reliable navigation of mounted network shares. The issue is that commands like rm -rf
that navigate the contents of directories while they are changing those contents (i.e. unlink-ing files) will not be able to iterate over all of the files in the directory if that directory is in an NFS or SMB mount.
It would seem that this is the same kind of issue that other distros ran into as far back as 2012; for example: https://bugs.centos.org/view.php?id=5496.
The issue is not reproducible with a ubuntu:16.04
container with the same file share, same mount settings, and same Kubernetes node (so exact same kernel). This implies that GNU LibC 6 has a more robust implementation of readdir()
.
Diagnosis for this issue started out as an issue in the Nextcloud server project; much of the information in this ticket was adapted from the original issue, which can be found here: https://github.com/nextcloud/server/issues/17980
Steps to Reproduce
-
Mount an NFS or SMB share within a folder of an Alpine Linux container.
In my case, this was done through the Azure File Kubernetes driver, like so:
kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: test-shares-standard provisioner: kubernetes.io/azure-file mountOptions: - uid=33 - gid=33 - dir_mode=0770 - file_mode=0770 parameters: skuName: Standard_LRS --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-shares-pvc spec: accessModes: - ReadWriteMany storageClassName: test-shares-standard resources: requests: storage: 1Gi --- kind: Deployment apiVersion: apps/v1 metadata: name: test-shares spec: replicas: 1 revisionHistoryLimit: 2 selector: matchLabels: app: backend-test-shares template: metadata: labels: app: backend-test-shares spec: volumes: - name: "volume-test-share" persistentVolumeClaim: claimName: "test-shares-pvc" containers: - name: share-tester image: alpine:3.10 command: [ "sleep" ] args: [ "99999" ] volumeMounts: - name: "volume-test-share" mountPath: "/var/www/html/data"
-
Shell in to the container.
-
Create the following script as
test.sh
inside the folder mounted in step 1:#!/usr/bin/env sh starting_file_count="$1" mkdir "test" echo "Creating '${starting_file_count}' test files..." for file_index in $(seq 1 "${starting_file_count}"); do filename="test/test_${file_index}" touch "${filename}" done echo "" echo "Trying to delete test files..." while [ -d "test" ]; do count_before=$(ls test/ | wc -l) count_deleted=$(rm -vrf test/ 2>/dev/null | wc -l) count_after=$( (ls test/ | wc -l) 2>/dev/null || echo 0) echo "DELETED: ${count_deleted} BEFORE: ${count_before} AFTER: ${count_after}" done echo ""
-
Make the script executable with
chmod 0755 test.sh
. -
Run the script with
./test.sh 128
.
What This Does
This script basically creates a bunch of zero-length files in a new folder under the current working directory (presumably, a mounted network share), and then tries to remove the folder with rm -rf
. It counts how many files are in the folder before it removes the folder vs. after attempting to remove the folder, as well as how many files that a verbose rm -rf
indicates it's removing. The script then loops until the folder is actually removed, printing counts along the way.
Expected Results
With a properly-functioning storage driver, kernel, and standard library, the script should be able to remove all files in a single pass.
Creating '128' test files...
Trying to delete test files...
DELETED: 129 BEFORE: 128 AFTER: 0
Actual Results
The script has to make two passes to remove all files. In the first pass, it deletes only 66 of the 128 files, as follows:
Creating '128' test files...
Trying to delete test files...
DELETED: 66 BEFORE: 128 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
In fact, the script has to make several passes as soon as it is creating more than 62 files in the test folder.
Analysis
I have not done a close examination of the code, but I can make some observations based on the behavior I am seeing.
The defect appears only after there are more than 62
files in the directory. My assumption is that the magic number would be 64
were it not for the fact that directories contain .
and ..
so those two take up 2 entries in the buffer.
Based on this, it appears that on the first call to readdir()
on a directory that is inside an NFS share, it gets access to a buffer of about 64
directory entries/files at a time and maintains a cookie/pointer/offset indicating which file will be next within the directory. Each time it hits the end of the buffer, it loads another 64
file entries from the NFS server.
If the directory has changed between the time it buffered the first 64
files and the time it requests the next 64
, the cookie/pointer/offset is not adjusted, so it's going to end up pointing at a file toward the end of the buffer instead of the first file in the new buffer.
For example, consider the contents of a folder that contains 128
sequentially-numbered files, like this:
.
..
file_1
file_2
file_3
...
file_126
file_127
file_128
In the first call to readdir()
, it will load .
, ..
, and file_1
through file_62
, and the offset for the next file will be pointed at file_63
.
If unlink()
are called on those first 62
files, the offset for the next file will now be pointing at file_125
instead of file_63
(a difference of 62
files). During the next call to readdir()
, another 64
directory entries will be read, but because of the offset, it will return files file_125
through file_128
. If those 4
files are unlink-ed, that leaves 62
files behind in the folder.
This explains why the number of files left behind during rm -rf
grows based on how many files are in the folder to begin with -- it's a paging offset bug.
Additional Data
What follows are various results of running the test script. At first it seems like the way Alpine behaves always forces the last few batches of files to be powers of 62 files, but that pattern breaks down as soon as there are more than 250 files (see the results for powers of 3).
Sequentially Increasing Files (1-50)
for i in $(seq 50); do count=$(expr $i \* 5); test.sh "${count}"; done
Creating '5' test files...
Trying to delete test files...
DELETED: 6 BEFORE: 5 AFTER: 0
Creating '10' test files...
Trying to delete test files...
DELETED: 11 BEFORE: 10 AFTER: 0
Creating '15' test files...
Trying to delete test files...
DELETED: 16 BEFORE: 15 AFTER: 0
Creating '20' test files...
Trying to delete test files...
DELETED: 21 BEFORE: 20 AFTER: 0
Creating '25' test files...
Trying to delete test files...
DELETED: 26 BEFORE: 25 AFTER: 0
Creating '30' test files...
Trying to delete test files...
DELETED: 31 BEFORE: 30 AFTER: 0
Creating '35' test files...
Trying to delete test files...
DELETED: 36 BEFORE: 35 AFTER: 0
Creating '40' test files...
Trying to delete test files...
DELETED: 41 BEFORE: 40 AFTER: 0
Creating '45' test files...
Trying to delete test files...
DELETED: 46 BEFORE: 45 AFTER: 0
Creating '50' test files...
Trying to delete test files...
DELETED: 51 BEFORE: 50 AFTER: 0
Creating '55' test files...
Trying to delete test files...
DELETED: 56 BEFORE: 55 AFTER: 0
Creating '60' test files...
Trying to delete test files...
DELETED: 61 BEFORE: 60 AFTER: 0
Creating '65' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 65 AFTER: 3
DELETED: 4 BEFORE: 3 AFTER: 0
Creating '70' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 70 AFTER: 8
DELETED: 9 BEFORE: 8 AFTER: 0
Creating '75' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 75 AFTER: 13
DELETED: 14 BEFORE: 13 AFTER: 0
Creating '80' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 80 AFTER: 18
DELETED: 19 BEFORE: 18 AFTER: 0
Creating '85' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 85 AFTER: 23
DELETED: 24 BEFORE: 23 AFTER: 0
Creating '90' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 90 AFTER: 28
DELETED: 29 BEFORE: 28 AFTER: 0
Creating '95' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 95 AFTER: 33
DELETED: 34 BEFORE: 33 AFTER: 0
Creating '100' test files...
DELETED: 62 BEFORE: 105 AFTER: 43
DELETED: 44 BEFORE: 43 AFTER: 0
Creating '110' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 110 AFTER: 48
DELETED: 49 BEFORE: 48 AFTER: 0
Creating '115' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 115 AFTER: 53
DELETED: 54 BEFORE: 53 AFTER: 0
Creating '120' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 120 AFTER: 58
DELETED: 59 BEFORE: 58 AFTER: 0
Creating '125' test files...
Trying to delete test files...
DELETED: 63 BEFORE: 125 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '130' test files...
Trying to delete test files...
DELETED: 68 BEFORE: 130 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '135' test files...
Trying to delete test files...
DELETED: 73 BEFORE: 135 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '140' test files...
Trying to delete test files...
DELETED: 78 BEFORE: 140 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '145' test files...
Trying to delete test files...
DELETED: 83 BEFORE: 145 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '150' test files...
Trying to delete test files...
DELETED: 88 BEFORE: 150 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '155' test files...
Trying to delete test files...
DELETED: 93 BEFORE: 155 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '160' test files...
Trying to delete test files...
DELETED: 98 BEFORE: 160 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '165' test files...
Trying to delete test files...
DELETED: 103 BEFORE: 165 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '170' test files...
Trying to delete test files...
DELETED: 108 BEFORE: 170 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '175' test files...
Trying to delete test files...
DELETED: 113 BEFORE: 175 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '180' test files...
Trying to delete test files...
DELETED: 118 BEFORE: 180 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '185' test files...
Trying to delete test files...
DELETED: 123 BEFORE: 185 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '190' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 190 AFTER: 64
DELETED: 62 BEFORE: 64 AFTER: 2
DELETED: 3 BEFORE: 2 AFTER: 0
Creating '195' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 195 AFTER: 69
DELETED: 62 BEFORE: 69 AFTER: 7
DELETED: 8 BEFORE: 7 AFTER: 0
Creating '200' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 200 AFTER: 74
DELETED: 62 BEFORE: 74 AFTER: 12
DELETED: 13 BEFORE: 12 AFTER: 0
Creating '205' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 205 AFTER: 79
DELETED: 62 BEFORE: 79 AFTER: 17
DELETED: 18 BEFORE: 17 AFTER: 0
Creating '210' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 210 AFTER: 84
DELETED: 62 BEFORE: 84 AFTER: 22
DELETED: 23 BEFORE: 22 AFTER: 0
Creating '215' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 215 AFTER: 89
DELETED: 62 BEFORE: 89 AFTER: 27
DELETED: 28 BEFORE: 27 AFTER: 0
Creating '220' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 220 AFTER: 94
DELETED: 62 BEFORE: 94 AFTER: 32
DELETED: 33 BEFORE: 32 AFTER: 0
Creating '225' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 225 AFTER: 99
DELETED: 62 BEFORE: 99 AFTER: 37
DELETED: 38 BEFORE: 37 AFTER: 0
Creating '230' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 230 AFTER: 104
DELETED: 62 BEFORE: 104 AFTER: 42
DELETED: 43 BEFORE: 42 AFTER: 0
Creating '235' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 235 AFTER: 109
DELETED: 62 BEFORE: 109 AFTER: 47
DELETED: 48 BEFORE: 47 AFTER: 0
Creating '240' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 240 AFTER: 114
DELETED: 62 BEFORE: 114 AFTER: 52
DELETED: 53 BEFORE: 52 AFTER: 0
Creating '245' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 245 AFTER: 119
DELETED: 62 BEFORE: 119 AFTER: 57
DELETED: 58 BEFORE: 57 AFTER: 0
Creating '250' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 250 AFTER: 124
DELETED: 62 BEFORE: 124 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating Files in Powers of 2
for i in $(seq 10); do count=$((2**$i)); test.sh "${count}"; done
Creating '2' test files...
Trying to delete test files...
DELETED: 3 BEFORE: 2 AFTER: 0
Creating '4' test files...
Trying to delete test files...
DELETED: 5 BEFORE: 4 AFTER: 0
Creating '8' test files...
Trying to delete test files...
DELETED: 9 BEFORE: 8 AFTER: 0
Creating '16' test files...
Trying to delete test files...
DELETED: 17 BEFORE: 16 AFTER: 0
Creating '32' test files...
Trying to delete test files...
DELETED: 33 BEFORE: 32 AFTER: 0
Creating '64' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 64 AFTER: 2
DELETED: 3 BEFORE: 2 AFTER: 0
Creating '128' test files...
Trying to delete test files...
DELETED: 66 BEFORE: 128 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '256' test files...
Trying to delete test files...
DELETED: 130 BEFORE: 256 AFTER: 126
DELETED: 64 BEFORE: 126 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '512' test files...
Trying to delete test files...
DELETED: 280 BEFORE: 512 AFTER: 232
DELETED: 126 BEFORE: 232 AFTER: 106
DELETED: 62 BEFORE: 106 AFTER: 44
DELETED: 45 BEFORE: 44 AFTER: 0
Creating '1024' test files...
Trying to delete test files...
DELETED: 558 BEFORE: 1024 AFTER: 466
DELETED: 234 BEFORE: 466 AFTER: 232
DELETED: 126 BEFORE: 232 AFTER: 106
DELETED: 62 BEFORE: 106 AFTER: 44
DELETED: 45 BEFORE: 44 AFTER: 0
Creating Files in Powers of 3
for i in $(seq 8); do count=$((3**$i)); test.sh "${count}"; done
Creating '3' test files...
Trying to delete test files...
DELETED: 4 BEFORE: 3 AFTER: 0
Creating '9' test files...
Trying to delete test files...
DELETED: 10 BEFORE: 9 AFTER: 0
Creating '27' test files...
Trying to delete test files...
DELETED: 28 BEFORE: 27 AFTER: 0
Creating '81' test files...
Trying to delete test files...
DELETED: 62 BEFORE: 81 AFTER: 19
DELETED: 20 BEFORE: 19 AFTER: 0
Creating '243' test files...
Trying to delete test files...
DELETED: 126 BEFORE: 243 AFTER: 117
DELETED: 62 BEFORE: 117 AFTER: 55
DELETED: 56 BEFORE: 55 AFTER: 0
Creating '729' test files...
Trying to delete test files...
DELETED: 402 BEFORE: 729 AFTER: 327
DELETED: 201 BEFORE: 327 AFTER: 126
DELETED: 64 BEFORE: 126 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0
Creating '2187' test files...
Trying to delete test files...
DELETED: 1101 BEFORE: 2187 AFTER: 1086
DELETED: 552 BEFORE: 1086 AFTER: 534
DELETED: 285 BEFORE: 534 AFTER: 249
DELETED: 126 BEFORE: 249 AFTER: 123
DELETED: 62 BEFORE: 123 AFTER: 61
DELETED: 62 BEFORE: 61 AFTER: 0
Creating '6561' test files...
Trying to delete test files...
DELETED: 3304 BEFORE: 6561 AFTER: 3257
DELETED: 1633 BEFORE: 3257 AFTER: 1624
DELETED: 857 BEFORE: 1624 AFTER: 767
DELETED: 388 BEFORE: 767 AFTER: 379
DELETED: 221 BEFORE: 379 AFTER: 158
DELETED: 96 BEFORE: 158 AFTER: 62
DELETED: 63 BEFORE: 62 AFTER: 0