Loop device partitions randomly dissapear
We have created an alpine based VM that only has docker installed from the package repository.
The docker container runs a gitlab-runner which then runs our CI/CD jobs. While developing our tests for a repository that was making use of loop devices, we noticed that during the tests, the loop device partitions would randomly disappear So not the entire loop device, just a partition on the loop device.
The effect is random enough, that our current master branch, which only uses the loop device once or twice, passes about 95% of the time (and a retry of the job makes it pass the other times), but our test-heavy branch fails at some point 99% of the time.
These same tests running on our local machines (under docker) worked fine however. Moving of the (KVM-based) virtual machine to a local developers machine made (as expected) no difference.
Creating a second VM, with debian stretch and docker-ce from the docker repositories however yielded a working situation. The main difference we spotted so far are the kernel versions, alpine was the only one on 4.14, stretch was on 4.9, our local machines on 4.15 and 4.17, and docker versions.
Since with docker, alpine itself (e.g. musl etc) should be of very little impact, it’s hard to find the culprit at the moment. We also tried updating alpine from latest-stable to edge, but that made no difference.
Our repository and tests can be found here, https://gitlab.com/ultimaker/embedded/platform/um-update\_toolbox\_armhf/ I think t fully test it, a fork of the repo and setting up a gitlab-runner are an option i suppose. I can share our runner creation script if someone wants to go that route, it will be in a repository at some point in the future.
At the moment of this writing, the master does not hold all our tests yet, those are in the EMP-302 branch. We are working the next few days these into the master branch.
It is understandable that this is a very hard to diagnose problem, but at the moment, we have currently ran out of idea’s where to even begin looking. In the future, we may use docker from docker itself rather then the packaged version to see if that makes a difference at all, though as far as I understand, docker does not really do anything itself once a container is running, and it would be just the files inside the container, and the kernel working.
(from redmine: issue id 9412, created on 2018-09-11)