Docker container DNS not working (3.12, 3.13)
I was having a problem with docker services not being able to resolve names in some containers. Which containers were affected was different after restart of all services. Running dockerd in debug mode showed errors on creating iptables rules and warnings about ip6tables missing. Adding ip6tables package to the system fixed the problem and resolved all the warnings and errors.
Fix
- Install ip6tables package.
Platforms Tested
- Raspberry Pi 4B (3.13)
- Raspberry Pi 3B+ (3.12)
Test docker-compose.yml
version: "3.5" services: alpine1: image: alpine:latest command: ping -q -c 5 google.com alpine2: image: alpine:latest command: ping -q -c 5 google.com alpine3: image: alpine:latest command: ping -q -c 5 google.com alpine4: image: alpine:latest command: ping -q -c 5 google.com alpine5: image: alpine:latest command: ping -q -c 5 google.com
dockerd -D errors and warnings
WARN[0000] Failed to find ip6tables: exec: "ip6tables": executable file not found in $PATH WARN[0000] Failed to find ip6tables: exec: "ip6tables": executable file not found in $PATH ERRO[0000] set up rule failed, [-t nat -I DOCKER_OUTPUT -d 127.0.0.11 -p udp --dport 53 -j DNAT --to-destination 127.0.0.11:50322] WARN[0000] Failed to find ip6tables: exec: "ip6tables": executable file not found in $PATH WARN[0000] Failed to find ip6tables: exec: "ip6tables": executable file not found in $PATH ERRO[0000] set up rule failed, [-t nat -I DOCKER_POSTROUTING -s 127.0.0.11 -p udp --sport 50322 -j SNAT --to-source :53] ERRO[0000] set up rule failed, [-t nat -I DOCKER_OUTPUT -d 127.0.0.11 -p tcp --dport 53 -j DNAT --to-destination 127.0.0.11:34851] ERRO[0000] set up rule failed, [-t nat -I DOCKER_POSTROUTING -s 127.0.0.11 -p tcp --sport 34851 -j SNAT --to-source :53] DEBU[2020-08-17T23:31:20.438337253Z] sandbox set key processing took 1.826758801s for container 742b778d8d4f3874cdc68e539e8e4bdc4b68cf2d71e307075b21c5cb9dd65d5b ERRO[0000] set up rule failed, [-t nat -I DOCKER_OUTPUT -d 127.0.0.11 -p udp --dport 53 -j DNAT --to-destination 127.0.0.11:58505] ERRO[0000] set up rule failed, [-t nat -I DOCKER_POSTROUTING -s 127.0.0.11 -p udp --sport 58505 -j SNAT --to-source :53] ERRO[0000] set up rule failed, [-t nat -I DOCKER_OUTPUT -d 127.0.0.11 -p udp --dport 53 -j DNAT --to-destination 127.0.0.11:45541] ERRO[0000] set up rule failed, [-t nat -I DOCKER_OUTPUT -d 127.0.0.11 -p tcp --dport 53 -j DNAT --to-destination 127.0.0.11:39729] DEBU[2020-08-17T23:31:20.505734492Z] event forwarded ns=moby topic=/tasks/create type=containerd.events.TaskCreate DEBU[2020-08-17T23:31:20.506387930Z] event module=libcontainerd namespace=moby topic=/tasks/create ERRO[0000] set up rule failed, [-t nat -I DOCKER_POSTROUTING -s 127.0.0.11 -p tcp --sport 39729 -j SNAT --to-source :53] ERRO[0000] set up rule failed, [-t nat -I DOCKER_OUTPUT -d 127.0.0.11 -p tcp --dport 53 -j DNAT --to-destination 127.0.0.11:43891]
Successful run output
rpi:~/test# docker-compose up Starting test_alpine2_1 ... done Starting test_alpine4_1 ... done Starting test_alpine3_1 ... done Starting test_alpine1_1 ... done Starting test_alpine5_1 ... done Attaching to test_alpine3_1, test_alpine2_1, test_alpine1_1, test_alpine4_1, test_alpine5_1 alpine2_1 | PING google.com (172.217.6.14): 56 data bytes alpine3_1 | PING google.com (172.217.6.14): 56 data bytes alpine1_1 | PING google.com (172.217.6.14): 56 data bytes alpine4_1 | PING google.com (172.217.6.14): 56 data bytes alpine5_1 | PING google.com (172.217.6.14): 56 data bytes alpine3_1 | alpine3_1 | --- google.com ping statistics --- alpine3_1 | 5 packets transmitted, 5 packets received, 0% packet loss alpine3_1 | round-trip min/avg/max = 14.771/16.923/18.196 ms alpine2_1 | alpine2_1 | --- google.com ping statistics --- alpine2_1 | 5 packets transmitted, 5 packets received, 0% packet loss alpine2_1 | round-trip min/avg/max = 18.134/23.069/31.259 ms test_alpine3_1 exited with code 0 alpine1_1 | alpine1_1 | --- google.com ping statistics --- alpine1_1 | 5 packets transmitted, 5 packets received, 0% packet loss alpine1_1 | round-trip min/avg/max = 13.781/16.928/21.306 ms test_alpine2_1 exited with code 0 alpine4_1 | alpine4_1 | --- google.com ping statistics --- alpine4_1 | 5 packets transmitted, 5 packets received, 0% packet loss alpine4_1 | round-trip min/avg/max = 13.684/19.815/24.997 ms test_alpine1_1 exited with code 0 test_alpine4_1 exited with code 0 alpine5_1 | alpine5_1 | --- google.com ping statistics --- alpine5_1 | 5 packets transmitted, 4 packets received, 20% packet loss alpine5_1 | round-trip min/avg/max = 16.196/16.672/17.281 ms test_alpine5_1 exited with code 0
Failure run output
rpi:~/test# docker-compose up Starting test_alpine4_1 ... done Starting test_alpine1_1 ... done Starting test_alpine2_1 ... done Starting test_alpine3_1 ... done Starting test_alpine5_1 ... done Attaching to test_alpine4_1, test_alpine3_1, test_alpine1_1, test_alpine2_1, test_alpine5_1 alpine3_1 | PING google.com (172.217.6.14): 56 data bytes alpine4_1 | PING google.com (172.217.6.14): 56 data bytes alpine4_1 | alpine4_1 | --- google.com ping statistics --- alpine4_1 | 5 packets transmitted, 5 packets received, 0% packet loss alpine4_1 | round-trip min/avg/max = 14.189/18.578/23.441 ms test_alpine4_1 exited with code 0 alpine3_1 | alpine3_1 | --- google.com ping statistics --- alpine3_1 | 5 packets transmitted, 5 packets received, 0% packet loss alpine3_1 | round-trip min/avg/max = 13.652/21.210/40.161 ms test_alpine3_1 exited with code 0 alpine1_1 | ping: bad address 'google.com' alpine2_1 | ping: bad address 'google.com' alpine5_1 | ping: bad address 'google.com' test_alpine1_1 exited with code 1 test_alpine2_1 exited with code 1 test_alpine5_1 exited with code 1