I have been using Docker for several years. I usually have about 14 containers running in parallel doing various things. At some point over the last 3-6 months I’ve seen network connections failing. I don’t know for certain, but I believe it started following an upgrade of MacOS to Sonoma. Restarting the docker engine resolves the issue, but it will re-occur after 2-4 days. When the issue happens, some connections will work, some connections won’t work, and new connections may or may not work, it is very unpredictable.
I attached an example screenshot, where I do an nslookup from a container and it fails to get a response, but then I do another nslookup a moment later and it succeeds. The DNS server shows it received the request and responded in both cases. I’m using a DNS request to demonstrate the intermittent nature, but the connection failures are more than just DNS. Even container-container connections in a private docker network will have connection failures; it’s not just external connections.
At this point I am looking for any suggestions to investigate troubleshooting. I’ve tried increasing Max Connections per this thread, but no luck.
I’ve found these messages in the dockerd.log file. It seems like a promising that this may be the cause of the issues, but I’m not sure what causes “i/o timeout” ?
In all cases I can see the DNS server received and replied to these DNS requests.
[2023-12-07T17:05:11.625989001Z][dockerd][I] time=“2023-12-07T17:05:11.625931126Z” level=error msg=“[resolver] failed to query DNS server: 10.0.0.100:53, query: ;cloudflare.com.\tIN\t A” error=“write udp 192.168.32.114:54390->10.0.0.100:53: write: operation not permitted”
[2023-12-07T17:05:16.627145003Z][dockerd][I] time=“2023-12-07T17:05:16.627026587Z” level=error msg=“[resolver] failed to query DNS server: 10.0.0.100:53, query: ;.\tIN\t A” error=“read udp 10.7.1.8:46440->10.0.0.100:53: i/o timeout”
[2023-12-07T17:23:48.957729046Z][dockerd][I] time=“2023-12-07T17:23:48.957269713Z” level=error msg=“[resolver] failed to query DNS server: 10.0.0.100:53, query: ;.\tIN\t A” error=“read udp 192.168.32.109:34166->10.0.0.100:53: i/o timeout”
[2023-12-07T17:50:48.951799879Z][dockerd][I] time=“2023-12-07T17:50:48.951327296Z” level=error msg=“[resolver] failed to query DNS server: 10.0.0.100:53, query: ;.\tIN\t A” error=“read udp 192.168.32.109:41591->10.0.0.100:53: i/o timeout”
[2023-12-07T17:50:48.964799629Z][dockerd][I] time=“2023-12-07T17:50:48.964671254Z” level=error msg=“[resolver] failed to query DNS server: 10.0.0.100:53, query: ;.\tIN\t AAAA” error=“read udp 192.168.32.109:34387->10.0.0.100:53: i/o timeout”
