Docker network on IP disconnected after few days

engageub · June 11, 2024, 5:44pm

Hi,
I am using docker network with multiple IP address on different subnets.
After few days some of the networks got disconnected over a course of time.
Creating a new network on same IP address with different subnet works fine.
Note that are are no conflicting subnets or networks created.
Restarting the docker using “systemctl restart docker” recovers the existing network.

Could you please let me know why this happens and why do I have to restart the docker daemon to recover the existing network.
Can docker improve this to auto recover network on its own.

Thank you

bluepuma77 · June 11, 2024, 6:06pm

Maybe some log files can help to understand what’s happening.

engageub · June 11, 2024, 6:12pm

I just had a look at the docker service logs using the following command.
DNS queries are failing most of the time.

journalctl -u docker.service -n 100

Jun 11 18:06:16 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:16.833952322Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:16 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:16.834185168Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:17 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:17.527288049Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:17 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:17.527507700Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:34 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:34.456487032Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:34 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:34.456678640Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:40 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:40.369870506Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:40 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:40.369901159Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:40 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:40.564952211Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:40 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:40.565189717Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:41 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:41.718753837Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:45 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:45.263851642Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:06:45 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:06:45.263891880Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:06 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:06.936810551Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:06 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:06.937014105Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:07 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:07.791962362Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:10 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:10.097803717Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:10 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:10.938796813Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:10 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:10.938819130Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:12 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:12.125050257Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:12 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:12.125060285Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:12 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:12.278296475Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:12 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:12.278299192Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:26 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:26.192721431Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:28 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:28.543513066Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:40 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:40.538576941Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:07:40 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:07:40.538864099Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:08:05 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:08:05.762598502Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:08:07 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:08:07.279573302Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:08:07 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:08:07.279816745Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:08:09 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:08:09.084609932Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:08:09 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:08:09.084873298Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:08:34 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:08:34.516047608Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:08:46 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:08:46.120191191Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:08:49 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:08:49.682985126Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:08:50 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:08:50.744680071Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:09:10 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:09:10.427143821Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:09:19 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:09:19.508309167Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:09:19 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:09:19.508312208Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:10:02 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:10:02.370612303Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:10:22 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:10:22.210964363Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:10:30 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:10:30.895766478Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:10:33 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:10:33.312123481Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:10:33 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:10:33.312242354Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:10:33 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:10:33.312306915Z" level=error msg="[resolver] failed to query DNS serve>
Jun 11 18:10:33 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T18:10:33.312372988Z" level=error msg="[resolver] failed to query DNS serve>

rimelek · June 11, 2024, 7:50pm

I updated your post to format the logs and the command. Please, follow this guide next time to make the message readable: How to format your forum posts

Which version of Docker are you using and on what OS? The output of docker info and docker version could help. Remove any details from the output you wouldn’t share.

About the shared error message: Is there anything at the end of the message afer “serve>”? If you add the --no-pager flag to journalctl it should return the whole output.

You should also go back to a point when ther were other kind of error messages that can explain why you see the DNS query error now. The --since parameter of journalctl expects a timestamp if you can scroll up enough to odler logs. It is also worth to check system logs of other services to see what happened before the error first occured so checking the logs without -u docker.service could be a good idea as well.

You mean it actually disconnected and when you checked the docker network inspect NETWORKNAME you didn’t see the container in the list of connected containers? I don’t think Docker would do that, but I could imagine an event that causes the network to disconnect. For example some cronjob or other event changes the network settings or an antivirus, firewall changes something.

engageub · June 11, 2024, 8:30pm

Hi,
Thank you for the response and providing details on how to get the logs.
Please find the results below.

Docker info output

Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false

Server:
 Containers: 57
  Running: 52
  Paused: 0
  Stopped: 5
 Images: 9
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-43-generic
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.931GiB
 Name: 178.254.168.48
 ID: 9e3e39ca-f8a5-40dc-885d-fda7c2fd3f4a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Docker Version output

Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.3
 Git commit:        24.0.5-0ubuntu1~22.04.1
 Built:             Mon Aug 21 19:50:14 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.3
  Git commit:       24.0.5-0ubuntu1~22.04.1
  Built:            Mon Aug 21 19:50:14 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~22.04.2
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

Used the following command to get the DNS query messages.
journalctl -u docker.service --no-pager|less

un 10 00:00:01 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:01.103769715Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;quality.mysterium.network.\tIN\t AAAA" error="read udp 127.0.0.1:58476->127.0.0.53:53: i/o timeout"
Jun 10 00:00:01 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:01.103774755Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;quality.mysterium.network.\tIN\t AAAA" error="read udp 127.0.0.1:46810->127.0.0.53:53: i/o timeout"
Jun 10 00:00:09 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:09.288277206Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;api.package.coreservice.io.\tIN\t AAAA" error="read udp 127.0.0.1:44969->127.0.0.53:53: i/o timeout"
Jun 10 00:00:12 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:12.882033898Z" level=info msg="ignoring event" container=7301c34bb05692f7d0b148c394dbd7de63c6ff66fc71e8a31918b9189a74271f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 10 00:00:13 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:13.107530610Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jun 10 00:00:13 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:13.108254274Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Jun 10 00:00:17 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:17.813327557Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;quality.mysterium.network.\tIN\t AAAA" error="read udp 127.0.0.1:41095->127.0.0.53:53: i/o timeout"
Jun 10 00:00:20 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:20.778010195Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;geo.myip.link.\tIN\t A" error="read udp 127.0.0.1:36255->127.0.0.53:53: i/o timeout"
Jun 10 00:00:24 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:24.684056088Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;api.package.coreservice.io.\tIN\t AAAA" error="read udp 127.0.0.1:60393->127.0.0.53:53: i/o timeout"
Jun 10 00:00:27 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:27.432886927Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jun 10 00:00:27 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:27.432933197Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Jun 10 00:00:27 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:27.627354869Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jun 10 00:00:27 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:27.628727443Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Jun 10 00:00:27 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:27.862785099Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jun 10 00:00:27 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:27.862821328Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Jun 10 00:00:29 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:29.284165531Z" level=info msg="ignoring event" container=3604352db7c38ae4868a9cead4dc3b410e96c650e9bc21535f98693b57b689b3 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 10 00:00:29 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:29.547586108Z" level=info msg="ignoring event" container=739081d589b03ece958554db04a0b5cb5d407a8fcaa918cff8fa17217d746293 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 10 00:00:29 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:29.564930282Z" level=info msg="ignoring event" container=3695c185d990e7367606a9541d7de1e39be3aacebb1a3cf49dfe6190354a3a66 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 10 00:00:32 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:32.792098038Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;location.mysterium.network.\tIN\t A" error="read udp 127.0.0.1:40384->127.0.0.53:53: i/o timeout"
Jun 10 00:00:32 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:32.792352424Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;location.mysterium.network.\tIN\t A" error="read udp 127.0.0.1:45163->127.0.0.53:53: i/o timeout"
Jun 10 00:00:44 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:44.043387362Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;client.packetstream.io.\tIN\t AAAA" error="read udp 127.0.0.1:53548->127.0.0.53:53: i/o timeout"
Jun 10 00:00:44 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:44.043387494Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;profile.ess.apple.com.\tIN\t AAAA" error="read udp 127.0.0.1:53753->127.0.0.53:53: i/o timeout"
Jun 10 00:00:44 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:44.304638950Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;quality.mysterium.network.\tIN\t AAAA" error="read udp 127.0.0.1:58686->127.0.0.53:53: i/o timeout"
Jun 10 00:00:44 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:44.304648345Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;quality.mysterium.network.\tIN\t AAAA" error="read udp 127.0.0.1:51203->127.0.0.53:53: i/o timeout"
Jun 10 00:00:44 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:44.757595408Z" level=info msg="No non-localhost DNS nameserve

The following command was used to get the errors that happened yesterday. But I could not find logs related to network which was not working.
journalctl -u docker.service --no-pager |grep -v "query DNS server" | grep "Jun 10" | grep -i error

Jun 10 01:16:35 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T01:16:35.098855083Z" level=error msg="stream copy error: read /proc/self/fd/379: file already closed"
Jun 10 01:16:35 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T01:16:35.098967562Z" level=error msg="stream copy error: read /proc/self/fd/391: file already closed"
Jun 10 02:08:48 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T02:08:48.141971092Z" level=error msg="stream copy error: read /proc/self/fd/42: file already closed"
Jun 10 02:08:48 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T02:08:48.142637028Z" level=error msg="stream copy error: read /proc/self/fd/118: file already closed"
Jun 10 06:32:35 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T06:32:35.028436643Z" level=error msg="stream copy error: read /proc/self/fd/462: file already closed"
Jun 10 06:32:35 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T06:32:35.028552918Z" level=error msg="stream copy error: read /proc/self/fd/470: file already closed"
Jun 10 08:18:55 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T08:18:55.786889930Z" level=error msg="stream copy error: read /proc/self/fd/470: file already closed"
Jun 10 08:18:55 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T08:18:55.786890195Z" level=error msg="stream copy error: read /proc/self/fd/468: file already closed"
Jun 10 08:21:08 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T08:21:08.439982245Z" level=error msg="stream copy error: read /proc/self/fd/313: file already closed"
Jun 10 08:21:08 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T08:21:08.439995011Z" level=error msg="stream copy error: read /proc/self/fd/42: file already closed"
Jun 10 09:40:32 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T09:40:32.238121599Z" level=error msg="stream copy error: read /proc/self/fd/456: file already closed"
Jun 10 09:40:32 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T09:40:32.238220521Z" level=error msg="stream copy error: read /proc/self/fd/458: file already closed"
Jun 10 11:33:04 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T11:33:04.053168570Z" level=error msg="stream copy error: read /proc/self/fd/377: file already closed"
Jun 10 11:33:04 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T11:33:04.053179220Z" level=error msg="stream copy error: read /proc/self/fd/206: file already closed"
Jun 10 11:36:15 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T11:36:15.995709394Z" level=error msg="stream copy error: read /proc/self/fd/346: file already closed"
Jun 10 11:36:15 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T11:36:15.995842397Z" level=error msg="stream copy error: read /proc/self/fd/455: file already closed"
Jun 10 12:08:45 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T12:08:45.355306470Z" level=error msg="stream copy error: read /proc/self/fd/377: file already closed"
Jun 10 12:08:45 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T12:08:45.355392913Z" level=error msg="stream copy error: read /proc/self/fd/379: file already closed"
Jun 10 15:08:35 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T15:08:35.891094818Z" level=error msg="stream copy error: read /proc/self/fd/216: file already closed"
Jun 10 15:08:35 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T15:08:35.911435910Z" level=error msg="stream copy error: read /proc/self/fd/206: file already closed"
Jun 10 15:53:10 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T15:53:10.506499809Z" level=error msg="stream copy error: read /proc/self/fd/458: file already closed"
Jun 10 15:53:10 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T15:53:10.506930372Z" level=error msg="stream copy error: read /proc/self/fd/461: file already closed"
Jun 10 16:56:16 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T16:56:16.178931184Z" level=error msg="stream copy er

Regarding docker inspect network, there was no issue. with the results as I compared it with other networks already. There was no internet connection through the container which is connected through docker network. When I created a new network with same IP on different subnet, it worked fine. For the existing container to work, I had to run “systemctl restart docker” command.

meyay · June 12, 2024, 6:46am

Two observations:

you seem to use the docker.io package from Ubuntu’s repositories: it is maintained and supported by Canonical.
The error message indicates that you use the dns stub resolver. I prefer to disable it on docker hosts. Here is a post from askubuntu on how to disable the stub resolver: dns - How to disable systemd-resolved in Ubuntu? - Ask Ubuntu.

rimelek · June 12, 2024, 8:11am

That was new to me too. I knew about the config file, but I always disabled the stub resolver by manually removing the original symbolic link (/etc/resolv.conf) and relinking it to /run/systemd/resolve/resolv.conf. It originally points to /run/systemd/resolve/stub-resolv.conf

engageub · June 12, 2024, 2:23pm

Hi,
Yes, I installed docker[dot]io on ubuntu.
I looked into the logs for resolv.conf and indeed docker looks at other location to resolve DNS queries.

Location of resolv.conf used /run/systemd/resolve/resolv.conf
journalctl -u docker.service --no-pager |grep "resolv.conf" | grep -v "No non-localhost"

Jun 11 17:34:52 <MY_IP_ADDRESS> dockerd[3919255]: time="2024-06-11T17:34:52.244094206Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Jun 11 19:11:32 <MY_IP_ADDRESS> dockerd[3945249]: time="2024-06-11T19:11:32.553221773Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Jun 11 19:17:46 <MY_IP_ADDRESS> dockerd[3954533]: time="2024-06-11T19:17:46.996644787Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Jun 11 19:37:48 <MY_IP_ADDRESS> dockerd[3969486]: time="2024-06-11T19:37:48.087530606Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"

The file /run/systemd/resolve/resolv.conf has DNS server 1.1.1.1
cat /run/systemd/resolve/resolv.conf

# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 1.1.1.1
search .

Looking at the file there is a DNS server and is not empty.

engageub · June 12, 2024, 2:53pm

I also have concerns with logging in docker service.
I am using 10 multiple IPs with multiple networks.
But the logs only show for 1 IP even though the issue is with other networks or IPs.
Also, this IP is not primary IP address.
What is the logic behind printing this IP address. From where does it get this info.
Can DNS throttling result in this. If so, how is it working when new network with new subnet on same IP address is created. Is it not recovering when DNS is throttled.

Jun 10 00:00:01 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:01.103774755Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;quality.mysterium.network.\tIN\t AAAA" error="read udp 127.0.0.1:46810->127.0.0.53:53: i/o timeout"
Jun 10 00:00:09 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:09.288277206Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;api.package.coreservice.io.\tIN\t AAAA" error="read udp 127.0.0.1:44969->127.0.0.53:53: i/o timeout"
Jun 10 00:00:12 <MY_IP_ADDRESS> dockerd[6350]: time="2024-06-10T00:00:12.882033898Z" level=info msg="ignoring event" container=7301c34bb05692f7d0b148c394dbd7de63c6ff66fc71e8a31918b9189a74271f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 10 00:00:13 <MY_IP_ADDRESS> dockerd[6350]

Thank you

Topic		Replies	Views
Intermittent loss of network connectivity General	2	9196	March 13, 2017
Intermittent Docker Network Failures Docker Desktop macos	1	1056	December 7, 2023
Rebooting network service requires docker service? General	0	623	August 2, 2021
Cannot create network (br-....) conflicts with (br-...): networks have overlapping IPv4 Compose	1	2365	March 11, 2021
Docker container loses network connectivity intermittently Docker Desktop dns , docker , docker-compose , macos	10	27018	October 17, 2023

Docker network on IP disconnected after few days

Related topics