I’ve been having issues for the last few days with constant network dropouts and alarms showing up on my netdata monitoring. Netdata is running on my docker server (along with a few other things) and there have been ongoing issues with ubiquity, the server of which is running on this machine as well.
I’ve just looked in the logs and I can see a constant stream of messages being written that seem to indicate that the networking is totally screwed up.
The log messages are like this
Jun 9 14:11:43 dockerhost networkd-dispatcher[672]: WARNING:Unknown index 274763 seen, reloading interface list Jun 9 14:11:43 dockerhost systemd-udevd[1675399]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jun 9 14:11:43 dockerhost systemd-udevd[1675405]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. Jun 9 14:11:43 dockerhost systemd-networkd[646]: vethd42a850: Link UP Jun 9 14:11:43 dockerhost systemd-udevd[1675405]: Using default interface naming scheme 'v245'. Jun 9 14:11:43 dockerhost systemd-udevd[1675399]: vethd42a850: Could not generate persistent MAC: No data available Jun 9 14:11:43 dockerhost kernel: [404650.029766] docker_gwbridge: port 9(vethd42a850) entered blocking state Jun 9 14:11:43 dockerhost kernel: [404650.029769] docker_gwbridge: port 9(vethd42a850) entered disabled state Jun 9 14:11:43 dockerhost kernel: [404650.029992] device vethd42a850 entered promiscuous mode Jun 9 14:11:43 dockerhost kernel: [404650.030873] docker_gwbridge: port 9(vethd42a850) entered blocking state Jun 9 14:11:43 dockerhost kernel: [404650.030875] docker_gwbridge: port 9(vethd42a850) entered forwarding state
Jun 9 14:11:43 dockerhost systemd-udevd[1675405]: veth3da96b6: Could not generate persistent MAC: No data available
Jun 9 14:11:43 dockerhost networkd-dispatcher[672]: ERROR:Unknown interface index 274763 seen even after reload
Jun 9 14:11:43 dockerhost networkd-dispatcher[672]: WARNING:Unknown index 274763 seen, reloading interface list
Jun 9 14:11:43 dockerhost networkd-dispatcher[672]: ERROR:Unknown interface index 274763 seen even after reload
Jun 9 14:11:43 dockerhost networkd-dispatcher[672]: WARNING:Unknown index 274766 seen, reloading interface list
Jun 9 14:11:43 dockerhost systemd-udevd[1675399]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 9 14:11:43 dockerhost systemd-udevd[1675399]: veth2bb22b8: Could not generate persistent MAC: No data available
Jun 9 14:11:43 dockerhost kernel: [404650.100421] veth45779: renamed from veth2bb22b8
Jun 9 14:11:43 dockerhost systemd-networkd[646]: veth2bb22b8: Failed to wait for the interface to be initialized: No such device
Jun 9 14:11:43 dockerhost kernel: [404650.116329] br0: port 4(veth45779) entered blocking state
Jun 9 14:11:43 dockerhost kernel: [404650.116330] br0: port 4(veth45779) entered disabled state
Jun 9 14:11:43 dockerhost kernel: [404650.116424] device veth45779 entered promiscuous mode
Jun 9 14:11:43 dockerhost kernel: [404650.116578] br0: port 4(veth45779) entered blocking state
Jun 9 14:11:43 dockerhost kernel: [404650.116580] br0: port 4(veth45779) entered forwarding state
Jun 9 14:11:43 dockerhost systemd-udevd[1675405]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 9 14:11:43 dockerhost systemd-udevd[1675405]: vethfa6bc19: Could not generate persistent MAC: No data available
Jun 9 14:11:43 dockerhost networkd-dispatcher[672]: WARNING:Unknown index 274767 seen, reloading interface list
Jun 9 14:11:43 dockerhost networkd-dispatcher[672]: ERROR:Unknown interface index 274767 seen even after reload
Jun 9 14:11:43 dockerhost networkd-dispatcher[672]: WARNING:Unknown index 274767 seen, reloading interface list
Jun 9 14:11:43 dockerhost networkd-dispatcher[672]: ERROR:Unknown interface index 274767 seen even after reload
Jun 9 14:11:43 dockerhost containerd[704]: time="2021-06-09T14:11:43.563159672+10:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/d607b3d5fbe7f583b30ed267b8d6277df2d21d8ee3ccda6af45daa4af12fcae0 pid=1675432
Jun 9 14:11:43 dockerhost kernel: [404650.536463] eth0: renamed from vethcddabeb
Jun 9 14:11:43 dockerhost kernel: [404650.552355] docker_gwbridge: port 9(vethd42a850) entered disabled state
Jun 9 14:11:43 dockerhost kernel: [404650.552529] br0: port 4(veth45779) entered disabled state
Jun 9 14:11:43 dockerhost kernel: [404650.552605] br0: port 6(veth45786) entered blocking state
Jun 9 14:11:43 dockerhost kernel: [404650.552606] br0: port 6(veth45786) entered forwarding state
etc etc...
This loop seems to be pretty constant and I guess is why things are not happy.
Any idea what this is? How to diagnose/fix? I don’t even know there to start, and my google skills aren’t helping.
Peter.