Hello everyone,
I’m incredibly happy to report that the root cause of the problem has been identified and resolved!
After extensive troubleshooting, including downgrading Docker, verifying iptables and br_netfilter, the issue remained. The key to understanding the problem was the ICMP time exceeded in-transit messages we observed in tcpdump.
The final crucial diagnostic step (suggested offline) was to connect my system to a different network environment – specifically, my mobile hotspot.
Here are the tcpdump results from my wlp0s20f3 interface, interface showing the difference that led to the solution.
1. tcpdump output when connected to the problematic Wi-Fi network (TTL=1 issue):
tcpdump: listening on wlp0s20f3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:53:49.589520 IP (tos 0x0, ttl 63, id 55966, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.180.48 > 8.8.8.8: ICMP echo request, id 9, seq 0, length 64
10:53:49.617033 IP (tos 0x0, ttl 1, id 0, offset 0, flags [none], proto ICMP (1), length 84)
8.8.8.8 > 192.168.180.48: ICMP echo reply, id 9, seq 0, length 64
10:53:49.617074 IP (tos 0xc0, ttl 64, id 58242, offset 0, flags [none], proto ICMP (1), length 112)
192.168.180.48 > 8.8.8.8: ICMP time exceeded in-transit, length 92
IP (tos 0x0, ttl 1, id 0, offset 0, flags [none], proto ICMP (1), length 84)
8.8.8.8 > 192.168.180.48: ICMP echo reply, id 9, seq 0, length 64
Observation from this output: The incoming ICMP echo reply packets from 8.8.8.8 consistently had a TTL of 1. This caused my host to send ICMP time exceeded in-transit messages when attempting to forward the packet internally to the Docker bridge, as TTL would decrement to 0.
2. tcpdump output when connected to a mobile hotspot network (TTL is normal):
tcpdump: listening on wlp0s20f3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:02:43.006938 IP (tos 0x0, ttl 63, id 11785, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.123.124 > 8.8.8.8: ICMP echo request, id 11, seq 0, length 64
11:02:43.086475 IP (tos 0x0, ttl 117, id 0, offset 0, flags [none], proto ICMP (1), length 84)
8.8.8.8 > 192.168.123.124: ICMP echo reply, id 11, seq 0, length 64
Observation from this output: The incoming ICMP echo reply packets from 8.8.8.8 now have a normal TTL of 117 (instead of 1). My host no longer sends “ICMP time exceeded in-transit” messages.
Conclusion:
Crucially, with the normal TTL, ping 8.8.8.8 from inside the Docker container now works perfectly!
This definitively confirms that my system and Docker setup are fully functional under normal network conditions. The persistent issue was caused by the specific network I was previously connected to. That network (likely a router, firewall, or some other device) was somehow reducing the TTL of incoming packets to 1, preventing them from surviving the internal hop from my host’s main interface to the Docker bridge.
I will need to investigate the configuration of that network or simply avoid using Docker on it.
added: sudo iptables -t mangle -I PREROUTING -j TTL --ttl-inc 2 works in this network