Completely newbie to docker
Have installed docker on a linux box and I have container run with bridge mode for long time could access outside network, but suddenly it doesn’t work anymore with bridge mode since 14th Feb. Not sure why is it. Now just using host mode to handle it.
Nothing was updated in the iptables / sysctl of host. Tried to follow different checking on internet, but still no clue.
In my iptables, the FORWARD flag is DROP not ACCEPT, I know Docker website said it needs to be ACCEPT, but the bridge mode did worked with DROP before 14th Feb. And this flag has not been touched since day 1 the box is setup.
Does anyone have any hint to check this problem?
System Details:
$ hostnamectl
Icon name: computer-vm
Chassis: vm
Virtualization: vmware
Operating System: Oracle Linux Server 7.9
CPE OS Name: cpe:/o:oracle:linux:7:9:server
Kernel: Linux 3.10.0-1160.53.1.el7.x86_64
Architecture: x86-64
$ docker version
Client:
Version: 18.03.1-ol
API version: 1.37
Go version: go1.9.4
Git commit: 0d51d18
Built: Wed Aug 22 21:59:42 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
$ sysctl -a | grep net.ipv4.ip_forward
sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
sysctl: permission denied on key 'kernel.cad_pid'
sysctl: permission denied on key 'kernel.usermodehelper.bset'
sysctl: permission denied on key 'kernel.usermodehelper.inheritable'
sysctl: permission denied on key 'net.core.bpf_jit_harden'
sysctl: permission denied on key 'net.core.bpf_jit_kallsyms'
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_use_pmtu = 0
sysctl: permission denied on key 'net.ipv4.tcp_fastopen_key'
sysctl: permission denied on key 'net.ipv6.conf.all.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.default.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.docker0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.eth0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.lo.stable_secret'
sysctl: permission denied on key 'vm.mmap_rnd_bits'
sysctl: permission denied on key 'vm.mmap_rnd_compat_bits'
sudo iptables -S FORWARD
-P FORWARD DROP
-A FORWARD -j ILO-FILTER-FORWARD
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-4df26a431f54 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-4df26a431f54 -j DOCKER
-A FORWARD -i br-4df26a431f54 ! -o br-4df26a431f54 -j ACCEPT
-A FORWARD -i br-4df26a431f54 -o br-4df26a431f54 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
I can ping the container ip from my box and ping the host ip from my container
docker network setup:
$ docker network inspect bridge
[
{
"Name": "bridge",
"Id": "89cab63242f17c649a44b02f9756831bb5cb0191af745cc6c6e69a029e31e635",
"Created": "2022-02-20T01:00:07.347908227+08:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16",
"Gateway": "172.17.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
Failure log: (Sorry I dont have log of successful case as I turned on the debug mode after the issue)
*xxx are value I masked
sshd[16251]: Connection closed by xx.xxx.xxx.xxx port 42558 [preauth]
sshd[16729]: Connection closed by xx.xxx.xxx.xxx port 58452 [preauth]
sshd[16900]: Connection closed by xx.xxx.xxx.xxx port 46616 [preauth]
kernel: [492911.772908] docker0: port 3(veth29137df) entered blocking state
kernel: [492911.772914] docker0: port 3(veth29137df) entered disabled state
kernel: [492911.772966] device: veth29137df entered promiscuous mode
kernel: [492911.773088] IPv6: ADDRCONF(NETDEV_UP): veth29137df: link is not ready
kernel: [492911.773092] docker0: port 3(veth29137df) entered blocking state
kernel: [492911.773095] docker0: port 3(veth29137df) entered forwarding state
kernel: [492911.780392] docker0: port 3(veth29137df) entered disabled state
kernel: [492912.049668] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
kernel: [492912.049691] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
kernel: [492912.049740] IPv6: ADDRCONF(NETDEV_CHANGE): veth29137df: link becomes ready
kernel: [492912.049740] docker0: port 3(veth29137df) entered blocking state
kernel: [492912.049740] docker0: port 3(veth29137df) entered forwarding state
sshd[17012]: Accepted publickey for root from xx.xxx.xx.xxx port 39133 ssh2: RSA SHA256:xxxxxxxxxxxxxxtheKeyxxxxxxxxxxxxx
kernel: [492923.292924] docker0: port 3(veth29137df) entered disabled state
kernel: [492923.295388] docker0: port 3(veth29137df) entered disabled state
kernel: [492923.304500] device: veth29137df left promiscuous mode
kernel: [492923.304527] docker0: port 3(veth29137df) entered disabled state
sshd[17128]: Connection closed by 10.132.241.153 port 36434 [preauth]