Issue: I can’t reach the container from a VM in the same subnet as the VM where docker is installed but it works from a VM from a different subnet.
os version:
Linux elk 5.4.0-77-generic #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
docker version:
Client:
Version: 20.10.2
API version: 1.40
Go version: go1.13.8
Git commit: 20.10.2-0ubuntu1~20.04.3
Built: Fri Jul 23 21:06:26 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.13.8
Git commit: afacb8b7f0
Built: Fri Dec 4 23:02:49 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.2-0ubuntu1~20.04.2
GitCommit:
runc:
Version: 1.0.0~rc95-0ubuntu1~20.04.2
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:
This is the diagram:
connectivity test from VM2 → VM3:
curl http://10.0.3.2:9200
{
"name" : "elk",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "ElvQdiCURke3peA7scMvYw",
"version" : {
"number" : "7.13.2",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "4d960a0733be83dd2543ca018aa4ddc42e956800",
"build_date" : "2021-06-10T21:01:55.251515791Z",
"build_snapshot" : false,
"lucene_version" : "8.8.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
connectivity test from VM1 → VM3:
ping 10.0.3.2
PING 10.0.3.2 (10.0.3.2) 56(84) bytes of data.
64 bytes from 10.0.3.2: icmp_seq=1 ttl=62 time=6.98 ms
64 bytes from 10.0.3.2: icmp_seq=2 ttl=62 time=8.04 ms
64 bytes from 10.0.3.2: icmp_seq=3 ttl=62 time=6.61 ms
64 bytes from 10.0.3.2: icmp_seq=4 ttl=62 time=13.7 ms
^C
--- 10.0.3.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 6.612/8.821/13.655/2.839 ms
curl http://10.0.3.2:9200
curl: (28) Failed to connect to 10.0.3.2 port 9200: Connection timed out
I did confirm that traffic from VM1 [10.0.3.3] reaches ens19 [10.0.3.2] but it’s not seen on docker0
iptables trace shows these:
When it works:
Jul 29 22:12:21 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: nat:PREROUTING:rule:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:12:21 elk kernel: TRACE: nat:DOCKER:rule:2 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-USER:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:4 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER:rule:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=docker0 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: nat:POSTROUTING:policy:5 IN= OUT=docker0 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307)
Jul 29 22:12:21 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=129 TOS=0x00 PREC=0x00 TTL=62 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=129 TOS=0x00 PREC=0x00 TTL=62 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=52 TOS=0x00 PREC=0x00 TTL=62 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:12:21 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=52 TOS=0x00 PREC=0x00 TTL=62 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-USER:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:3 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=docker0 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-USER:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:3 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB)
Jul 29 22:12:21 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=docker0 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB)
When it doesn’t work:
Jul 29 22:09:50 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
Jul 29 22:09:50 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
Jul 29 22:09:50 elk kernel: TRACE: nat:PREROUTING:rule:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:09:50 elk kernel: TRACE: nat:DOCKER:rule:2 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
Jul 29 22:09:50 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:09:50 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
Jul 29 22:09:50 elk kernel: TRACE: filter:DOCKER-USER:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
Jul 29 22:09:50 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
Jul 29 22:09:50 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=ens19 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
Jul 29 22:09:50 elk kernel: TRACE: nat:POSTROUTING:policy:5 IN= OUT=ens19 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
Jul 29 22:09:51 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307)
Jul 29 22:09:51 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307)
Jul 29 22:09:51 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307)
Jul 29 22:09:51 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307)
Jul 29 22:09:51 elk kernel: TRACE: filter:DOCKER-USER:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307)
Jul 29 22:09:51 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307)
Jul 29 22:09:51 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=ens19 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307)
Jul 29 22:09:53 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307)
Jul 29 22:09:53 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307)
Jul 29 22:09:53 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307)
Jul 29 22:09:53 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307)
Jul 29 22:09:53 elk kernel: TRACE: filter:DOCKER-USER:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307)
Jul 29 22:09:53 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307)
Jul 29 22:09:53 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=ens19 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307)
This the IP configuration of VM3 :
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether ee:95:39:76:c7:f9 brd ff:ff:ff:ff:ff:ff
inet 10.0.3.2/24 brd 10.0.3.255 scope global ens19
valid_lft forever preferred_lft forever
inet6 fe80::ec95:39ff:fe76:c7f9/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:29:0f:2c:8b brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:29ff:fe0f:2c8b/64 scope link
valid_lft forever preferred_lft forever
4: veth1650f38@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 3e:dc:8c:f7:6b:13 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::3cdc:8cff:fef7:6b13/64 scope link
valid_lft forever preferred_lft forever
docker bridge details:
{
"Name": "bridge",
"Id": "cd2d8dde2f9c11980926f53175b738696d792db1097c2db8697e7a438ab4eee2",
"Created": "2021-07-21T03:00:01.289033231Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16",
"Gateway": "172.17.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"db9f7e1f765b7b05fd967d722ac05c41551aa2c42fa224b3bd4270f1667e9c59": {
"Name": "sam_elk_1",
"EndpointID": "0db88ed751b47ac6c214f5423da8769be8b496779bb685695bb97b2cb407eae3",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
the routes are:
default via 10.0.3.1 dev ens19 proto static
10.0.3.0/24 dev ens19 proto kernel scope link src 10.0.3.2
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
the content of iptables is
################################################################################
iptables:filter table:
################################################################################
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain FORWARD (policy DROP)
num target prot opt source destination
1 DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
2 DOCKER-ISOLATION-STAGE-1 all -- 0.0.0.0/0 0.0.0.0/0
3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
4 DOCKER all -- 0.0.0.0/0 0.0.0.0/0
5 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
6 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Chain DOCKER (1 references)
num target prot opt source destination
1 ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:9200
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num target prot opt source destination
1 DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0
2 RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
num target prot opt source destination
1 DROP all -- 0.0.0.0/0 0.0.0.0/0
2 RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
num target prot opt source destination
1 ACCEPT tcp -- 10.0.3.0/24 0.0.0.0/0 tcp dpt:9200
2 RETURN all -- 0.0.0.0/0 0.0.0.0/0
################################################################################
iptables:nat table:
################################################################################
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
2 MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:9200
Chain DOCKER (2 references)
num target prot opt source destination
1 RETURN all -- 0.0.0.0/0 0.0.0.0/0
2 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9200 to:172.17.0.2:9200
################################################################################
iptables:mangle table:
################################################################################
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
################################################################################
iptables:raw table:
################################################################################
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 TRACE tcp -- 0.0.0.0/0 10.0.3.2 tcp dpt:9200
2 TRACE tcp -- 10.0.3.2 0.0.0.0/0 tcp spt:9200
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 TRACE tcp -- 0.0.0.0/0 10.0.3.2 tcp dpt:9200
2 TRACE tcp -- 10.0.3.2 0.0.0.0/0 tcp spt:9200
################################################################################
iptables:security table:
################################################################################
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
I suspect it has something related to the fact that the traffic is never sent to docker0 but I don’t see why:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:09:50 elk kernel: TRACE: nat:DOCKER:rule:2 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
Jul 29 22:09:50 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
