Docker Community Forums

Share and learn in the Docker community.

Docker dropping traffic from a vm in the same subnet

Issue: I can’t reach the container from a VM in the same subnet as the VM where docker is installed but it works from a VM from a different subnet.

os version:

Linux elk 5.4.0-77-generic #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

docker version:

Client:
 Version:           20.10.2
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        20.10.2-0ubuntu1~20.04.3
 Built:             Fri Jul 23 21:06:26 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Fri Dec  4 23:02:49 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.2-0ubuntu1~20.04.2
  GitCommit:        
 runc:
  Version:          1.0.0~rc95-0ubuntu1~20.04.2
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:

This is the diagram:

connectivity test from VM2 → VM3:

curl http://10.0.3.2:9200
{
 "name" : "elk",
 "cluster_name" : "elasticsearch",
 "cluster_uuid" : "ElvQdiCURke3peA7scMvYw",
 "version" : {
   "number" : "7.13.2",
   "build_flavor" : "default",
   "build_type" : "tar",
   "build_hash" : "4d960a0733be83dd2543ca018aa4ddc42e956800",
   "build_date" : "2021-06-10T21:01:55.251515791Z",
   "build_snapshot" : false,
   "lucene_version" : "8.8.2",
   "minimum_wire_compatibility_version" : "6.8.0",
   "minimum_index_compatibility_version" : "6.0.0-beta1"
 },
 "tagline" : "You Know, for Search"
}

connectivity test from VM1 → VM3:

ping 10.0.3.2
PING 10.0.3.2 (10.0.3.2) 56(84) bytes of data.
64 bytes from 10.0.3.2: icmp_seq=1 ttl=62 time=6.98 ms
64 bytes from 10.0.3.2: icmp_seq=2 ttl=62 time=8.04 ms
64 bytes from 10.0.3.2: icmp_seq=3 ttl=62 time=6.61 ms
64 bytes from 10.0.3.2: icmp_seq=4 ttl=62 time=13.7 ms
^C
--- 10.0.3.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 6.612/8.821/13.655/2.839 ms

curl http://10.0.3.2:9200
curl: (28) Failed to connect to 10.0.3.2 port 9200: Connection timed out

I did confirm that traffic from VM1 [10.0.3.3] reaches ens19 [10.0.3.2] but it’s not seen on docker0

iptables trace shows these:

When it works:

Jul 29 22:12:21 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
Jul 29 22:12:21 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
Jul 29 22:12:21 elk kernel: TRACE: nat:PREROUTING:rule:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:12:21 elk kernel: TRACE: nat:DOCKER:rule:2 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
Jul 29 22:12:21 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-USER:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:4 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER:rule:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
Jul 29 22:12:21 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
Jul 29 22:12:21 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=docker0 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 
Jul 29 22:12:21 elk kernel: TRACE: nat:POSTROUTING:policy:5 IN= OUT=docker0 SRC=10.0.4.35 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=48299 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993933 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AD52228100000000001030307) 

Jul 29 22:12:21 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=129 TOS=0x00 PREC=0x00 TTL=62 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=129 TOS=0x00 PREC=0x00 TTL=62 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=52 TOS=0x00 PREC=0x00 TTL=62 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB) 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:12:21 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=10.0.3.2 LEN=52 TOS=0x00 PREC=0x00 TTL=62 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB) 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-USER:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:3 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=docker0 SRC=10.0.4.35 DST=172.17.0.2 LEN=129 TOS=0x00 PREC=0x00 TTL=61 ID=48301 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-USER:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: filter:FORWARD:rule:3 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=docker0 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB) 
Jul 29 22:12:21 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=docker0 SRC=10.0.4.35 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=61 ID=48300 DF PROTO=TCP SPT=40114 DPT=9200 SEQ=4278993934 ACK=696567754 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080AD522281FD7D3F4AB) 

When it doesn’t work:

Jul 29 22:09:50 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 
Jul 29 22:09:50 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 
Jul 29 22:09:50 elk kernel: TRACE: nat:PREROUTING:rule:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:09:50 elk kernel: TRACE: nat:DOCKER:rule:2 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 
Jul 29 22:09:50 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:09:50 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 
Jul 29 22:09:50 elk kernel: TRACE: filter:DOCKER-USER:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 
Jul 29 22:09:50 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 
Jul 29 22:09:50 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=ens19 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 
Jul 29 22:09:50 elk kernel: TRACE: nat:POSTROUTING:policy:5 IN= OUT=ens19 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 

Jul 29 22:09:51 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307) 
Jul 29 22:09:51 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307) 
Jul 29 22:09:51 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307) 
Jul 29 22:09:51 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307) 
Jul 29 22:09:51 elk kernel: TRACE: filter:DOCKER-USER:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307) 
Jul 29 22:09:51 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307) 
Jul 29 22:09:51 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=ens19 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15698 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67582180000000001030307) 

Jul 29 22:09:53 elk kernel: TRACE: raw:PREROUTING:policy:3 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307) 
Jul 29 22:09:53 elk kernel: TRACE: mangle:PREROUTING:policy:1 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307) 
Jul 29 22:09:53 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307) 
Jul 29 22:09:53 elk kernel: TRACE: filter:FORWARD:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307) 
Jul 29 22:09:53 elk kernel: TRACE: filter:DOCKER-USER:rule:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307) 
Jul 29 22:09:53 elk kernel: TRACE: security:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:b6:84:98:5c:a0:60:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307) 
Jul 29 22:09:53 elk kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=ens19 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15699 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB67589F80000000001030307) 

This the IP configuration of VM3 :

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
   inet6 ::1/128 scope host 
      valid_lft forever preferred_lft forever
2: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
   link/ether ee:95:39:76:c7:f9 brd ff:ff:ff:ff:ff:ff
   inet 10.0.3.2/24 brd 10.0.3.255 scope global ens19
      valid_lft forever preferred_lft forever
   inet6 fe80::ec95:39ff:fe76:c7f9/64 scope link 
      valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
   link/ether 02:42:29:0f:2c:8b brd ff:ff:ff:ff:ff:ff
   inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
      valid_lft forever preferred_lft forever
   inet6 fe80::42:29ff:fe0f:2c8b/64 scope link 
      valid_lft forever preferred_lft forever
4: veth1650f38@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default 
   link/ether 3e:dc:8c:f7:6b:13 brd ff:ff:ff:ff:ff:ff link-netnsid 0
   inet6 fe80::3cdc:8cff:fef7:6b13/64 scope link 
      valid_lft forever preferred_lft forever

docker bridge details:


    {
        "Name": "bridge",
        "Id": "cd2d8dde2f9c11980926f53175b738696d792db1097c2db8697e7a438ab4eee2",
        "Created": "2021-07-21T03:00:01.289033231Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "db9f7e1f765b7b05fd967d722ac05c41551aa2c42fa224b3bd4270f1667e9c59": {
                "Name": "sam_elk_1",
                "EndpointID": "0db88ed751b47ac6c214f5423da8769be8b496779bb685695bb97b2cb407eae3",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }

the routes are:

default via 10.0.3.1 dev ens19 proto static 
10.0.3.0/24 dev ens19 proto kernel scope link src 10.0.3.2 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 

the content of iptables is

################################################################################
iptables:filter table:
################################################################################
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain FORWARD (policy DROP)
num  target     prot opt source               destination         
1    DOCKER-USER  all  --  0.0.0.0/0            0.0.0.0/0           
2    DOCKER-ISOLATION-STAGE-1  all  --  0.0.0.0/0            0.0.0.0/0           
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
4    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0           
5    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
6    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain DOCKER (1 references)
num  target     prot opt source               destination         
1    ACCEPT     tcp  --  0.0.0.0/0            172.17.0.2           tcp dpt:9200


Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num  target     prot opt source               destination         
1    DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0           
2    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
num  target     prot opt source               destination         
1    DROP       all  --  0.0.0.0/0            0.0.0.0/0           
2    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
num  target     prot opt source               destination         
1    ACCEPT     tcp  --  10.0.3.0/24          0.0.0.0/0            tcp dpt:9200
2    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
################################################################################
iptables:nat table:
################################################################################
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
2    MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:9200


Chain DOCKER (2 references)
num  target     prot opt source               destination         
1    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:9200 to:172.17.0.2:9200
################################################################################
iptables:mangle table:
################################################################################
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
################################################################################
iptables:raw table:
################################################################################
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    TRACE      tcp  --  0.0.0.0/0            10.0.3.2             tcp dpt:9200
2    TRACE      tcp  --  10.0.3.2             0.0.0.0/0            tcp spt:9200

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    TRACE      tcp  --  0.0.0.0/0            10.0.3.2             tcp dpt:9200
2    TRACE      tcp  --  10.0.3.2             0.0.0.0/0            tcp spt:9200
################################################################################
iptables:security table:
################################################################################
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination   

I suspect it has something related to the fact that the traffic is never sent to docker0 but I don’t see why:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Jul 29 22:09:50 elk kernel: TRACE: nat:DOCKER:rule:2 IN=ens19 OUT= MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=10.0.3.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 
Jul 29 22:09:50 elk kernel: TRACE: mangle:FORWARD:policy:1 IN=ens19 OUT=ens19 MAC=ee:95:39:76:c7:f9:44:d3:ca:5c:54:f4:08:00 SRC=10.0.3.3 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=15697 DF PROTO=TCP SPT=34200 DPT=9200 SEQ=2207217953 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AB6757E150000000001030307) 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!