I am running statsd inside of a docker container and exposing port 8125/udp on to the host. On the client side I am using Drupal’s statsd module to send out metrics to the host running statsd container.
Following are the details of environment -
OS: Ubuntu 14.04.4 LTS
Kernel: 3.13.0-92-generic
Docker Version: docker-engine, 17.03.1~ce-0~ubuntu-trusty
The entire setup is working just fine and I can see metrics coming to statsd from the client. However few particular metrics are just getting dropped on host.
Following is tcpdump on client -
14:25:45.046853 IP 10.0.13.114.35037 > 10.0.13.43.8125: UDP, length 40
0x0000: 4500 0044 445d 4000 4011 c7af 0a00 0d72 E…DD]@.@…r
0x0010: 0a00 0d2b 88dd 1fbd 0030 2ede 7379 6e64 …+…0…synd
0x0020: 7569 742e 7761 7463 6864 6f67 2e74 7970 uit.watchdog.typ
0x0030: 652e 7061 6765 206e 6f74 2066 6f75 6e64 e.page.not.found
0x0040: 3a31 7c63 :1|c
14:25:45.046866 IP 10.0.13.114.35037 > 10.0.13.43.8125: UDP, length 37
0x0000: 4500 0041 445e 4000 4011 c7b1 0a00 0d72 E…AD^@.@…r
0x0010: 0a00 0d2b 88dd 1fbd 002d 2edb 7379 6e64 …+…-…synd
0x0020: 7569 742e 7761 7463 6864 6f67 2e73 6576 uit.watchdog.sev
0x0030: 6572 6974 792e 7761 726e 696e 673a 317c erity.warning:1|
0x0040: 63 c
Following is tcpdump on host running statsd container -
14:25:45.043425 IP 10.0.13.114.35037 > 10.0.13.43.8125: UDP, length 40
0x0000: 4500 0044 445d 4000 4011 c7af 0a00 0d72 E…DD]@.@…r
0x0010: 0a00 0d2b 88dd 1fbd 0030 6fbf 7379 6e64 …+…0o.synd
0x0020: 7569 742e 7761 7463 6864 6f67 2e74 7970 uit.watchdog.typ
0x0030: 652e 7061 6765 206e 6f74 2066 6f75 6e64 e.page.not.found
0x0040: 3a31 7c63 :1|c
14:25:45.043425 IP 10.0.13.114.35037 > 10.0.13.43.8125: UDP, length 37
0x0000: 4500 0041 445e 4000 4011 c7b1 0a00 0d72 E…AD^@.@…r
0x0010: 0a00 0d2b 88dd 1fbd 002d 6e62 7379 6e64 …+…-nbsynd
0x0020: 7569 742e 7761 7463 6864 6f67 2e73 6576 uit.watchdog.sev
0x0030: 6572 6974 792e 7761 726e 696e 673a 317c erity.warning:1|
0x0040: 63 c
Following tcpdump on the container -
14:25:45.043444 IP 10.0.13.114.35037 > 10.42.43.133.8125: UDP, length 40
0x0000: 4500 0044 445d 4000 3f11 aa2b 0a00 0d72 E…DD]@.?..+…r
0x0010: 0a2a 2b85 88dd 1fbd 0030 513b 7379 6e64 .*+…0Q;synd
0x0020: 7569 742e 7761 7463 6864 6f67 2e74 7970 uit.watchdog.typ
0x0030: 652e 7061 6765 206e 6f74 2066 6f75 6e64 e.page.not.found
0x0040: 3a31 7c63 :1|c
I’ve also enabled logging on iptables through this rule -
iptables -A FORWARD -m limit --limit 50/minute -j LOG --log-level 7 --log-prefix "Dropped fw: "
And I can see following entries in syslog -
Jun 20 14:25:44 ip-10-0-13-43 kernel: [3052186.469395] Dropped fw: IN=eth0 OUT=docker0 MAC=02:b5:48:73:3a:00:02:e2:9a:4f:0c:f3:08:00 SRC=10.0.13.33 DST=10.42.43.133 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=17559 DF PROTO=UDP SPT=45671 DPT=8125 LEN=40
I am confused by two things here -
- Where did the following packet go? -
14:25:45.043425 IP 10.0.13.114.35037 > 10.0.13.43.8125: UDP, length 37
0x0000: 4500 0041 445e 4000 4011 c7b1 0a00 0d72 E…AD^@.@…r
0x0010: 0a00 0d2b 88dd 1fbd 002d 6e62 7379 6e64 …+…-nbsynd
0x0020: 7569 742e 7761 7463 6864 6f67 2e73 6576 uit.watchdog.sev
0x0030: 6572 6974 792e 7761 726e 696e 673a 317c erity.warning:1|
0x0040: 63 c
- From what I see in syslog, the packet with LEN=40 should be lost but this is the packet that I actually got on the container and packet with LEN=37 should’ve been reported as lost.
Please help me understand what’s happening here.
Thanks,
Mayank