I noticed on a swarm node running “tcpdump -ni any” I don’t see any of the traffic flowing between services (even between nodes). How are people following traffic between services?
I have been having the same problem. If you solved it, would you mind sharing?
where is the tcpdump done from? we can do tcpdump from individual containers in service, network namespace or from the node. In the node, we should see vxlan encap traffic and in container, we should see decapsulated traffic. In the node, we should do tcpdump on interface that is connecting the nodes.
I have a single-host swarm, how can I check that the traffic generated by two containers in an overlay network is encrypted? I tried to do the dump from the different network interfaces (docker0, docker_gwbridge) but I couldn’t see traffic, not even cleartext one. If you could help with this it would be very appreciated.
overlay network traffic in swarm does not go thru docker0 or docker_gwbridge.
There are 2 options:
option 1:
Go inside container and do tcpdump:
nicolaka/netshoot is container with all network debug tools.
docker run -ti --net container: <container name/id> nicolaka/netshoot
tcpdump -i <eth0>
Option 2:
Go inside network namespace of overlay network and do tcpdump:
First find overlay network id with docker network inspect
Start debug container mounting network namespace:
docker run -it --rm -v /var/run/docker/netns:/var/run/docker/netns --privileged=true nicolaka/netshoot
All namespaces are listed under:
/var/run/docker/netns
Find your swarm overlay network namespace matching with overlay networkid of previous command.
Then nsenter into network namespace:
nsenter --net=/var/run/docker/netns/ sh
tcpdump -i vxlan0
Thanks for the suggestions.
I tried the first option but still I can’t see the traffic flowing.
I have a wordpress service and a db service deployed on a overlay network used by the swarm.
I ran the nicolaka/netshoot container with --net container:
Then, from the db container I made a request to the wordpress service, got a request, so traffic going but still can’t see anything as a result of the tcpdump.
Is there anything wrong I’m doing?
Update:
I tried also the second option, but I didn’t find vxlan0 inside the container.
I tried to filter esp traffic with ‘tcpdump -i vxlan1 -p esp’ but I can’t see any traffic yet.
Inside the container I have br0, vxlan1 and some veth interfaces.
I’m going to try the second option now.
If you have other suggestions I would really appreciate that!
With option1, did you use the container id of db or wordpress container? If so, that is very weird how you dont see the traffic…
With option2, vxlanid itself might be different based on the environment.
I tried again, with a multi-host swarm setting (labs.play-with-docker.com).
I created 2 nodes and added that to a swarm cluster in an overlay network with encryption option enabled.
I created two containers and placed each one of them on a different node. Service_1 is an nginx container placed on node_1 and service_2 is a container in which I have installed curl to act as a client.
On one of the node I run tcpdump -i eth0 esp
I generated traffic with service_2 and I can see the esp packets.
If I place service_1 and service_2 on the same node and I run ‘tcpdump -i eth0’ in nicolaka/netshoot container attaching it to the network of one of the services, l can see traffic but it’s not encrypted.
Unless there is another interface on which I could see encrypted traffic before it gets to the containers, it seems that if containers are on the same host, traffic is not encrypted. I cannot find any definite answer about this.
Thank you for suggesting to use the netshoot container!