How can I trace Docker Swarm Encrypted outbound traffic?
As part of working on this issue: Docker Swarm Overlay network - encrypted containers can not communicate
I need to learn how docker swarm is supposed to work so I can narrow down the search. I would like to be able to trace traffic from a container to the physical traffic leaving the node.
The setup is a docker swarm with two machines
- one encrypted overlay network
- one container on that network running nginx on NODE_2
- one container also on the network running curl on NODE_1 to get to the service.
(In the unencrypted case this works and the encrypted one just times out with no response so I know I need to trace the encrypted path and find where it is broken)
My assumption is that I should see ESP traffic leaving NODE_1 heading to NODE_2
So running the command:
sudo tcpdump -p esp
on NODE_1 should see some output.
Firstly, is this assumption correct? (If it is then I have narrowed the issue to NODE_1)
Secondly, what are the components involved that this traffic should pass through on their way out of NODE_1 and how can I debug each one?
I think the path should be something like:
container -> ovelay network -> ipsec -> network interface
The reason I want to know this path is to help me debug the flow.
I have been trailing through /etc/syslog watching the messages that appear when the container starts up, as well as runs curl, I have been comparing what appears both on encrypted and unencrypted use cases but I can’t see any differences. Something is broken with the encryption part but I don’t know what or where!