How can I trace Docker Swarm Encrypted outbound traffic?

How can I trace Docker Swarm Encrypted outbound traffic?

As part of working on this issue: Docker Swarm Overlay network - encrypted containers can not communicate

I need to learn how docker swarm is supposed to work so I can narrow down the search. I would like to be able to trace traffic from a container to the physical traffic leaving the node.

The setup is a docker swarm with two machines

  • one encrypted overlay network
  • one container on that network running nginx on NODE_2
  • one container also on the network running curl on NODE_1 to get to the service.

(In the unencrypted case this works and the encrypted one just times out with no response so I know I need to trace the encrypted path and find where it is broken)

My assumption is that I should see ESP traffic leaving NODE_1 heading to NODE_2
So running the command:

sudo tcpdump -p esp

on NODE_1 should see some output.

Firstly, is this assumption correct? (If it is then I have narrowed the issue to NODE_1)
Secondly, what are the components involved that this traffic should pass through on their way out of NODE_1 and how can I debug each one?

I think the path should be something like:

container -> ovelay network -> ipsec -> network interface

The reason I want to know this path is to help me debug the flow.
I have been trailing through /etc/syslog watching the messages that appear when the container starts up, as well as runs curl, I have been comparing what appears both on encrypted and unencrypted use cases but I can’t see any differences. Something is broken with the encryption part but I don’t know what or where!

The routing mesh enables each node in the swarm to accept connections on published ports for any service running in the swarm , even if there’s no task running on the node. The routing mesh routes all incoming requests to published ports on available nodes to an active container. upsers