TLS Handshake Issues in Docker Swarm with Overlay Network

Docker Engine General
,
1

I have a simple setup:

  • Leader Node, Worker Node and one overlay network (attachable)
  • Leader Node has a https-echo-server container running on port 8443
  • Worker Node has a client container

When I make a curl -v -k request to the https-echo-server (using private ip of container) from inside the client container I get TLS error:

*   Trying 10.0.0.20:8443...
* Connected to 10.0.0.20 (10.0.0.20) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to 10.0.0.20:8443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 10.0.0.20:8443

It looks like the server does not reply back to client.

If I do the same thing but using the Leader Node to run the client container, it works fine and TLS Handshake is successful. I can ping and traceroute the server/client containers.

Client: Docker Engine - Community
 Version:           23.0.6
 API version:       1.42
 Go version:        go1.19.9
 Git commit:        ef23cbc
 Built:             Fri May  5 21:18:28 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.6
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.9
  Git commit:       9dbdbd4
  Built:            Fri May  5 21:18:28 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
2

The fix was to reduce the MTU value for the overlay network with --opt com.docker.network.driver.mtu=1400. Interface on the leader node MTU value showed 1460 which is below the default 1500 by docker. All good now.

1 Like