Description
I have did a tcp connect test, start a tcp server in a host, and start other host run containers in containers connect to tcp server.
when total number of connections (sum all connectoin number of all containers) almost reach to 65535 (eg: 64512 just now), the other tcp connection throw exception “connection timeout”.
I tried to use 3 hosts without docker to connect tcp server each 60K connections is ok, it said that the tcp server do 180K connections is ok.
Steps to reproduce the issue:
- start tcp server in host A
- start container C1 in host B
- start 60K connection to tcp server in C1
- start container C2 in host B
- start 60K connection to tcp server in C2
Describe the results you received:
total tcp connection more less than 120K
Describe the results you expected:
each 60K connection in two container total 120K connection made.
Additional information you deem important (e.g. issue happens only occasionally):
host configurations:
in host
root@debian:~# ulimit -n
800000
net.ipv4.ip_local_port_range = 5000 65535
net.ipv4.tcp_mem = 8388608 12582912 16777216
net.ipv4.netfilter.ip_conntrack_max = 1000000
net.netfilter.nf_conntrack_max = 1000000
net.netfilter.nf_conntrack_count = 64515
net.nf_conntrack_max = 1000000
total used free shared buffers cached
Mem: 986M 808M 178M 8.5M 5.0M 132M
-/+ buffers/cache: 670M 316M
Swap: 1.0G 872K 1.0G
in container
ulimit -n
800000
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.netfilter.ip_conntrack_max = 1000000
net.netfilter.nf_conntrack_max = 1000000
net.netfilter.nf_conntrack_count = 6653
Output of docker version:
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:07:28 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:07:28 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 6
Running: 2
Paused: 0
Stopped: 4
Images: 15
Server Version: 17.03.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 27
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 986.9 MiB
Name: debian
ID: GEUG:DUKW:FDXQ:DFUP:GOI7:S4YO:GIH7:RX55:SQ6Z:ZZFQ:MEXB:2EHP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false