Docker Community Forums

Share and learn in the Docker community.

Running a lot of containers -> pthread_create failed

Hello everyone,

i am trying to build a server with multiple services running on docker (jira, gitlab, owncloud, mailcow).
I have built docker compose setups for all these services and it worked fine until, i started the 4th service (either jira or gitlab). When i do so the containers start up and a few seconds later i hit some thread limit i guess.
Nothing on my system is able to start any threads. For example my zsh autocompletion starts to fail while i type.

When i run docker ps i get this error:

runtime/cgo: pthread_create failed: Resource temporarily unavailable

My Task Limit is okay:

Tasks: 164 (limit: 512)

I would love to provide any logs and information you need but i am relatively new to docker and you have to tell me what you need.

Big thanks in advance to everyone who tries to help me out!

docker info:

Containers: 23
Running: 23
Paused: 0
Stopped: 0
Images: 83
Server Version: 18.09.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 32GiB
Name: h2667679.stratoserver.net
ID: 6X5W:R4S6:LL5U:M6VV:UKFY:M2VO:XTUP:WEUD:FNUV:IW4N:4HOR:P3TL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

System:

Debian 9 
Linux 4.9.0

Stack Trace:

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f62d7380fff m=4 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f62d7380fff
stack: frame={sp:0x7f62d634a978, fp:0x0} stack=[0x7f62d5b4b2f0,0x7f62d634aef0)
00007f62d634a878:  000000c4201aefc0  3fd3333333333333
00007f62d634a888:  0000000000000000  3fd0000000000000
00007f62d634a898:  0000000000000000  3fedaf1f95df65ca
00007f62d634a8a8:  0000000000000000  3fab3fb514b91010
00007f62d634a8b8:  0000000000000000  3fc0136eedb0acdc
00007f62d634a8c8:  0000000000000000  000000c4201bb650
00007f62d634a8d8:  000000c4201bb890  000000c4202d2030
00007f62d634a8e8:  000000c4201ae810  000000c4201aefc0
00007f62d634a8f8:  000000c4201ba690  0000000000000000
00007f62d634a908:  000000c4202d2450  000000c4202d2780
00007f62d634a918:  000000c420283440  0000000000000000
00007f62d634a928:  0000000000000000  0000000000000000
00007f62d634a938:  0000000000000000  0000000000000000
00007f62d634a948:  0000000000000000  0000000000000000
00007f62d634a958:  0000000000000000  0000000000000000
00007f62d634a968:  0000000000000000  0000000000000000
00007f62d634a978: <0000000000000000  fffffffe7fffffff
00007f62d634a988:  ffffffffffffffff  ffffffffffffffff
00007f62d634a998:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9a8:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9b8:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9c8:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9d8:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9e8:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9f8:  ffffffffffffffff  00007f62d76e86e0
00007f62d634aa08:  00007f62d738242a  0000000000000020
00007f62d634aa18:  0000000000000000  0000000000000000
00007f62d634aa28:  0000000000000000  0000000000000000
00007f62d634aa38:  0000000000000000  0000000000000000
00007f62d634aa48:  0000000000000000  0000000000000000
00007f62d634aa58:  0000000000000000  0000000000000000
00007f62d634aa68:  0000000000000000  0000000000000000
runtime: unknown pc 0x7f62d7380fff
stack: frame={sp:0x7f62d634a978, fp:0x0} stack=[0x7f62d5b4b2f0,0x7f62d634aef0)
00007f62d634a878:  000000c4201aefc0  3fd3333333333333
00007f62d634a888:  0000000000000000  3fd0000000000000
00007f62d634a898:  0000000000000000  3fedaf1f95df65ca
00007f62d634a8a8:  0000000000000000  3fab3fb514b91010
00007f62d634a8b8:  0000000000000000  3fc0136eedb0acdc
00007f62d634a8c8:  0000000000000000  000000c4201bb650
00007f62d634a8d8:  000000c4201bb890  000000c4202d2030
00007f62d634a8e8:  000000c4201ae810  000000c4201aefc0
00007f62d634a8f8:  000000c4201ba690  0000000000000000
00007f62d634a908:  000000c4202d2450  000000c4202d2780
00007f62d634a918:  000000c420283440  0000000000000000
00007f62d634a928:  0000000000000000  0000000000000000
00007f62d634a938:  0000000000000000  0000000000000000
00007f62d634a948:  0000000000000000  0000000000000000
00007f62d634a958:  0000000000000000  0000000000000000
00007f62d634a968:  0000000000000000  0000000000000000
00007f62d634a978: <0000000000000000  fffffffe7fffffff
00007f62d634a988:  ffffffffffffffff  ffffffffffffffff
00007f62d634a998:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9a8:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9b8:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9c8:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9d8:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9e8:  ffffffffffffffff  ffffffffffffffff
00007f62d634a9f8:  ffffffffffffffff  00007f62d76e86e0
00007f62d634aa08:  00007f62d738242a  0000000000000020
00007f62d634aa18:  0000000000000000  0000000000000000
00007f62d634aa28:  0000000000000000  0000000000000000
00007f62d634aa38:  0000000000000000  0000000000000000
00007f62d634aa48:  0000000000000000  0000000000000000
00007f62d634aa58:  0000000000000000  0000000000000000
00007f62d634aa68:  0000000000000000  0000000000000000

goroutine 34 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:363 fp=0xc4204a6550 sp=0xc4204a6548 pc=0x559fd4418820
runtime.gcMarkTermination(0x3fed47e96b1518a4)
/usr/local/go/src/runtime/mgc.go:1647 +0x40d fp=0xc4204a6720 sp=0xc4204a6550 pc=0x559fd43dcacd
runtime.gcMarkDone()
/usr/local/go/src/runtime/mgc.go:1513 +0x230 fp=0xc4204a6748 sp=0xc4204a6720 pc=0x559fd43dc660
runtime.gcBgMarkWorker(0xc42004aa00)
/usr/local/go/src/runtime/mgc.go:1912 +0x2f1 fp=0xc4204a67d8 sp=0xc4204a6748 pc=0x559fd43dd5e1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4204a67e0 sp=0xc4204a67d8 pc=0x559fd441b3e1
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1723 +0x7b

goroutine 1 [semacquire]:
text/tabwriter.(*Writer).terminateCell(...)
/usr/local/go/src/text/tabwriter/tabwriter.go:447
text/tabwriter.(*Writer).Write(0xc420680000, 0xc420668980, 0x35, 0x39, 0x2a, 0x0, 0x0)
/usr/local/go/src/text/tabwriter/tabwriter.go:508 +0x515
bytes.(*Buffer).WriteTo(0xc420196380, 0x559fd60c05a0, 0xc420680000, 0x559fd5e9ccc0, 0xc4205f03c0, 0x0)
/usr/local/go/src/bytes/buffer.go:240 +0xb8
github.com/docker/cli/cli/command/formatter.(*Context).postFormat(0xc4205fe050, 0xc4205f4180, 0x559fd60bdee0, 0xc420602300)
/go/src/github.com/docker/cli/cli/command/formatter/formatter.go:77 +0x1bb
github.com/docker/cli/cli/command/formatter.(*Context).Write(0xc4205fe050, 0x559fd60bdee0, 0xc420602300, 0xc4205f0390, 0xc4204301e8, 0x0)
/go/src/github.com/docker/cli/cli/command/formatter/formatter.go:117 +0x18e
github.com/docker/cli/cli/command/formatter.ContainerWrite(0x559fd60bde40, 0xc42002d1d0, 0x559fd572a444, 0x57, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/docker/cli/cli/command/formatter/container.go:74 +0x59e
github.com/docker/cli/cli/command/container.runPs(0x559fd60ff420, 0xc420397b00, 0xc4205ab1d0, 0x0, 0x0)
/go/src/github.com/docker/cli/cli/command/container/list.go:139 +0x273
github.com/docker/cli/cli/command/container.NewPsCommand.func1(0xc4205bd400, 0x559fd6be1c20, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/docker/cli/cli/command/container/list.go:36 +0x3e
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).execute(0xc4205bd400, 0xc42000c070, 0x0, 0x0, 0xc4205bd400, 0xc42000c070)
/go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:762 +0x46a
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420102a00, 0xc420029fa0, 0x559fd5db8fa0, 0xc420029fb0)
/go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:852 +0x30c
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420102a00, 0xc420102a00, 0x559fd60c0360)
/go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:800 +0x2d
main.main()
/go/src/github.com/docker/cli/cmd/docker/docker.go:180 +0xde

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:139 +0xa8
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x24
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:28 +0x43

goroutine 37 [chan receive]:
github.com/docker/cli/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x559fd6bc32e0)
/go/src/github.com/docker/cli/vendor/github.com/golang/glog/glog.go:882 +0x8d
created by github.com/docker/cli/vendor/github.com/golang/glog.init.0
/go/src/github.com/docker/cli/vendor/github.com/golang/glog/glog.go:410 +0x205

goroutine 12 [IO wait]:
internal/poll.runtime_pollWait(0x7f62d7c8bf00, 0x72, 0xc4205e79a8)
/usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4205d4098, 0x72, 0xffffffffffffff00, 0x559fd60c26a0, 0x559fd6b55aa8)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4205d4098, 0xc4205af000, 0x1000, 0x1000)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4205d4080, 0xc4205af000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4205d4080, 0xc4205af000, 0x1000, 0x1000, 0x559fd4417260, 0xc420000180, 0x4)
/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc420388848, 0xc4205af000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:176 +0x6c
net/http.(*persistConn).Read(0xc420257560, 0xc4205af000, 0x1000, 0x1000, 0xc4205e7b98, 0x559fd43c7435, 0xc4205e8000)
/usr/local/go/src/net/http/transport.go:1453 +0x138
bufio.(*Reader).fill(0xc42046ec00)
/usr/local/go/src/bufio/bufio.go:100 +0x120
bufio.(*Reader).Peek(0xc42046ec00, 0x1, 0x0, 0x0, 0x0, 0xc4205e80c0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0x3c
net/http.(*persistConn).readLoop(0xc420257560)
/usr/local/go/src/net/http/transport.go:1601 +0x187
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1237 +0x95c

goroutine 13 [select]:
net/http.(*persistConn).writeLoop(0xc420257560)
/usr/local/go/src/net/http/transport.go:1822 +0x14d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1238 +0x981

rax    0x0
rbx    0x6
rcx    0xffffffffffffffff
rdx    0x0
rdi    0x2
rsi    0x7f62d634a900
rbp    0x559fd58771ea
rsp    0x7f62d634a978
r8     0x0
r9     0x7f62d634a900
r10    0x8
r11    0x246
r12    0x7f62c80008c0
r13    0xf1
r14    0x11
r15    0x0
rip    0x7f62d7380fff
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

has anyone any ideas?

Hello @user010101,

were you able to fix the issue?
I am asking because I just got the same error message while running ~400 containers.

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f6132f20e97 m=4 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f6132f20e97
stack: frame={sp:0x7f612fccd830, fp:0x0} stack=[0x7f612f4ce288,0x7f612fccde88)
00007f612fccd730:  00007f61332ce560  00007f61332d1bc0 
00007f612fccd740:  0000007e0000007d  000000800000007f 
00007f612fccd750:  00007f6118000010  00007f6118000010 
00007f612fccd760:  000000000000000e  000000000000000e 
00007f612fccd770:  0000000000000000  0000000000000000 
00007f612fccd780:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd790:  0000000400000004  0000000400000004 
00007f612fccd7a0:  00007f6118000800  00007f6118000800 
00007f612fccd7b0:  33706d2e2a3a3633  2a3a36333b30303d 
00007f612fccd7c0:  3b30303d63706d2e  67676f2e2a3a3633 
00007f612fccd7d0:  2a3a36333b30303d  333b30303d61722e 
00007f612fccd7e0:  3d7661772e2a3a36  2e2a3a36333b3030 
00007f612fccd7f0:  333b30303d61676f  7375706f2e2a3a36 
00007f612fccd800:  2a3a36333b30303d  3b30303d7870732e 
00007f612fccd810:  7073782e2a3a3633  3a36333b30303d66 
00007f612fccd820:  0000000000000000  0000000000000000 
00007f612fccd830: <0000000000000000  0000000000000000 
00007f612fccd840:  0000000000000000  0000000000000000 
00007f612fccd850:  0000000000000000  0000000000000000 
00007f612fccd860:  0000000000000000  0000000000000000 
00007f612fccd870:  0000000000000000  0000000000000000 
00007f612fccd880:  0000000000000002  800000000000000e 
00007f612fccd890:  0000000000000000  0000000000000000 
00007f612fccd8a0:  0000000000000000  0000000000000000 
00007f612fccd8b0:  fffffffe7fffffff  ffffffffffffffff 
00007f612fccd8c0:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd8d0:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd8e0:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd8f0:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd900:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd910:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd920:  ffffffffffffffff  ffffffffffffffff 
runtime: unknown pc 0x7f6132f20e97
stack: frame={sp:0x7f612fccd830, fp:0x0} stack=[0x7f612f4ce288,0x7f612fccde88)
00007f612fccd730:  00007f61332ce560  00007f61332d1bc0 
00007f612fccd740:  0000007e0000007d  000000800000007f 
00007f612fccd750:  00007f6118000010  00007f6118000010 
00007f612fccd760:  000000000000000e  000000000000000e 
00007f612fccd770:  0000000000000000  0000000000000000 
00007f612fccd780:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd790:  0000000400000004  0000000400000004 
00007f612fccd7a0:  00007f6118000800  00007f6118000800 
00007f612fccd7b0:  33706d2e2a3a3633  2a3a36333b30303d 
00007f612fccd7c0:  3b30303d63706d2e  67676f2e2a3a3633 
00007f612fccd7d0:  2a3a36333b30303d  333b30303d61722e 
00007f612fccd7e0:  3d7661772e2a3a36  2e2a3a36333b3030 
00007f612fccd7f0:  333b30303d61676f  7375706f2e2a3a36 
00007f612fccd800:  2a3a36333b30303d  3b30303d7870732e 
00007f612fccd810:  7073782e2a3a3633  3a36333b30303d66 
00007f612fccd820:  0000000000000000  0000000000000000 
00007f612fccd830: <0000000000000000  0000000000000000 
00007f612fccd840:  0000000000000000  0000000000000000 
00007f612fccd850:  0000000000000000  0000000000000000 
00007f612fccd860:  0000000000000000  0000000000000000 
00007f612fccd870:  0000000000000000  0000000000000000 
00007f612fccd880:  0000000000000002  800000000000000e 
00007f612fccd890:  0000000000000000  0000000000000000 
00007f612fccd8a0:  0000000000000000  0000000000000000 
00007f612fccd8b0:  fffffffe7fffffff  ffffffffffffffff 
00007f612fccd8c0:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd8d0:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd8e0:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd8f0:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd900:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd910:  ffffffffffffffff  ffffffffffffffff 
00007f612fccd920:  ffffffffffffffff  ffffffffffffffff

goroutine 1 [syscall, locked to thread]:
github.com/docker/cli/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.init.ializers()
	/go/src/github.com/docker/cli/vendor/k8s.io/apimachinery/pkg/apis/meta/v1        /types_swagger_doc_generated.go:179 +0x12cb fp=0xc00021bf08 sp=0xc00021bdc8 pc=0x55dd2a7ff64b

goroutine 4 [syscall]:
os/signal.signal_recv(0x55dd29c86426)
/usr/local/go/src/runtime/sigqueue.go:139 +0x9e
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x24
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x43

goroutine 6 [chan receive]:
github.com/docker/cli/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x55dd2d1a5ca0)
/go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:943 +0x8d
created by github.com/docker/cli/vendor/k8s.io/klog.init.0
/go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:403 +0x6e

rax    0x0
rbx    0x7f61332ce840
rcx    0x7f6132f20e97
rdx    0x0
rdi    0x2
rsi    0x7f612fccd830
rbp    0x55dd2b540d76
rsp    0x7f612fccd830
r8     0x0
r9     0x7f612fccd830
r10    0x8
r11    0x246
r12    0x55dd2d4cb540
r13    0x11
r14    0x55dd2b4e472c
r15    0x0
rip    0x7f6132f20e97
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

docker info

Client
Mode: false

Server:
 Containers: 412
  Running: 405
  Paused: 0
  Stopped: 7
 Images: 112
 Server Version: 19.03.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: pbtzv1rpfsbs4yqjfcgnpz2nx
  Is Manager: true
  ClusterID: 6uqnv21pvjb7jfj8x8tlo4p8o
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.0.0-31-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 62.85GiB
 Name:
 ID: 
 Docker Root Dir: /datastore/docker
 Debug Mode: false
 Username: nikitach
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  localhost:5000
  127.0.0.1:5000
  127.0.0.1:65001
  172.17.0.1:5000
  127.0.0.0/8
 Registry Mirrors:
  http:localhost:5000/
  http://127.0.0.1:65001/
  http://172.17.0.1:5000/
  http://127.0.0.1:5000/
 Live Restore Enabled: false

Yes, I had a V-Server which shared its kernel across multiple systems and some kernel restrictions limited my threads at a dynamic level depending on the overall kernel load.

I’m not 100% sure about that but swapping to bare metal solved my issues.

Hi, Did you find any solution for this? I am also getting the same error.