Hello everyone,
i am trying to build a server with multiple services running on docker (jira, gitlab, owncloud, mailcow).
I have built docker compose setups for all these services and it worked fine until, i started the 4th service (either jira or gitlab). When i do so the containers start up and a few seconds later i hit some thread limit i guess.
Nothing on my system is able to start any threads. For example my zsh autocompletion starts to fail while i type.
When i run docker ps i get this error:
runtime/cgo: pthread_create failed: Resource temporarily unavailable
My Task Limit is okay:
Tasks: 164 (limit: 512)
I would love to provide any logs and information you need but i am relatively new to docker and you have to tell me what you need.
Big thanks in advance to everyone who tries to help me out!
docker info:
Containers: 23
Running: 23
Paused: 0
Stopped: 0
Images: 83
Server Version: 18.09.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 32GiB
Name: h2667679.stratoserver.net
ID: 6X5W:R4S6:LL5U:M6VV:UKFY:M2VO:XTUP:WEUD:FNUV:IW4N:4HOR:P3TL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
System:
Debian 9
Linux 4.9.0
Stack Trace:
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f62d7380fff m=4 sigcode=18446744073709551610
goroutine 0 [idle]:
runtime: unknown pc 0x7f62d7380fff
stack: frame={sp:0x7f62d634a978, fp:0x0} stack=[0x7f62d5b4b2f0,0x7f62d634aef0)
00007f62d634a878: 000000c4201aefc0 3fd3333333333333
00007f62d634a888: 0000000000000000 3fd0000000000000
00007f62d634a898: 0000000000000000 3fedaf1f95df65ca
00007f62d634a8a8: 0000000000000000 3fab3fb514b91010
00007f62d634a8b8: 0000000000000000 3fc0136eedb0acdc
00007f62d634a8c8: 0000000000000000 000000c4201bb650
00007f62d634a8d8: 000000c4201bb890 000000c4202d2030
00007f62d634a8e8: 000000c4201ae810 000000c4201aefc0
00007f62d634a8f8: 000000c4201ba690 0000000000000000
00007f62d634a908: 000000c4202d2450 000000c4202d2780
00007f62d634a918: 000000c420283440 0000000000000000
00007f62d634a928: 0000000000000000 0000000000000000
00007f62d634a938: 0000000000000000 0000000000000000
00007f62d634a948: 0000000000000000 0000000000000000
00007f62d634a958: 0000000000000000 0000000000000000
00007f62d634a968: 0000000000000000 0000000000000000
00007f62d634a978: <0000000000000000 fffffffe7fffffff
00007f62d634a988: ffffffffffffffff ffffffffffffffff
00007f62d634a998: ffffffffffffffff ffffffffffffffff
00007f62d634a9a8: ffffffffffffffff ffffffffffffffff
00007f62d634a9b8: ffffffffffffffff ffffffffffffffff
00007f62d634a9c8: ffffffffffffffff ffffffffffffffff
00007f62d634a9d8: ffffffffffffffff ffffffffffffffff
00007f62d634a9e8: ffffffffffffffff ffffffffffffffff
00007f62d634a9f8: ffffffffffffffff 00007f62d76e86e0
00007f62d634aa08: 00007f62d738242a 0000000000000020
00007f62d634aa18: 0000000000000000 0000000000000000
00007f62d634aa28: 0000000000000000 0000000000000000
00007f62d634aa38: 0000000000000000 0000000000000000
00007f62d634aa48: 0000000000000000 0000000000000000
00007f62d634aa58: 0000000000000000 0000000000000000
00007f62d634aa68: 0000000000000000 0000000000000000
runtime: unknown pc 0x7f62d7380fff
stack: frame={sp:0x7f62d634a978, fp:0x0} stack=[0x7f62d5b4b2f0,0x7f62d634aef0)
00007f62d634a878: 000000c4201aefc0 3fd3333333333333
00007f62d634a888: 0000000000000000 3fd0000000000000
00007f62d634a898: 0000000000000000 3fedaf1f95df65ca
00007f62d634a8a8: 0000000000000000 3fab3fb514b91010
00007f62d634a8b8: 0000000000000000 3fc0136eedb0acdc
00007f62d634a8c8: 0000000000000000 000000c4201bb650
00007f62d634a8d8: 000000c4201bb890 000000c4202d2030
00007f62d634a8e8: 000000c4201ae810 000000c4201aefc0
00007f62d634a8f8: 000000c4201ba690 0000000000000000
00007f62d634a908: 000000c4202d2450 000000c4202d2780
00007f62d634a918: 000000c420283440 0000000000000000
00007f62d634a928: 0000000000000000 0000000000000000
00007f62d634a938: 0000000000000000 0000000000000000
00007f62d634a948: 0000000000000000 0000000000000000
00007f62d634a958: 0000000000000000 0000000000000000
00007f62d634a968: 0000000000000000 0000000000000000
00007f62d634a978: <0000000000000000 fffffffe7fffffff
00007f62d634a988: ffffffffffffffff ffffffffffffffff
00007f62d634a998: ffffffffffffffff ffffffffffffffff
00007f62d634a9a8: ffffffffffffffff ffffffffffffffff
00007f62d634a9b8: ffffffffffffffff ffffffffffffffff
00007f62d634a9c8: ffffffffffffffff ffffffffffffffff
00007f62d634a9d8: ffffffffffffffff ffffffffffffffff
00007f62d634a9e8: ffffffffffffffff ffffffffffffffff
00007f62d634a9f8: ffffffffffffffff 00007f62d76e86e0
00007f62d634aa08: 00007f62d738242a 0000000000000020
00007f62d634aa18: 0000000000000000 0000000000000000
00007f62d634aa28: 0000000000000000 0000000000000000
00007f62d634aa38: 0000000000000000 0000000000000000
00007f62d634aa48: 0000000000000000 0000000000000000
00007f62d634aa58: 0000000000000000 0000000000000000
00007f62d634aa68: 0000000000000000 0000000000000000
goroutine 34 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:363 fp=0xc4204a6550 sp=0xc4204a6548 pc=0x559fd4418820
runtime.gcMarkTermination(0x3fed47e96b1518a4)
/usr/local/go/src/runtime/mgc.go:1647 +0x40d fp=0xc4204a6720 sp=0xc4204a6550 pc=0x559fd43dcacd
runtime.gcMarkDone()
/usr/local/go/src/runtime/mgc.go:1513 +0x230 fp=0xc4204a6748 sp=0xc4204a6720 pc=0x559fd43dc660
runtime.gcBgMarkWorker(0xc42004aa00)
/usr/local/go/src/runtime/mgc.go:1912 +0x2f1 fp=0xc4204a67d8 sp=0xc4204a6748 pc=0x559fd43dd5e1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4204a67e0 sp=0xc4204a67d8 pc=0x559fd441b3e1
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1723 +0x7b
goroutine 1 [semacquire]:
text/tabwriter.(*Writer).terminateCell(...)
/usr/local/go/src/text/tabwriter/tabwriter.go:447
text/tabwriter.(*Writer).Write(0xc420680000, 0xc420668980, 0x35, 0x39, 0x2a, 0x0, 0x0)
/usr/local/go/src/text/tabwriter/tabwriter.go:508 +0x515
bytes.(*Buffer).WriteTo(0xc420196380, 0x559fd60c05a0, 0xc420680000, 0x559fd5e9ccc0, 0xc4205f03c0, 0x0)
/usr/local/go/src/bytes/buffer.go:240 +0xb8
github.com/docker/cli/cli/command/formatter.(*Context).postFormat(0xc4205fe050, 0xc4205f4180, 0x559fd60bdee0, 0xc420602300)
/go/src/github.com/docker/cli/cli/command/formatter/formatter.go:77 +0x1bb
github.com/docker/cli/cli/command/formatter.(*Context).Write(0xc4205fe050, 0x559fd60bdee0, 0xc420602300, 0xc4205f0390, 0xc4204301e8, 0x0)
/go/src/github.com/docker/cli/cli/command/formatter/formatter.go:117 +0x18e
github.com/docker/cli/cli/command/formatter.ContainerWrite(0x559fd60bde40, 0xc42002d1d0, 0x559fd572a444, 0x57, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/docker/cli/cli/command/formatter/container.go:74 +0x59e
github.com/docker/cli/cli/command/container.runPs(0x559fd60ff420, 0xc420397b00, 0xc4205ab1d0, 0x0, 0x0)
/go/src/github.com/docker/cli/cli/command/container/list.go:139 +0x273
github.com/docker/cli/cli/command/container.NewPsCommand.func1(0xc4205bd400, 0x559fd6be1c20, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/docker/cli/cli/command/container/list.go:36 +0x3e
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).execute(0xc4205bd400, 0xc42000c070, 0x0, 0x0, 0xc4205bd400, 0xc42000c070)
/go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:762 +0x46a
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420102a00, 0xc420029fa0, 0x559fd5db8fa0, 0xc420029fb0)
/go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:852 +0x30c
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420102a00, 0xc420102a00, 0x559fd60c0360)
/go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:800 +0x2d
main.main()
/go/src/github.com/docker/cli/cmd/docker/docker.go:180 +0xde
goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:139 +0xa8
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x24
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:28 +0x43
goroutine 37 [chan receive]:
github.com/docker/cli/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x559fd6bc32e0)
/go/src/github.com/docker/cli/vendor/github.com/golang/glog/glog.go:882 +0x8d
created by github.com/docker/cli/vendor/github.com/golang/glog.init.0
/go/src/github.com/docker/cli/vendor/github.com/golang/glog/glog.go:410 +0x205
goroutine 12 [IO wait]:
internal/poll.runtime_pollWait(0x7f62d7c8bf00, 0x72, 0xc4205e79a8)
/usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4205d4098, 0x72, 0xffffffffffffff00, 0x559fd60c26a0, 0x559fd6b55aa8)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4205d4098, 0xc4205af000, 0x1000, 0x1000)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4205d4080, 0xc4205af000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4205d4080, 0xc4205af000, 0x1000, 0x1000, 0x559fd4417260, 0xc420000180, 0x4)
/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc420388848, 0xc4205af000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:176 +0x6c
net/http.(*persistConn).Read(0xc420257560, 0xc4205af000, 0x1000, 0x1000, 0xc4205e7b98, 0x559fd43c7435, 0xc4205e8000)
/usr/local/go/src/net/http/transport.go:1453 +0x138
bufio.(*Reader).fill(0xc42046ec00)
/usr/local/go/src/bufio/bufio.go:100 +0x120
bufio.(*Reader).Peek(0xc42046ec00, 0x1, 0x0, 0x0, 0x0, 0xc4205e80c0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0x3c
net/http.(*persistConn).readLoop(0xc420257560)
/usr/local/go/src/net/http/transport.go:1601 +0x187
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1237 +0x95c
goroutine 13 [select]:
net/http.(*persistConn).writeLoop(0xc420257560)
/usr/local/go/src/net/http/transport.go:1822 +0x14d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1238 +0x981
rax 0x0
rbx 0x6
rcx 0xffffffffffffffff
rdx 0x0
rdi 0x2
rsi 0x7f62d634a900
rbp 0x559fd58771ea
rsp 0x7f62d634a978
r8 0x0
r9 0x7f62d634a900
r10 0x8
r11 0x246
r12 0x7f62c80008c0
r13 0xf1
r14 0x11
r15 0x0
rip 0x7f62d7380fff
rflags 0x246
cs 0x33
fs 0x0
gs 0x0
has anyone any ideas?