Hello and thanks for reading. We use docker containers on linux virtual machines. Sometimes these exist in AWS, sometimes VMware. However on occasion when running docker commands we will get the following error.
runtime/cgo: pthread_create failed: Resource temporarily unavailable
Our team has scoured the internet looking for a fix, all of which to no avail.
Some of the things we checked were.
-
TasksMax attribute for systemd, the number of active tasks for the docker systemd service always seems to be low so I don’t think this could be a problem.
-
Thread and Process limits don’t appear to be reached. These numbers are set pretty high. Of course, these numbers will differ pending the resources allotted to the virtual machine.
cat /proc/sys/kernel/pid_max 32768 cat /proc/sys/kernel/threads-max 31396 -
We monitored file descriptors as well, there could be more to check here but these too appeared to not tinker around the threshold
Steps to Reproduce
We weren’t able to reliably reproduce until we manually modified the kernel threads-max attribute down to around 500 which seemed to produce the same sort of behavior we were seeing. Again, we check the current thread and process counts whenever we have problems so we believe the problem can’t be the kernel threads max attribute.
Help
- Has anyone figured out a workaround or solution to this problem?
- Perhaps someone has a way to reproduce this issue reliably using a different method than what I provided?
Ulimit output
I am still wrapping my head around how ulimit works but I am pretty sure we are sticking with system defaults. All the numbers are large or unlimited. All the lines are commented out in /etc/security/limits.conf
ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15698
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 15698
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Docker Version
Docker version 19.03.11-ol, build 9bb540d
OS Release
NAME="Oracle Linux Server"
VERSION="7.9"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Oracle Linux Server 7.9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:9:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"
ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.9
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.9
Redhat release
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
SystemD Version
systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
Goroutine logs
Attempting to load up new docker images
runtime/cgo: runtime/cgo: pthread_create failed: Resource temporarily unavailablepthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f0a1ceda387 m=0 sigcode=18446744073709551610
goroutine 0 [idle]:
runtime: unknown pc 0x7f0a1ceda387
stack: frame={sp:0x7fff6601e5a8, fp:0x0} stack=[0x7fff6581fd38,0x7fff6601ed70)
00007fff6601e4a8: 00007f09f413c958 0000000100000000
00007fff6601e4b8: 0000000100000514 00007f0a1d8a5728
00007fff6601e4c8: 00007fff6601e6e0 00007fff6601e620
00007fff6601e4d8: 0000000000000001 00005583c1594f80
00007fff6601e4e8: 00005583c1595c88 00005583c1595930
00007fff6601e4f8: 00007f0a1d69c09f 0000000000000000
00007fff6601e508: 00005583c1594f80 0000007700000001
00007fff6601e518: 0000000000000000 00007fff00000001
00007fff6601e528: 00005583c1595930 441780e976e51880
00007fff6601e538: 0000000000000000 000000000000007c
00007fff6601e548: 0000000000000000 0000000000000000
00007fff6601e558: 00005583c1595c88 00007fff6601e590
00007fff6601e568: 00007fff6601e580 0000000022494966
00007fff6601e578: 00007f0a1d26c868 00005583c00e792a
00007fff6601e588: 00005583c1594fa0 0000000000000000
00007fff6601e598: 00005583c00bfe0a 0000000000000000
00007fff6601e5a8: <00007f0a1cedba78 0000000000000020
00007fff6601e5b8: 0000000000000000 0000000000000000
00007fff6601e5c8: 0000000000000000 0000000000000000
00007fff6601e5d8: 0000000000000000 0000000000000000
00007fff6601e5e8: 0000000000000000 0000000000000000
00007fff6601e5f8: 0000000000000000 0000000000000000
00007fff6601e608: 0000000000000000 0000000000000000
00007fff6601e618: 0000000000000000 0000000000000000
00007fff6601e628: 0000000000000000 000000000000000d
00007fff6601e638: 000000000000000a 00007f0a1d26d9f0
00007fff6601e648: 00007f0a1cf1b28a 000000000000000d
00007fff6601e658: 00007f0a1d26c1c0 0000000000000001
00007fff6601e668: 00007f0a1d26c243 00007f0a1d268380
00007fff6601e678: 00007f0a1cf1fb0e 00007f0a1d26c1c0
00007fff6601e688: 000000000000000a 00005583c1594fa0
00007fff6601e698: 0000000000000000 00005583c00bfe0a
runtime: unknown pc 0x7f0a1ceda387
stack: frame={sp:0x7fff6601e5a8, fp:0x0} stack=[0x7fff6581fd38,0x7fff6601ed70)
00007fff6601e4a8: 00007f09f413c958 0000000100000000
00007fff6601e4b8: 0000000100000514 00007f0a1d8a5728
00007fff6601e4c8: 00007fff6601e6e0 00007fff6601e620
00007fff6601e4d8: 0000000000000001 00005583c1594f80
00007fff6601e4e8: 00005583c1595c88 00005583c1595930
00007fff6601e4f8: 00007f0a1d69c09f 0000000000000000
00007fff6601e508: 00005583c1594f80 0000007700000001
00007fff6601e518: 0000000000000000 00007fff00000001
00007fff6601e528: 00005583c1595930 441780e976e51880
00007fff6601e538: 0000000000000000 000000000000007c
00007fff6601e548: 0000000000000000 0000000000000000
00007fff6601e558: 00005583c1595c88 00007fff6601e590
00007fff6601e568: 00007fff6601e580 0000000022494966
00007fff6601e578: 00007f0a1d26c868 00005583c00e792a
00007fff6601e588: 00005583c1594fa0 0000000000000000
00007fff6601e598: 00005583c00bfe0a 0000000000000000
00007fff6601e5a8: <00007f0a1cedba78 0000000000000020
00007fff6601e5b8: 0000000000000000 0000000000000000
00007fff6601e5c8: 0000000000000000 0000000000000000
00007fff6601e5d8: 0000000000000000 0000000000000000
00007fff6601e5e8: 0000000000000000 0000000000000000
00007fff6601e5f8: 0000000000000000 0000000000000000
00007fff6601e608: 0000000000000000 0000000000000000
00007fff6601e618: 0000000000000000 0000000000000000
00007fff6601e628: 0000000000000000 000000000000000d
00007fff6601e638: 000000000000000a 00007f0a1d26d9f0
00007fff6601e648: 00007f0a1cf1b28a 000000000000000d
00007fff6601e658: 00007f0a1d26c1c0 0000000000000001
00007fff6601e668: 00007f0a1d26c243 00007f0a1d268380
00007fff6601e678: 00007f0a1cf1fb0e 00007f0a1d26c1c0
00007fff6601e688: 000000000000000a 00005583c1594fa0
00007fff6601e698: 0000000000000000 00005583c00bfe0a
goroutine 1 [runnable, locked to thread]:
reflect.resolveTypeOff(0x5583c02ea6e0, 0xc0002b84a0, 0x5583c035b3c9)
/usr/lib/golang/src/runtime/runtime1.go:503 +0x4e
github.com/docker/cli/vendor/github.com/modern-go/reflect2.loadGo17Types()
/go/src/github.com/docker/cli/vendor/github.com/modern-go/reflect2/type_map.go:70 +0x11f
github.com/docker/cli/vendor/github.com/modern-go/reflect2.init.0()
/go/src/github.com/docker/cli/vendor/github.com/modern-go/reflect2/type_map.go:28 +0x98
goroutine 18 [chan receive]:
github.com/docker/cli/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x5583c1456680)
/go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:1010 +0x8d
created by github.com/docker/cli/vendor/k8s.io/klog.init.0
/go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:411 +0xd8
rax 0x0
rbx 0x7f0a1d26c868
rcx 0xffffffffffffffff
rdx 0x6
rdi 0x75e9
rsi 0x75e9
rbp 0x5583c00e792a
rsp 0x7fff6601e5a8
r8 0xa
r9 0x7f0a1d8a5740
r10 0x8
r11 0x202
r12 0x5583c1594fa0
r13 0x0
r14 0x5583c00bfe0a
r15 0x0
rip 0x7f0a1ceda387
rflags 0x202
cs 0x33
fs 0x0
gs 0x0
Unable to load up docker images. Sleeping and retrying (1)
Attempting to load up new docker images
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f54d0781387 m=9 sigcode=18446744073709551610
goroutine 0 [idle]:
runtime: unknown pc 0x7f54d0781387
stack: frame={sp:0x7f54a1ffa928, fp:0x0} stack=[0x7f54a17fb2a8,0x7f54a1ffaea8)
00007f54a1ffa828: 0000000000000000 00007f54a80e4000
00007f54a1ffa838: 0000562e33b072a0 <runtime.mmap.func1+0> 0000000000000000
00007f54a1ffa848: 0000000000040000 0000002200000003
00007f54a1ffa858: 00000000ffffffff 00000000001722c4
00007f54a1ffa868: 00007f54a1ffa8b8 0000562e33abdea5 <runtime.sysAlloc+261>
00007f54a1ffa878: 0000562e366a6ea0 0000000000040000
00007f54a1ffa888: 0000002200000003 00000000ffffffff
00007f54a1ffa898: 00007f54a80e4000 0000000000000000
00007f54a1ffa8a8: 0000000000002030 000000000016e2c4
00007f54a1ffa8b8: 00007f54a1ffa908 0000000033ab2f46
00007f54a1ffa8c8: 0000562e366a6ea0 ffffffffffffc000
00007f54a1ffa8d8: 00007f54a80e4000 0000000000000008
00007f54a1ffa8e8: fffffffffffffff8 000000c00006e650
00007f54a1ffa8f8: 00007f54d0b13868 0000562e3530592a
00007f54a1ffa908: 00007f54d1158010 0000000000000000
00007f54a1ffa918: 0000562e352dde0a 0000000000000000
00007f54a1ffa928: <00007f54d0782a78 0000000000000020
00007f54a1ffa938: 0000000000000000 0000000000000000
00007f54a1ffa948: 0000000000000000 0000000000000000
00007f54a1ffa958: 0000000000000000 0000000000000000
00007f54a1ffa968: 0000000000000000 0000000000000000
00007f54a1ffa978: 0000000000000000 0000000000000000
00007f54a1ffa988: 0000000000000000 0000000000000000
00007f54a1ffa998: 0000000000000000 0000000000000000
00007f54a1ffa9a8: 0000000000000000 000000000000000d
00007f54a1ffa9b8: 00007f54d083aab9 0000000000000000
00007f54a1ffa9c8: 00007f54d07c52f3 000000000000000d
00007f54a1ffa9d8: 00007f54d0b131c0 0000000000000001
00007f54a1ffa9e8: 00007f54d0b13243 00007f54d0b0f380
00007f54a1ffa9f8: 00007f54d07c6b0e 00007f54d0b131c0
00007f54a1ffaa08: 000000000000000a 00007f54d1158010
00007f54a1ffaa18: 0000000000000000 0000562e352dde0a
runtime: unknown pc 0x7f54d0781387
stack: frame={sp:0x7f54a1ffa928, fp:0x0} stack=[0x7f54a17fb2a8,0x7f54a1ffaea8)
00007f54a1ffa828: 0000000000000000 00007f54a80e4000
00007f54a1ffa838: 0000562e33b072a0 <runtime.mmap.func1+0> 0000000000000000
00007f54a1ffa848: 0000000000040000 0000002200000003
00007f54a1ffa858: 00000000ffffffff 00000000001722c4
00007f54a1ffa868: 00007f54a1ffa8b8 0000562e33abdea5 <runtime.sysAlloc+261>
00007f54a1ffa878: 0000562e366a6ea0 0000000000040000
00007f54a1ffa888: 0000002200000003 00000000ffffffff
00007f54a1ffa898: 00007f54a80e4000 0000000000000000
00007f54a1ffa8a8: 0000000000002030 000000000016e2c4
00007f54a1ffa8b8: 00007f54a1ffa908 0000000033ab2f46
00007f54a1ffa8c8: 0000562e366a6ea0 ffffffffffffc000
00007f54a1ffa8d8: 00007f54a80e4000 0000000000000008
00007f54a1ffa8e8: fffffffffffffff8 000000c00006e650
00007f54a1ffa8f8: 00007f54d0b13868 0000562e3530592a
00007f54a1ffa908: 00007f54d1158010 0000000000000000
00007f54a1ffa918: 0000562e352dde0a 0000000000000000
00007f54a1ffa928: <00007f54d0782a78 0000000000000020
00007f54a1ffa938: 0000000000000000 0000000000000000
00007f54a1ffa948: 0000000000000000 0000000000000000
00007f54a1ffa958: 0000000000000000 0000000000000000
00007f54a1ffa968: 0000000000000000 0000000000000000
00007f54a1ffa978: 0000000000000000 0000000000000000
00007f54a1ffa988: 0000000000000000 0000000000000000
00007f54a1ffa998: 0000000000000000 0000000000000000
00007f54a1ffa9a8: 0000000000000000 000000000000000d
00007f54a1ffa9b8: 00007f54d083aab9 0000000000000000
00007f54a1ffa9c8: 00007f54d07c52f3 000000000000000d
00007f54a1ffa9d8: 00007f54d0b131c0 0000000000000001
00007f54a1ffa9e8: 00007f54d0b13243 00007f54d0b0f380
00007f54a1ffa9f8: 00007f54d07c6b0e 00007f54d0b131c0
00007f54a1ffaa08: 000000000000000a 00007f54d1158010
00007f54a1ffaa18: 0000000000000000 0000562e352dde0a
goroutine 1 [semacquire, locked to thread]:
github.com/docker/cli/vendor/github.com/json-iterator/go.Config.Froze(0x0, 0x100, 0x0, 0x0, 0x0, 0x562e365f1458, 0x562e3561b880)
/go/src/github.com/docker/cli/vendor/github.com/json-iterator/go/config.go:130 +0x33
github.com/docker/cli/vendor/github.com/json-iterator/go.init()
/go/src/github.com/docker/cli/vendor/github.com/json-iterator/go/config.go:52 +0x58b
goroutine 7 [chan receive]:
github.com/docker/cli/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x562e36674680)
/go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:1010 +0x8d
created by github.com/docker/cli/vendor/k8s.io/klog.init.0
/go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:411 +0xd8
rax 0x0
rbx 0x7f54d0b13868
rcx 0xffffffffffffffff
rdx 0x6
rdi 0x75f5
rsi 0x75fd
rbp 0x562e3530592a
rsp 0x7f54a1ffa928
r8 0xa
r9 0x7f54a1ffb700
r10 0x8
r11 0x206
r12 0x7f54d1158010
r13 0x0
r14 0x562e352dde0a
r15 0x0
rip 0x7f54d0781387
rflags 0x206
cs 0x33
fs 0x0
gs 0x0
Unable to load up docker images. Sleeping and retrying (2)
Attempting to load up new docker images
Any insight is appreciated. Throughout docker, golangnuts, and stack overflow forums all of the issues similar to mine are either solved with a method that did not work for me, or are at a complete standstill. Links returned from a multitude of google searches have turned purple with exhaustion.