I am building an image against the private github repository on dockerhub but it keeps failing with the exit code: 137.
I believe this happens when there’s not enough memory in the container. Why would memory be an issue on docker hub? Anyone ever faced a similar issue and fixed this issue before?
Exit Code 137 is for SIGKILL. Could be due to out of memory or something else.
Please share the build logs, so we have the chance to get an idea of what actually happens.
2023-05-28T04:12:27Z > [stage-1 15/17] RUN cd /stable-diffusion-webui && python cache.py --use-cpu=all --ckpt /model.safetensors:
2023-05-28T04:12:27Z #29 26.15 Downloading (…)okenizer_config.json: 0%| | 0.00/905 [00:00<?, ?B/s] Downloading (…)okenizer_config.json: 100%|██████████| 905/905 [00:00<00:00, 4.66MB/s]
2023-05-28T04:12:27Z #29 26.30 Downloading (…)lve/main/config.json: 0%| | 0.00/4.52k [00:00<?, ?B/s] Downloading (…)lve/main/config.json: 100%|██████████| 4.52k/4.52k [00:00<00:00, 7.18MB/s]
2023-05-28T04:12:27Z #29 28.05 Downloading (…)solve/main/vocab.txt: 0%| | 0.00/232k [00:00<?, ?B/s] Downloading (…)solve/main/vocab.txt: 100%|██████████| 232k/232k [00:00<00:00, 39.0MB/s]
2023-05-28T04:12:27Z #29 28.12 Downloading (…)okenizer_config.json: 0%| | 0.00/28.0 [00:00<?, ?B/s] Downloading (…)okenizer_config.json: 100%|██████████| 28.0/28.0 [00:00<00:00, 123kB/s]
2023-05-28T04:12:27Z #29 28.16 Downloading (…)lve/main/config.json: 0%| | 0.00/570 [00:00<?, ?B/s] Downloading (…)lve/main/config.json: 100%|██████████| 570/570 [00:00<00:00, 2.71MB/s]
2023-05-28T04:12:27Z #29 43.50 Applying cross attention optimization (InvokeAI).
2023-05-28T04:12:27Z #29 43.56 Textual inversion embeddings loaded(0):
2023-05-28T04:12:27Z #29 43.57 Model loaded in 29.5s (calculate hash: 10.5s, load weights from disk: 0.6s, create model: 1.6s, apply weights to model: 15.8s, apply half(): 0.6s, load textual inversion embeddings: 0.2s).
2023-05-28T04:12:27Z #29 49.52 load checkpoint from /stable-diffusion-webui/models/BLIP/model_base_caption_capfilt_large.pth
2023-05-28T04:12:27Z 100%|███████████████████████████████████████▊| 886M/890M [00:05<00:00, 212MiB/s] 100%|████████████████████████████████████████| 890M/890M [00:05<00:00, 172MiB/s]
2023-05-28T04:12:27Z ------
2023-05-28T04:12:27Z ERROR: failed to solve: executor failed running [/bin/bash -o pipefail -c cd /stable-diffusion-webui && python cache.py --use-cpu=all --ckpt /model.safetensors]: exit code: 137
2023-05-28T04:12:27Z Build failed using Buildkit (1)
Sometimes it succeeds for some models and fails for others.
Thanks for sharing more context information.
Do you mind sharing the Dockerfile? “failed to solve: executor failed running” looks like buildkit struggles with the Instruction. But I have no idea why python cache.py is terminated using a SIGKILL signal.
Have you checked whether it makes a difference when you ignore the build cache with docker build --no-cache ...?
i’m not building from the command line. Dockerhub autobuilds whenever i push to the github repository. here’s the repo i’m basing off of, GitHub - runpod-workers/worker-a1111: Automatic1111 serverless worker.
also see no difference when i toggle off build caching button from the build configurations page.
I wrongly assumed that you use custom build phase hooks. I used them in the past to enrich build-arguments with meta information I used in the image labels.
Do the same builds work with the github actions you have in your repo?
I am not really familiar with python, but if there is a chance to get debug information during script execution you might want to enable it, to track down the root cause.
It could indeed the resource limits for the automated build. I couldn’t find resource limits in the official documentation. Though, If found an old forum post about it Automated Build resource restrictions - #2. I have no idea whether these limits still apply. A 2GB resource limit could indeed cause your python process to be oom-killed if its usage is above the resource limit.
I would suggest to open a ticket at Dockerhub’s support and ask for clearification.
ah i see. I will reach out to the support.
Thank you so much for your help.