How to increase memory size that is available for a Docker container?

Hello everyone,

I use Docker for Desktop on Windows 10.

I run this Docker environment (postgresql container + airflow container):

I don’t know how to increase memory for a container,
in the airflow container I need to save trained scikit-learn model,
which is around 3GB and I can’t do it, but everything works fine for smaller models.

This is the volumes part from the docker-compose file:

volumes:
- ./dags:/usr/local/airflow/dags

Thanks

In the docker settings you can set how much mem to use.
Unless defined, it dosnt limit a containers max mem usage, but on windows, this setting is there for the whole docker installation

1 Like

Yes I know about these settings, I send mine, but for unknown reason this Airflow container I use has virtual memory limited to 1GB and when I try to store something bigger it fails…

can you post how you start the container, also the output of “docker inspect containername”

I send the docker-compose.yml file below as plain text because I couldn’t upload it. I start it the standard way, from command line:

docker-compose up -d

Thank you for any feedback

version: ‘3’

services:
postgres:
image: postgres:9.6
environment:
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
ports:
- “5432:5432”

webserver:
image: puckel/docker-airflow:1.10.1
build:
context: https://github.com/puckel/docker-airflow.git#1.10.1
dockerfile: Dockerfile
#args:
#AIRFLOW_DEPS: gcp_api,s3
restart: always
depends_on:
- postgres
environment:
- LOAD_EX=n
- EXECUTOR=Local
- FERNET_KEY=jsDPRErfv8Z_eVTnGfF8ywd19j4pyqE3NpdUBA_oRTo=
volumes:
- ./dags:/usr/local/airflow/dags
- ./requirements.txt:/requirements.txt
# Uncomment to include custom plugins
# - ./plugins:/usr/local/airflow/plugins
ports:
- “8080:8080”
command: webserver
healthcheck:
test: [“CMD-SHELL”, “[ -f /usr/local/airflow/airflow-webserver.pid ]”]
interval: 30s
timeout: 30s
retries: 3

I also send the result from docker inspect:

PS C:\Users\Astra\Desktop\airflow> docker inspect 9993d3170178
[
{
“Id”: “9993d3170178a7f55fe60a192bb7114e47e5a8a9122bb5df60cc9d427777a01b”,
“Created”: “2019-07-30T19:39:13.152516Z”,
“Path”: “/entrypoint.sh”,
“Args”: [
“webserver”
],
“State”: {
“Status”: “running”,
“Running”: true,
“Paused”: false,
“Restarting”: false,
“OOMKilled”: false,
“Dead”: false,
“Pid”: 4052,
“ExitCode”: 0,
“Error”: “”,
“StartedAt”: “2019-07-30T19:39:14.269083Z”,
“FinishedAt”: “0001-01-01T00:00:00Z”,
“Health”: {
“Status”: “healthy”,
“FailingStreak”: 0,
“Log”: [
{
“Start”: “2019-07-30T19:39:44.2703586Z”,
“End”: “2019-07-30T19:39:44.3530689Z”,
“ExitCode”: 1,
“Output”: “”
},
{
“Start”: “2019-07-30T19:40:14.3613675Z”,
“End”: “2019-07-30T19:40:14.4176028Z”,
“ExitCode”: 1,
“Output”: “”
},
{
“Start”: “2019-07-30T19:40:44.4315755Z”,
“End”: “2019-07-30T19:40:44.4881413Z”,
“ExitCode”: 0,
“Output”: “”
},
{
“Start”: “2019-07-30T19:41:14.496157Z”,
“End”: “2019-07-30T19:41:14.5497988Z”,
“ExitCode”: 0,
“Output”: “”
},
{
“Start”: “2019-07-30T19:41:44.5568318Z”,
“End”: “2019-07-30T19:41:44.6236122Z”,
“ExitCode”: 0,
“Output”: “”
}
]
}
},
“Image”: “sha256:17a6afda37b4bff3147ecc0c9a1f20c058090744990f525923c07bc36a6ba626”,
“ResolvConfPath”: “/var/lib/docker/containers/9993d3170178a7f55fe60a192bb7114e47e5a8a9122bb5df60cc9d427777a01b/resolv.conf”,
“HostnamePath”: “/var/lib/docker/containers/9993d3170178a7f55fe60a192bb7114e47e5a8a9122bb5df60cc9d427777a01b/hostname”,
“HostsPath”: “/var/lib/docker/containers/9993d3170178a7f55fe60a192bb7114e47e5a8a9122bb5df60cc9d427777a01b/hosts”,
“LogPath”: “/var/lib/docker/containers/9993d3170178a7f55fe60a192bb7114e47e5a8a9122bb5df60cc9d427777a01b/9993d3170178a7f55fe60a192bb7114e47e5a8a9122bb5df60cc9d427777a01b-json.log”,
“Name”: “/airflow_webserver_1”,
“RestartCount”: 0,
“Driver”: “overlay2”,
“Platform”: “linux”,
“MountLabel”: “”,
“ProcessLabel”: “”,
“AppArmorProfile”: “”,
“ExecIDs”: null,
“HostConfig”: {
“Binds”: [
“/host_mnt/c/Users/Astra/Desktop/airflow/requirements.txt:/requirements.txt:rw”,
“/host_mnt/c/Users/Astra/Desktop/airflow/dags:/usr/local/airflow/dags:rw”
],
“ContainerIDFile”: “”,
“LogConfig”: {
“Type”: “json-file”,
“Config”: {}
},
“NetworkMode”: “airflow_default”,
“PortBindings”: {
“8080/tcp”: [
{
“HostIp”: “”,
“HostPort”: “8080”
}
]
},
“RestartPolicy”: {
“Name”: “always”,
“MaximumRetryCount”: 0
},
“AutoRemove”: false,
“VolumeDriver”: “”,
“VolumesFrom”: ,
“CapAdd”: null,
“CapDrop”: null,
“Dns”: null,
“DnsOptions”: null,
“DnsSearch”: null,
“ExtraHosts”: null,
“GroupAdd”: null,
“IpcMode”: “shareable”,
“Cgroup”: “”,
“Links”: null,
“OomScoreAdj”: 0,
“PidMode”: “”,
“Privileged”: false,
“PublishAllPorts”: false,
“ReadonlyRootfs”: false,
“SecurityOpt”: null,
“UTSMode”: “”,
“UsernsMode”: “”,
“ShmSize”: 67108864,
“Runtime”: “runc”,
“ConsoleSize”: [
0,
0
],
“Isolation”: “”,
“CpuShares”: 0,
“Memory”: 0,
“NanoCpus”: 0,
“CgroupParent”: “”,
“BlkioWeight”: 0,
“BlkioWeightDevice”: null,
“BlkioDeviceReadBps”: null,
“BlkioDeviceWriteBps”: null,
“BlkioDeviceReadIOps”: null,
“BlkioDeviceWriteIOps”: null,
“CpuPeriod”: 0,
“CpuQuota”: 0,
“CpuRealtimePeriod”: 0,
“CpuRealtimeRuntime”: 0,
“CpusetCpus”: “”,
“CpusetMems”: “”,
“Devices”: null,
“DeviceCgroupRules”: null,
“DiskQuota”: 0,
“KernelMemory”: 0,
“MemoryReservation”: 0,
“MemorySwap”: 0,
“MemorySwappiness”: null,
“OomKillDisable”: false,
“PidsLimit”: 0,
“Ulimits”: null,
“CpuCount”: 0,
“CpuPercent”: 0,
“IOMaximumIOps”: 0,
“IOMaximumBandwidth”: 0,
“MaskedPaths”: [
“/proc/asound”,
“/proc/acpi”,
“/proc/kcore”,
“/proc/keys”,
“/proc/latency_stats”,
“/proc/timer_list”,
“/proc/timer_stats”,
“/proc/sched_debug”,
“/proc/scsi”,
“/sys/firmware”
],
“ReadonlyPaths”: [
“/proc/bus”,
“/proc/fs”,
“/proc/irq”,
“/proc/sys”,
“/proc/sysrq-trigger”
]
},
“GraphDriver”: {
“Data”: {
“LowerDir”: “/var/lib/docker/overlay2/90d33bd96665223731bb11dce19605c1d4261af961d373acdbc9279410fd76c1-init/diff:/var/lib/docker/overlay2/e052cea596d238b33b43e73ad5e59a2226e1e3e0b82d15ca0cd32361fd5b029a/diff:/var/lib/docker/overlay2/672fa288ac5851e2af072b730333d4595895d73d42b0f10671929dac5dc7360d/diff:/var/lib/docker/overlay2/9e323a6acd5fd82f4858083d71ca3d48cba8aaa03ba900879007549c80260cce/diff:/var/lib/docker/overlay2/5ac4cc86968437a80825a5df46292fd6298f0ba1de1c8be055c6e8e8bed616a9/diff:/var/lib/docker/overlay2/502b8863eebcf8ba8554cff96ddb843b89b4df33c862d9872f4f5835f55d5fea/diff:/var/lib/docker/overlay2/126544bb17f8ffe51f3aaf4223ceb13de73ee8da7cf3641835554049e0479dd8/diff:/var/lib/docker/overlay2/8e5020b9858042036cd1f263846877ab05ddaf5ba6d6fa7892118af16de916e5/diff:/var/lib/docker/overlay2/d35e26415ca6e36eef24a2a7b6e1460c709ae2bb0cec0eaa057b568d2d47b20f/diff:/var/lib/docker/overlay2/70e6bbf6b19b591f8808807e8c6388c91e26d4063f67c02f27b624fbacf037b1/diff”,
“MergedDir”: “/var/lib/docker/overlay2/90d33bd96665223731bb11dce19605c1d4261af961d373acdbc9279410fd76c1/merged”,
“UpperDir”: “/var/lib/docker/overlay2/90d33bd96665223731bb11dce19605c1d4261af961d373acdbc9279410fd76c1/diff”,
“WorkDir”: “/var/lib/docker/overlay2/90d33bd96665223731bb11dce19605c1d4261af961d373acdbc9279410fd76c1/work”
},
“Name”: “overlay2”
},
“Mounts”: [
{
“Type”: “bind”,
“Source”: “/host_mnt/c/Users/Astra/Desktop/airflow/requirements.txt”,
“Destination”: “/requirements.txt”,
“Mode”: “rw”,
“RW”: true,
“Propagation”: “rprivate”
},
{
“Type”: “bind”,
“Source”: “/host_mnt/c/Users/Astra/Desktop/airflow/dags”,
“Destination”: “/usr/local/airflow/dags”,
“Mode”: “rw”,
“RW”: true,
“Propagation”: “rprivate”
}
],
“Config”: {
“Hostname”: “9993d3170178”,
“Domainname”: “”,
“User”: “airflow”,
“AttachStdin”: false,
“AttachStdout”: false,
“AttachStderr”: false,
“ExposedPorts”: {
“5555/tcp”: {},
“8080/tcp”: {},
“8793/tcp”: {}
},
“Tty”: false,
“OpenStdin”: false,
“StdinOnce”: false,
“Env”: [
“LOAD_EX=n”,
“EXECUTOR=Local”,
“FERNET_KEY=jsDPRErfv8Z_eVTnGfF8ywd19j4pyqE3NpdUBA_oRTo=”,
“PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin”,
“LANG=en_US.UTF-8”,
“GPG_KEY=0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D”,
“PYTHON_VERSION=3.6.9”,
“PYTHON_PIP_VERSION=19.1.1”,
“DEBIAN_FRONTEND=noninteractive”,
“TERM=linux”,
“AIRFLOW_GPL_UNIDECODE=yes”,
“LANGUAGE=en_US.UTF-8”,
“LC_ALL=en_US.UTF-8”,
“LC_CTYPE=en_US.UTF-8”,
“LC_MESSAGES=en_US.UTF-8”
],
“Cmd”: [
“webserver”
],
“Healthcheck”: {
“Test”: [
“CMD-SHELL”,
“[ -f /usr/local/airflow/airflow-webserver.pid ]”
],
“Interval”: 30000000000,
“Timeout”: 30000000000,
“Retries”: 3
},
“Image”: “puckel/docker-airflow:1.10.1”,
“Volumes”: {
“/requirements.txt”: {},
“/usr/local/airflow/dags”: {}
},
“WorkingDir”: “/usr/local/airflow”,
“Entrypoint”: [
“/entrypoint.sh”
],
“OnBuild”: null,
“Labels”: {
“com.docker.compose.config-hash”: “6b2a9b2ba3171f42ae2e05942e20f122c9f0019f3215b415a199ebafb44b1f73”,
“com.docker.compose.container-number”: “1”,
“com.docker.compose.oneoff”: “False”,
“com.docker.compose.project”: “airflow”,
“com.docker.compose.service”: “webserver”,
“com.docker.compose.version”: “1.23.2”,
“maintainer”: “Puckel_”
}
},
“NetworkSettings”: {
“Bridge”: “”,
“SandboxID”: “76005518e566da909508b03a5ddf9d29bf7db8a1f944e9ded77afb51066a26fe”,
“HairpinMode”: false,
“LinkLocalIPv6Address”: “”,
“LinkLocalIPv6PrefixLen”: 0,
“Ports”: {
“5555/tcp”: null,
“8080/tcp”: [
{
“HostIp”: “0.0.0.0”,
“HostPort”: “8080”
}
],
“8793/tcp”: null
},
“SandboxKey”: “/var/run/docker/netns/76005518e566”,
“SecondaryIPAddresses”: null,
“SecondaryIPv6Addresses”: null,
“EndpointID”: “”,
“Gateway”: “”,
“GlobalIPv6Address”: “”,
“GlobalIPv6PrefixLen”: 0,
“IPAddress”: “”,
“IPPrefixLen”: 0,
“IPv6Gateway”: “”,
“MacAddress”: “”,
“Networks”: {
“airflow_default”: {
“IPAMConfig”: null,
“Links”: null,
“Aliases”: [
“webserver”,
“9993d3170178”
],
“NetworkID”: “bbfdeeff821031e6cf7d3ca7814479af9c10e6c13d78a1a759248fbe400197bb”,
“EndpointID”: “d6a0f20c969a240852d146762d557b3d5ddef59f945ba0e9748103130c8a2cf6”,
“Gateway”: “172.18.0.1”,
“IPAddress”: “172.18.0.3”,
“IPPrefixLen”: 16,
“IPv6Gateway”: “”,
“GlobalIPv6Address”: “”,
“GlobalIPv6PrefixLen”: 0,
“MacAddress”: “02:42:ac:12:00:03”,
“DriverOpts”: null
}
}
}
}
]

I send the result from docker inspect also: https://justpaste.it/454rv

Hello, having the same issue. Has anyone had any luck?

You can also set the memory used by docker by editing the json file C:\Users\Personal\AppData\Roaming\Docker\settings.json . Look for a property called MemoryMiB and update its value to be the number of megabytes you want your docker installation to use.