Docker Community Forums

Share and learn in the Docker community.

AWS S3 DTR Backend limitations - Layer size

,

Note: While you maybe able to build a docker image that is of a large size locally (say 10GB), the layers cannot be bigger than 5GB when Amazon S3 is used as the storage backend. This isn’t a DTR limitation, rather an S3 limitation it appears. According to AWS, while the maximum object size can be up to 5TB, the maximum single file upload (PUT) is 5GB.

Trying a docker push to DTR with a layer larger than 5GB will result in an error

from: https://aws.amazon.com/s3/faqs/
The total volume of data and number of objects you can store are unlimited. Individual Amazon S3 objects can range in size from a minimum of 0 bytes to a maximum of 5 terabytes. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.

This is less of a complaint, and more of a notification for folks on what i’ve found. I don’t think DTR will split a layer up as a multi-part upload by default.

1 Like

Hey Michael,

I think that’s more of a limitation of the docker daemon than DTR per se, but I’ll ask around. I’m not 100% sure if the push is doing a multi-part upload or not (haven’t looked at that code in a while). What are the typical sizes for the layers that you’re pushing?

Last night I split the 6 GB layer up into 200 MB parts… Which to be honest isn’t a bad idea because you’re more likely to get a failure and a 6 GB layer download than you are on 32 200 MB layers. There’s a big database in there… So you can’t really chop parts of it up unless you gz/split it.

-michael

splunk> docker, bro.