Docker Community Forums

Share and learn in the Docker community.

AWS S3 DTR Backend limitations - Layer size


Note: While you maybe able to build a docker image that is of a large size locally (say 10GB), the layers cannot be bigger than 5GB when Amazon S3 is used as the storage backend. This isn’t a DTR limitation, rather an S3 limitation it appears. According to AWS, while the maximum object size can be up to 5TB, the maximum single file upload (PUT) is 5GB.

Trying a docker push to DTR with a layer larger than 5GB will result in an error

The total volume of data and number of objects you can store are unlimited. Individual Amazon S3 objects can range in size from a minimum of 0 bytes to a maximum of 5 terabytes. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.

This is less of a complaint, and more of a notification for folks on what i’ve found. I don’t think DTR will split a layer up as a multi-part upload by default.

1 Like

Hey Michael,

I think that’s more of a limitation of the docker daemon than DTR per se, but I’ll ask around. I’m not 100% sure if the push is doing a multi-part upload or not (haven’t looked at that code in a while). What are the typical sizes for the layers that you’re pushing?

Last night I split the 6 GB layer up into 200 MB parts… Which to be honest isn’t a bad idea because you’re more likely to get a failure and a 6 GB layer download than you are on 32 200 MB layers. There’s a big database in there… So you can’t really chop parts of it up unless you gz/split it.


splunk> docker, bro.