Ways to speed up Docker startup on EC2 launched from an AMI with Docker installed

General
1

Expected behavior

When starting a stopped EC2 instance with docker previously installed - docker starts up fast. When an instance is launched from an AMI created from the same stopped instance - docker would be expected to start equally fast

Actual behavior

It takes 1.5-2 min for systemctl start docker to complete after a new instance launch from the AMI (with the docker directory on a 100Gb EBS)

Additional Information

Here are docker logs from launching the instance (Oct 02) and then starting the instance after stop (Oct 03). The content is identical except it takes a long time during the launch (twice) getting to "NewControl…

The message appears to be coming from the Quota module. It is likely that the time spent is in the overlay2 driver initializing over the 100Gb drive. Is there a way to preserve and reuse its low level configuration to save time on re-initializing the driver after the launch from AMI or does the driver use some native hardware-specific data and must be re-initialized every time on new hardware?

(EC2 launch from AMI)
Oct 02 19:14:52 ip-10-1-20-83 dockerd[1055]: time=“2021-10-02T19:14:52.565623132Z” level=debug msg=“processing event stream” module=libcontainerd namespace=plugins.moby
Oct 02 19:15:45 ip-10-1-20-83 dockerd[1055]: time=“2021-10-02T19:15:45.612769261Z” level=debug msg=“NewControl(/data/docker/overlay2): nextProjectID = 2”
Oct 02 19:15:45 ip-10-1-20-83 dockerd[1055]: time=“2021-10-02T19:15:45.612863982Z” level=debug msg=“backingFs=xfs, projectQuotaSupported=true, indexOff="index=off,", userxattr=""” storage-driver=overlay2
Oct 02 19:15:45 ip-10-1-20-83 dockerd[1055]: time=“2021-10-02T19:15:45.612885962Z” level=info msg=“[graphdriver] using prior storage driver: overlay2”
Oct 02 19:15:45 ip-10-1-20-83 dockerd[1055]: time=“2021-10-02T19:15:45.612905613Z” level=debug msg=“Initialized graph driver overlay2”
Oct 02 19:16:15 ip-10-1-20-83 dockerd[1055]: time=“2021-10-02T19:16:15.242390926Z” level=debug msg=“NewControl(/data/docker/volumes): nextProjectID = 2”

(EC2 start after stop)
Oct 03 01:06:39 ip-10-1-20-83 dockerd[718]: time=“2021-10-03T01:06:39.853501121Z” level=debug msg=“processing event stream” module=libcontainerd namespace=plugins.moby
Oct 03 01:06:41 ip-10-1-20-83 dockerd[718]: time=“2021-10-03T01:06:41.422319178Z” level=debug msg=“NewControl(/data/docker/overlay2): nextProjectID = 2”
Oct 03 01:06:41 ip-10-1-20-83 dockerd[718]: time=“2021-10-03T01:06:41.422399550Z” level=debug msg=“backingFs=xfs, projectQuotaSupported=true, indexOff="index=off,", userxattr=""” storage-driver=overlay2
Oct 03 01:06:41 ip-10-1-20-83 dockerd[718]: time=“2021-10-03T01:06:41.422420548Z” level=info msg=“[graphdriver] using prior storage driver: overlay2”
Oct 03 01:06:41 ip-10-1-20-83 dockerd[718]: time=“2021-10-03T01:06:41.422444306Z” level=debug msg=“Initialized graph driver overlay2”
Oct 03 01:06:43 ip-10-1-20-83 dockerd[718]: time=“2021-10-03T01:06:43.696334329Z” level=debug msg=“NewControl(/data/docker/volumes): nextProjectID = 2”

Steps to reproduce the behavior

  1. Set up an EC2 Ubuntu 20 instance with a large 2nd EBS drive, install Docker with the data directory on that 2nd drive
  2. Stop the instance, start the instance - docker starts up fast. Stop the instance, create an AMI (with both drives) and launch a new EC2 instance from the AMI - observe the considerable extra time for docker to start