Problem starting docker relating to device mapper

Docker Engine General
,
1

Expected behavior

I had docker running with several containers on an EC2 instance running AMI amzn-ami-hvm-2017.03.0.20170417-x86_64-gp2 (ami-7a85a01a) with Docker installed via Yum

On shutting down the instance and later attaching the EBS volume to a new instance docker could not start.

service docker start

should start the docker service and work with my existing containers

Actual behavior

[ec2-user@ docker-elk-master]$ sudo service docker start
Starting docker: . [ OK ]
[ec2-user@ docker-elk-master]$ sudo service docker status
docker dead but subsys locked

in the logs I see

time=“2017-05-30T22:32:39Z” level=fatal msg=“unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: devicemapper, from file: devicemapper)\n”
\nTue May 30 22:47:05 UTC 2017\n
time=“2017-05-30T22:47:05.674979605Z” level=info msg="libcontainerd: new containerd process, pid: 3600"
time=“2017-05-30T22:47:06.679386676Z” level=fatal msg=“Error starting daemon: error initializing graphdriver: devicemapper: Non existing device docker-docker–pool”

Additional Information

Docker version 1.12.6, build 7392c3b/1.12.6

Steps to reproduce the behavior

  1. Start an EC2 container with EBS storage
  2. sudo yum install -y docker
  3. Start some containers
  4. Shut down the EC2 container
  5. Start a new EC2 instance
  6. Remove any EBS volumes
  7. Attach the previous EC2 instances volumes as a root volume
  8. Try to start docker service
2

I recommend upgrading to a more recent version of Docker (17.03 is the latest stable release). 1.12.6 is not longer supported.

Also note that Docker is not supported on Amazon Linux. You can find supported Docker versions on Docker Store: https://store.docker.com/

3

Thanks for the reply. Amazon Linux is a very popular AMI and I was hoping that, along with installing the latest Docker using ‘yum’ would be all that was needed :frowning: I’ll do some more research

4

If you added the new disk to add space to /var/lib/docker, this will not work. I believe you can use the new space by removing /var/lib/docker/*, but this will also remove all of your previous images/containers (destructive). To gracefully add space, I have been successful using LVM for volume mgmt. Storage using devicemapper is somewhat involved.

5

My aim was to recover the data on the volumes after recovering a snapshot. It was just a development environment though so I’m not too concerned, we’ll be using external volumes for important data.

6

Just for the record, that didn’t work.

Blockquote
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
Built: Mon Mar 4 21:25:23 2019
OS/Arch: linux/amd64
Experimental: false

We’re integrating ECS with SPOTINST and this started happening. SPOTINST does the same thing @justinhj was doing here, take a snapshot and attach it to a new AWS EC2 instance.

7

@friism Update: this came down as to how AWS configured their ECS agent and by extension docker. They require an extra volume to be mounted on /dev/xvdcz. /var/log/cloud-init-output.log shows that docker couldn’t find that volume. Usually the AMI enforces it but if you create a copy, as in a stateful spot instance, this becomes a problem as only the /dev/svda/ volume appears as mounted.