Docker-compose restart too impatient?

Open Source Projects Compose
1

Summary
“docker-compose up -d” is trying to start containers more than once. This is non-fatal (because the second instance fails and exits) but is a bit worrisome.

This appears to be behavior that has begun recently, after an update.

Background
I am running a docker-compose configuration that I’ve had for a while. Nothing in my configuration itself has changed, but I did recently update the OS (Ubuntu 20.04) and all packages, and did fresh pulls on the underlying containers.

What I’m Seeing
Here’s a partial snapshot of what I’m seeing:

brad@stanley:~/nextcloud$ docker-compose up -d
Creating network “nextcloud_nextcloud” with the default driver
Creating network “nextcloud_default” with the default driver
Creating nextcloud-mariadb …
Creating nextcloud-redis …
Creating nextcloud_db_1 … done
Creating nextcloud-coturn …
Creating nextcloud-mariadb … done
Creating nextcloud-redis … done
WARNING: Host is already in use by another container

Creating nextcloud-coturn … error
> f0acd4520d21d4ead2b7a77b942e7c46a5ee70095b964912b5f): Error starting userland proxy: listen tcp 127.0.0.1:9980: bind: address already in use

The process continues, but I’ve snipped the output there. I’ve italicized the error at the bottom.

The reason for the failure appears to be that

  • The “nextcloud-coturn” container is being started a second time.
  • The first instance of the container is already bound to the socket that it needs, so the second instance fails.

This happens with 3 of my containers in all.

What Appears to be Happening
In general, for each of the three failed second-instance-containers, the second instance appears to be attempting to start 8-10 seconds after the first instance.

I have the “restart” keyword configured to “always” or “unless-stopped” for all my services. I read in the docs that restart detection begins about 10 seconds after launch, so this is consistent with the time delay I’m seeing.

What I’ve Tried
I have searched and searched the docs for a way to lengthen the delay before restart detection commences. I see that it is possible to specify a delay when using the “deploy” keyword, but not when using the simple “restart” keyword.

I have also tried specifying “-t 25” or the like on the docker-compose command line, but it doesn’t make any difference. I’ve also tried the --no-recreate switch, again with no effect.

Thanks for reading. Any tips would be appreciated.

Brad

2

I wanted to follow up before anyone digs in too much. It’s possible that this is not a docker problem. I noticed that my “mariadb” container is restarting every 25 or 30 seconds. I will investigate further and update.

3

Update - Major insight. It turns out that something in the updates changed where docker stores its volumes.

Previously, it was storing them in

/var/snap/docker/common/var-lib-docker/volumes

Now it is storing them in

/var/lib/docker/volumes/

I did a wholesale copy of the stuff in the old location into the new, and now everything is working much better. Something is still off with my website, but it appears to be a Wordpress-specific issue related to content, not a docker issue.

4

Looks like you use the Snap Docker package. To have a reliable behavior, you might want to consider to use the Docker package from Docker’s official repository instead. You can be sure that Docker’s Docker behaves like intendend, every other distribution is a redistribution that may or may not be modified to allign with the package managers philosophy, which is definitly the case for the Snap package.

Hint: your problem wouldn’t have happend with the docker package from the official repository.

5

Hi Metin,

Thanks so much for the quick reply.

About Snap
I am definitely confused by what’s going on. I have kept detailed notes as I have set up my server (now at 43 pages and counting).

Looking at those notes, I see that I installed docker using the instructions from the official docker docs at Install Docker Engine on Ubuntu | Docker Documentation, which amounts to doing an “apt install”. (No snap involved, but there is a package manager, but it’s the one recommended by Docker.) The only thing I’ve used snap for in the process of setting up my server was installing certbot.

So I’m puzzled as to why things ever ended up under /var/snap. But I think we’re okay now.

Things that Have Improved
At this point, I’ve copied the docker volumes from /var/snap/… into /var/lib/…, and all seemed to be working well when I went to bed last night.

Also, at boot time, I’m no longer getting doubly-started containers. Everything starts up smoothly.

Lastly, after booting, all containers seemed healthy. MariaDb is not continously restarting.

What Hasn’t Improved
But this morning when I looked, mariadb has once again begun restarting every 30 seconds or so.

I will investigate the logs to try to understand what’s going on. I assume this is a mariadb problem and not a docker problem.

Thanks for your help, much appreciated.

–Brad

6

Well, a bunch of my problems have returned.

It appears that I have two copies of dockerd running. This can’t be right, can it?

Should I just find and blow away the one related to snap?

4 S root 1340 1 0 80 0 - 567383 - 14:08 ? 00:00:02 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
4 S root 4224 1 2 80 0 - 483957 - 14:10 ? 00:00:08 dockerd --group docker --exec-root=/run/snap.docker --data-root=/var/snap/docker/common/var-lib-docker --pidfile=/run/snap.docker/docker.pid --config-file=/var/snap/docker/796/config/daemon.json

Thanks,
Brad

7

Docker usualy creates the unix daemon socket /var/run/docker.sock - which would have prevented a second docker engine to run. Seems like the snap version stores it’s unix daemon socket somewhere else and both versions can run side by side. Uhh. I already didn’t like the snap package, it deserves a hazard sticker on top. Just because something is forked and compiled from the docker sources, doesn’t mean every docker distribution behaves identicaly…

Though when it commes to the client aka the docker cli command, it can only instruct the docker engine whichs socket location it knows. But in therory different users can have different PATH’s set, which might result in different users using diffferent client binaries with different settings.

I hope at least one of those docker engines has a consistent copy of the latest state of your volumes…

8

I now think that I may have had two dockers running all along, without realizing it. I am sort of piecing together that I may have done some early experiments and may have used snap, and then forgot about it.

I did “snap disable docker”, and now the lone remaining docker instance seems to be running smoothly.

I copied the one volume that was in use in snap-docker over and things are mostly okay. Just a little mopping up to do.

Thanks for your help.