Docker Compose seems to start containers multiple times

General
1

Hi,

I hope this is the right place for my problem: We are using docker-compose for running our services by now. This was all running fine until yesterday. Since yesterday it seems that docker-compose runs some containers twice so that their ports conflict.

And I have no clue, why this happens. It looks like one of the containers is getting out of control of docker-compose.

my docker-compose file:

version: '3.0'
services:
  stag-lx-pcs-reporting-api:
    networks:
      - internal_compose_stag
    environment:
        ...
    image: ...
    container_name: stag-lx-pcs-reporting-api
    command: ["/lx-pcs-reporting-ctl", "http"]
    ports:
        - 51000:51000
    restart: on-failure

networks:
  internal_compose_stag:
    external:
      name: compose_stag

The network is because we have a couple of services that communicate with each other. Our other services are working fine.

Docker-Compose is 1.20.1

The way it happens is just by calling

docker-compose -f this-file.yml up

The result is:

docker-compose -f /opt/docker-compose-services/docker-compose.stag.lx-pcs-reporting-api.yml up
Creating stag-lx-pcs-reporting-api ... error

ERROR: for stag-lx-pcs-reporting-api  Cannot start service stag-lx-pcs-reporting-api: driver failed programming external connectivity on endpoint stag-lx-pcs-reporting-api (e728e0fd4dcd193e12bae530d39ab85ffe3984c96c93fdd2ea309e2048f593ea): Error starting userland proxy: listen tcp 0.0.0.0:51000: bind: address already in use

ERROR: for stag-lx-pcs-reporting-api  Cannot start service stag-lx-pcs-reporting-api: driver failed programming external connectivity on endpoint stag-lx-pcs-reporting-api (e728e0fd4dcd193e12bae530d39ab85ffe3984c96c93fdd2ea309e2048f593ea): Error starting userland proxy: listen tcp 0.0.0.0:51000: bind: address already in use
ERROR: Encountered errors while bringing up the project.

Strange thing is that one container remains event though they should be removed when I stop docker-compose. That’s why I think that docker-compose looses control over one of the containers.

EDIT: This is the output of docker-compose events when I call docker-compose up:

2018-04-06 07:40:37.636293 container create 5271f4efd27106208057d6f6532bf90e263627a71270b450b800f66cdb48361c (image=looplinesystems/lx-pcs-reporting:0.11.9, name=stag-lx-pcs-reporting-api)
2018-04-06 07:40:37.641205 container attach 5271f4efd27106208057d6f6532bf90e263627a71270b450b800f66cdb48361c (image=looplinesystems/lx-pcs-reporting:0.11.9, name=stag-lx-pcs-reporting-api)
2

I found the solution… somehow. We had datadog-agent running which is tailing our log files.

When I started the container aprox. 1 of 20 times it started but there was a particular folder not linked which was defined as volume.

After restarting datadog-agent it worked well. My suspicion is that datadog-agent somehow had influence on the folder (maybe locked it or whatever) and for this reason docker-compose lost the container which got orphaned then.

I will report this to Datadog - So no real solution at this place.