Docker Community Forums

Share and learn in the Docker community.

Issue with docker-compose: container did not start before the specified timeout


(Mikulicheugene) #1

Hello
I’ll very appreciative if you help me with my issue.

OS: 14.04.3 LTS (GNU/Linux 3.13.0-74-generic x86_64)
AWS AMI: ami-fce3c696 (Those, that AWS suggest on Launch instance page)
docker-compose version: 1.7.1, build 0a9ab35
docker version: 1.11.1, build 5604cbe

I’m trying to launch docker-compose application via Chef on AWS EC2 instance launched with Vagrant (instance launched on EC2 with Vagrant and provisioned with Chef).
Now I’m getting different results with my command (sometimes success, but most likely it fails).

We can see that docker_jenkins-data_1 container exited with code 128 and container with Jenkins itself has not even been created (see details below).
As I said command can succeed, but it can fail with 128 exit code of other containers

More details are below

docker-compose.yml:

version: '2'
services:
  mongodb-data:
    image: {my private image for mongodb data}
    command: echo 'Data Container for mongodb'
  mongodb:
    image: mongo:3.2
    ports: 
     - "27017:27017"
    hostname: mongodb
    volumes_from: 
     - "mongodb-data"
  jenkins-data:
    image: {my private image for Jenkins data}
    command: echo 'Data Container for jenkins'
  bare-repo:
    image: {my private image for git bare repo}
    command: echo 'Data Container for repo'
  jenkins:
    image: {my private image for Jenkins}
    ports: 
     - "8080:8080"
     - "50000:50000"
    hostname: jenkins
    volumes_from: 
     - "jenkins-data"
     - "bare-repo"
  rabbitmq:
    image: {my private image for RabbitMQ}
    ports: 
     - "15672:15672"
    hostname: rabbitmq
    environment:
      RABBITMQ_DEFAULT_USER: <%= @rabbitmq_user %>
      RABBITMQ_DEFAULT_PASS: <%= @rabbitmq_password %>
      RABBITMQ_DEFAULT_VHOST: <%= @rabbitmq_vhost %>
    volumes:
     - ./rabbitmq/logs:/var/log/rabbitmq
  graylog:
    image: {my private image for Graylog}
    ports: 
     - "9000:9000"
     - "9200:9200"
     - "12900:12900"
    hostname: graylog
    environment:
      GRAYLOG_PASSWORD: <%= @graylog_password %>
      GRAYLOG_SERVER_SECRET: <%= @graylog_secret %>
    volumes:
     - ./graylog/logs:/var/log/graylog

Docker data volumes containers based on busybox.

Chef recipe:

<...>
template "/docker/docker-compose.yml" do
  source "infrastructure.compose.erb"
  variables ({
    :rabbitmq_user     => {CENSORED}
    :rabbitmq_password => {CENSORED}
    :rabbitmq_vhost    => {CENSORED}
    :graylog_password  => {CENSORED}
    :graylog_secret    => {CENSORED}
  })
end

secret = Chef::EncryptedDataBagItem.load_secret({CENSORED})
docker_creds = Chef::EncryptedDataBagItem.load({CENSORED})

execute 'login' do
  command "docker login -e #{docker_creds["docker_email"]} -u #{docker_creds["docker_user"]} -p #{docker_creds["docker_password"]}"
end

ENV['COMPOSE_HTTP_TIMEOUT'] = '1200'

execute "compose up" do
  cwd "/docker"
  command "docker-compose up -d"
  timeout 1200
end

Result:

<...>
==> infrastructure_dev: Creating docker_mongodb-data_1
==> infrastructure_dev: Creating docker_jenkins-data_1
==> infrastructure_dev: Creating docker_graylog_1
==> infrastructure_dev: Creating docker_rabbitmq_1
==> infrastructure_dev: Creating docker_bare-repo_1
==> infrastructure_dev: Creating docker_mongodb_1
==> infrastructure_dev:
==> infrastructure_dev: ERROR: for jenkins-data  rpc error: code = 2 desc = "containerd: container did not start before the specified timeout"
==> infrastructure_dev: Traceback (most recent call last):
==> infrastructure_dev:   File "<string>", line 3, in <module>
==> infrastructure_dev:   File "compose/cli/main.py", line 63, in main
==> infrastructure_dev: AttributeError: 'ProjectError' object has no attribute 'msg'
==> infrastructure_dev: docker-compose returned -1
==> infrastructure_dev: ---- End output of docker-compose up -d ----
==> infrastructure_dev: Ran docker-compose up -d returned 255
==> infrastructure_dev: [2016-05-24T06:48:52+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
ubuntu@docker-infrastructure:/docker$ docker-compose ps
        Name                       Command                State                                                                    Ports
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
docker_bare-repo_1      echo Data Container for repo     Exit 0
docker_graylog_1        /opt/graylog/embedded/shar ...   Up         12201/tcp, 12201/udp, 0.0.0.0:12900->12900/tcp, 4001/tcp, 443/tcp, 514/tcp, 514/udp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:9200->9200/tcp
docker_jenkins-data_1   echo Data Container for je ...   Exit 128
docker_mongodb-data_1   echo Data Container for mo ...   Exit 0
docker_mongodb_1        /entrypoint.sh mongod            Up         0.0.0.0:27017->27017/tcp
docker_rabbitmq_1       /docker-entrypoint.sh rabb ...   Up         15671/tcp, 0.0.0.0:15672->15672/tcp, 25672/tcp, 4369/tcp, 5671/tcp, 5672/tcp

Those containers, that exited with 128 code marked as “Created” in docker ps -a

ubuntu@docker-infrastructure:/docker$ docker ps -a
CONTAINER ID        IMAGE        COMMAND                  CREATED             STATUS                      PORTS                      NAMES
47f3b5f269b9        mongo:3.2    "/entrypoint.sh mongo"   33 minutes ago      Up 33 minutes               0.0.0.0:27017->27017/tcp   docker_mongodb_1
4ced312fd60f        {CENSORED}   "/docker-entrypoint.s"   35 minutes ago      Created                                                docker_rabbitmq_1
5f57e3f9ab81        {CENSORED}   "echo 'Data Container"   35 minutes ago      Exited (0) 33 minutes ago                              docker_mongodb-data_1
7b699a99658a        {CENSORED}   "echo 'Data Container"   35 minutes ago      Created                                                docker_jenkins-data_1
808152e0915a        {CENSORED}   "/opt/graylog/embedde"   35 minutes ago      Created                                                docker_graylog_1
ef91c7c7e873        {CENSORED}   "echo 'Data Container"   35 minutes ago      Exited (0) 34 minutes ago                              docker_bare-repo_1

ubuntu@docker-infrastructure:/docker$ docker-compose ps
        Name                       Command                State              Ports
--------------------------------------------------------------------------------------------
docker_bare-repo_1      echo Data Container for repo     Exit 0
docker_graylog_1        /opt/graylog/embedded/shar ...   Exit 128
docker_jenkins-data_1   echo Data Container for je ...   Exit 128
docker_mongodb-data_1   echo Data Container for mo ...   Exit 0
docker_mongodb_1        /entrypoint.sh mongod            Up         0.0.0.0:27017->27017/tcp
docker_rabbitmq_1       /docker-entrypoint.sh rabb ...   Exit 128