Docker Community Forums

Share and learn in the Docker community.

Mysql docker will not come up until we restart the container

We are attempting to automate the creation of a VM. We use ansible and docker-compose to create 8 containers for our various services.

When we first create the docker container, the other containers can not connect. We get the message connection refused.

I then pared down the build to be just the one my-sql container. I run the ansible playbook that does the pull, then the start. Here is the log that I find:

sysop@ubuntu-server200:/opt/broker$ docker-compose logs
Attaching to mirth-db
mirth-db    | 2021-03-15 21:15:58+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.23-1debian10 started.
mirth-db    | 2021-03-15 21:15:58+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
mirth-db    | 2021-03-15 21:15:58+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.23-1debian10 started.
mirth-db    | 2021-03-15 21:15:58+00:00 [Note] [Entrypoint]: Initializing database files
mirth-db    | 2021-03-15T21:15:58.162137Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.23) initializing of server in progress as process 42
mirth-db    | 2021-03-15T21:15:58.167230Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
mirth-db    | 2021-03-15T21:16:00.525219Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mirth-db    | 2021-03-15T21:16:06.295226Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
mirth-db    | 2021-03-15 21:16:09+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.23-1debian10 started.
mirth-db    | 2021-03-15 21:16:09+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
mirth-db    | 2021-03-15 21:16:09+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.23-1debian10 started.
mirth-db    | 2021-03-15T21:16:10.484565Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.23) starting as process 1
mirth-db    | 2021-03-15T21:16:10.571715Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
mirth-db    | 2021-03-15T21:16:17.086467Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mirth-db    |
mirth-db    | InnoDB: Progress in percents: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 1002021-03-15T21:16:17.495701Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
mirth-db    | 2021-03-15T21:16:17.905639Z 0 [ERROR] [MY-011947] [InnoDB] Cannot open '/var/lib/mysql/ib_buffer_pool' for reading: No such file or directory
mirth-db    | 2021-03-15T21:16:17.953251Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
mirth-db    | 2021-03-15T21:16:17.953436Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
mirth-db    | 2021-03-15T21:16:17.962165Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
mirth-db    | 2021-03-15T21:16:17.976546Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.23'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.

So far, it looks good to my untrained eye. I’m a bit suspicious that the entrypoint script is called twice. Could the first attempt have crashed? I then run bash on the container, and in there run mysql:

sysop@ubuntu-server200:/opt/broker$ docker-compose exec mirth-db bash
root@89b878947000:/# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.23 MySQL Community Server - GPL

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
ERROR 1449 (HY000): The user specified as a definer ('mysql.infoschema'@'localhost') does not exist
mysql> ^C
mysql> ^DBye
root@89b878947000:/# exit

I have no idea what that error is telling me. I then down and up the docker system:

sysop@ubuntu-server200:/opt/broker$ docker-compose down -v
Stopping mirth-db ... done
Removing mirth-db ... done
Removing network broker_net-mirth
sysop@ubuntu-server200:/opt/broker$ docker-compose up -d
Creating network "broker_net-mirth" with the default driver
Creating mirth-db ... done

And when I try and run mysql again (in the container), it now knows that I asked for a password for root, and it sees I have the predefined databases:

sysop@ubuntu-server200:/opt/broker$ docker-compose exec mirth-db bash
root@636f08b3114e:/# mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
root@636f08b3114e:/# mysql -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.23 MySQL Community Server - GPL

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mirthdb            |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

mysql>

The mysql database container is pretty simple, and is close to what the docker-compose example shows:

version: "3.4"
#
# DBI Services
#
services:
  #
  # Mirth DB
  #
  mirth-db:
    image: mysql:8.0.23
    container_name: mirth-db
    env_file:
      - "{{ dbi_configs_dir }}/mirth-db.env"
    expose:
      - 3306
    networks:
      - net-mirth
    restart: always
#
# Define Networks
#
networks:
  net-mirth:

Nothing special in the env file either. We defined a root password, database and user credentials:

MYSQL_ROOT_PASSWORD="{{ dbi_mirth_db_mysql_root_password }}"
MYSQL_DATABASE={{ dbi_mirth_db_mysql_database }}
MYSQL_USER={{ dbi_mirth_db_mysql_user }}
MYSQL_PASSWORD={{ dbi_mirth_db_mysql_password }}

Can someone please help me to understand why I need to restart the containers on what should be a perfectly good mysql container? Why does mysql apparently not get initialized on the first docker-compose?

After evaluating the answers and studying the topic I’d like to summarize.

The Docker way to upgrade containers seems to be the following:

Application containers should not store application data. This way you can replace app container with its newer version at any time by executing something like this:

docker pull mysql
docker stop my-mysql-container
docker rm my-mysql-container
docker run --name=my-mysql-container --restart=always
-e MYSQL_ROOT_PASSWORD=mypwd -v /my/data/dir:/var/lib/mysql -d mysql
You can store data either on host (in directory mounted as volume) or in special data-only container(s). Read more about it

About volumes (Docker docs)
Tiny Docker Pieces, Loosely Joined (by Tom Offermann)
How to deal with persistent storage (e.g. databases) in Docker (Stack Overflow question)
Upgrading applications (eg. with yum/apt-get upgrade) within containers is considered to be an anti-pattern. Application containers are supposed to be immutable, which shall guarantee reproducible behavior. Some official application images (mysql:5.6 in particular) are not even designed to self-update (apt-get upgrade won’t work).

I’d like to thank everybody who gave their answers, so we could see all different approaches.

Not sure this answered my question. Perhaps this was meant for another topic? Not sure it’s a summary to the answers to my problem, since there have been none so far.

In my case, we had been storing data on a host container, but that statement got removed in my stripping down of the test case. With or without the host volumes had no effect for me.

One more data point. I and another co-worker are experiencing this problem, whereas the author of the playbook & docker-container is not. We’re all running Ubuntu 18 under a VMWare Workstation 16 VM. All the host VMs were minimal Linux systems built straight from the ISO images, but built up by different people.