Regovar project: need help to dockerize a complexe application

General Discussions General
1

Hello,

I would like to encapsulate an application in a docker image. But this is a complex application, and I would like your opinion on several points:

The open source application Regovar is a tool for analyzing genetic data.

  • Regovar is a web server in python 3.6,
  • It expose REST API (HTTP or HTTPS),
  • Use a postgresql 9.6 database
  • Is able to create, start and monitor scripts (bioinformatic pipelines) via docker containers.
  • The serveur and the pipelines need to share lot of heavy files (genetic databases).
  • Pipelines (in docker containers) can notify regovar seveur by using the provided url to the Rest API

My (firsts) questions:
1- Is it possible to create a Docker container that uses the host Docker service to create/pause/start/stop/rm other containers ?

2- Is it better to create a single container with the python application and the postgresql server ? Or do we have to create two distinct containers for that ?

You can find the dockerfile I started to do here.

2

To 1. Mount the docker socket into your container. Then your application should be able to communicate with the local docker demon:

  docker run -v /var/run/docker.sock:/var/run/docker.sock

To 2. It’s hard to maintain multiple processes inside the same docker container. Reasons:

  • Logging: Multiple processes produce multiple logs.
  • Restarting: Which process should trigger a restart?
  • Monitoring: Monitoring container is easy, monitoring processes inside containers is not.
  • All in all: It’s possible (See runit, supervisord) but your entering a domain without direct tool support.

At least, you can install everything in one image and start multiple containers:

docker run ... -yourParameters- ... yourImage pythonApp
docker run ... -yourParameters- ... yourImage postgres
3

Thanks for your answer :slight_smile:

To 1, I was reading an interresting blog article that give more details about the solution you propose.

To 2, ok I understand, and in addition, I can use the official docker for postgres which is clean and maintained by the community.

So, I will now dive into docker-compose to do what we said :metal:

4

Hi,

I’m stuck with another problem.
So I’m creating 2 containers, one for the db, and one for the python app.

But I can not connect the python app to the db…
Below the source of my docker-compose

version: '3'
services:
  regovar_pg:
    container_name: regovar_pg
    image: postgres
    restart: always
    volumes:
      - /var/regovar/pgdata:/var/lib/postgresql/data
      - /var/regovar/databases:/var/regovar/databases
    environment:
      - POSTGRES_USER=regovar
      - POSTGRES_PASSWORD=regovar
      - POSTGRES_DB=regovar
    ports:
      - "5432:5432"

  regovar_app:
    container_name: regovar_app
    build: .
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/regovar:/var/regovar
    ports:
      - "8500:8500"
    depends_on:
      - regovar_pg

According to what I read on the web, the docker-compose seems good, so I supposed that maybe the probleme is in my python code…
I tried to connect to 0.0.0.0:5432, localhost:5432, regovar_pg:5432, but same error…

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool.py", line 345, in unique_connection
    return _ConnectionFairy._checkout(self)
  ......
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 410, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
        Is the server running on host "172.0.0.1" and accepting
        TCP/IP connections on port 5432?

Have someone an idea why it didn’t works ?

5

You can’t connect over localhost. Docker containers communicate through a network and docker-compose is setting up one for you! The hostnames of the containers are the same as their service name or container name. E.g.: regovar_pg:5432 for your database.

More details: compose/networking

But your problem seems to be misconfiguration of the postgres database: Important note: you must set listen_addresses = '*'so that other containers will be able to access postgres. See here: Docker Hub Postgres

1 Like
6

Thanks, I fixed the postgresql config. Now my regovar_pg container is listening outside

$ docker exec -it regovar_pg psql -U regovar -c "SHOW listen_addresses"
 listen_addresses
------------------
 *
(1 row)

And my docker-compose look like that now:

version: '3'
services:

  regovar_pg:
    container_name: regovar_pg
    image: postgres
    restart: always
    volumes:
      - /var/regovar/pgdata:/var/lib/postgresql/data
      - /var/regovar/databases:/var/regovar/databases
      - /etc/passwd:/etc/passwd:ro
      - /var/regovar/config/postgres.conf:/etc/postgresql/postgresql.conf
    environment:
      - POSTGRES_USER=regovar
      - POSTGRES_PASSWORD=regovar
      - POSTGRES_DB=regovar
    ports:
      - "5432:5432"
    user: regovar_user
    command: postgres -c config_file=/etc/postgresql/postgresql.conf


  regovar_app:
    container_name: regovar_app
    build: .
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/regovar:/var/regovar
      - localgit/regovar:/var/regovar_app
    ports:
      - "8500:8500"
    depends_on:
      - regovar_pg
    user: regovar_user
    working_dir: /var/regovar_app

But the connexion between my 2 containers is still not working. I noticed, the hostname “regovar_pg” is not defined in the container regovar_app.

docker run -a stdin -a stdout -i -t config_regovar_app
# ping regovar_pg
ping: unknown host

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
550: eth0@if551: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

# cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2      66c3423a8ca0

# ping 66c3423a8ca0
PING 66c3423a8ca0 (172.17.0.2): 56 data bytes
64 bytes from 172.17.0.2: icmp_seq=0 ttl=64 time=0.139 ms
64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.067 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=64 time=0.070 ms
^C--- 66c3423a8ca0 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.067/0.092/0.139/0.033 ms

# apt install postgresql
Reading package lists... Done
...
# psql -h 66c3423a8ca0 -p 5432 -d regovar -U regovar -c "select * from parameter"
psql: could not connect to server: Connection refused
        Is the server running on host "66c3423a8ca0" (172.17.0.2) and accepting
        TCP/IP connections on port 5432?

And below, the result of “Network” part for the container’s inspection:

$ docker container inspect regovar_db
"NetworkSettings": 
{
    "Bridge": "",
    "SandboxID": "8ee237ddd48650becd79cef17c242b9ef8cd4115f6b21323e12fbfc09ed6d829",
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "Ports": {
        "5432/tcp": [
            {
                "HostIp": "0.0.0.0",
                "HostPort": "5432"
            }
        ]
    },
    "SandboxKey": "/var/run/docker/netns/8ee237ddd486",
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "",
    "Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "",
    "IPPrefixLen": 0,
    "IPv6Gateway": "",
    "MacAddress": "",
    "Networks": {
        "config_default": {
            "IPAMConfig": null,
            "Links": null,
            "Aliases": [
                "regovar_pg",
                "dd37e99fc210"
            ],
            "NetworkID": "db6c724411d26d638ab9c950cc2e21bd2561f65dbe1db558968fb801883894b4",
            "EndpointID": "9312503465f996234a8aeb0315c60edef5f0af6b290a619736960aea7ed08dc9",
            "Gateway": "172.19.0.1",
            "IPAddress": "172.19.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "02:42:ac:13:00:02",
            "DriverOpts": null
        }
    }
}

$ docker container inspect regovar_app
"NetworkSettings": 
{
    "Bridge": "",
    "SandboxID": "1a396aa99d8d6dac964757ba8f30c8db16ac2da9bc945882ff4f4abc6e9f5a81",
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "Ports": {},
    "SandboxKey": "/var/run/docker/netns/1a396aa99d8d",
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "",
    "Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "",
    "IPPrefixLen": 0,
    "IPv6Gateway": "",
    "MacAddress": "",
    "Networks": {
        "config_default": {
            "IPAMConfig": null,
            "Links": null,
            "Aliases": [
                "6e5364352389",
                "regovar_app"
            ],
            "NetworkID": "db6c724411d26d638ab9c950cc2e21bd2561f65dbe1db558968fb801883894b4",
            "EndpointID": "",
            "Gateway": "",
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "",
            "DriverOpts": null
        }
    }
}