Aborted connection 3389 to db: (Got timeout reading communication packets) when calling mariaDB from Docker Swarm services

We currently have multiple VMs running on our server with VMWare. Three of these VMs are running Docker Swarm and have multiple services/containers that connect to a MariaDB. This database is installed on another VM, not in a docker container.

In the database logs we find the following warnings:

3389 [Warning] Aborted connection 3389 to db: '<db>' user: '<user>' host: '<ip>' (Got timeout reading communication packets)

We can see that when a service from node 1 contacts the database, the database has 10 connections to the IP of node 1. When then a service from node 2 contacts the database, all connections to node 1 are immediately dropped.

Our services use Spring Boot 2.
We are running Docker version 18.09.7 on Ubuntu 18.04.2.

What we have tried:

  • using “extra_hosts” in our docker-compose file to contact the database through a DNS name rather than an IP address. This didn’t change anything.

  • using “endpoint_mode: dnsrr” in our compose file. This didn’t allow us to have ports exposed in our container, which we absolutely need.

  • changing the sysctl setting on each of the hosts to increase available sockets and faster reusability of those sockets. This didn’t change anything.

Docker compose file (in reality it holds many more services that are all like this one, they all have the same issue with the database):

version: "3.7"
services:

        ovinto-api:
                image: <private_registry>/ovinto-api:ovinto-dev
                hostname: ovinto-api
                ports:
                        - 8203:8090
                deploy:
                        resources:
                                limits:
                                        memory: 768M
                                reservations:
                                        memory: 256M
                        replicas: 1
                        update_config:
                                order: start-first
                networks:
                        - ovinto-route-calculation
                extra_hosts:
                        - "ovinto-mongodb-dev:<ip>"
                        - "ovinto-mysql-dev:<ip>"
                        - "ovinto-old-dev:<ip>"
                        - "ovinto-rabbitmq-dev:<ip>"
                        - "ovinto-redis-dev:<ip>"
                        - "ovinto-swarm-dev:<ip>"
                        - "ovinto-swarm-dev:<ip>"
                        - "ovinto-swarm-dev:<ip>"


networks:
        ovinto-api:
                driver: overlay

Has anyone ever had this problem before? Or do you have an idea of what might be giving us this issue? Any advice is appreciated

did you get a solution, facing the same problem here

Hello,

Yes I did find an answer in fact.

It turned out that the timeout of the connections to the MariaDB were set to longer on the application side than on the database side. And so, Spring would still try to connect to the database using a timed out connection. Why this happens, I don’t know exactly. However, I did fix it by giving the connections on the application side a smaller timeout than on the database side.

Hope this helps.