We currently have multiple VMs running on our server with VMWare. Three of these VMs are running Docker Swarm and have multiple services/containers that connect to a MariaDB. This database is installed on another VM, not in a docker container.
In the database logs we find the following warnings:
3389 [Warning] Aborted connection 3389 to db: '<db>' user: '<user>' host: '<ip>' (Got timeout reading communication packets)
We can see that when a service from node 1 contacts the database, the database has 10 connections to the IP of node 1. When then a service from node 2 contacts the database, all connections to node 1 are immediately dropped.
Our services use Spring Boot 2.
We are running Docker version 18.09.7 on Ubuntu 18.04.2.
What we have tried:
-
using “extra_hosts” in our docker-compose file to contact the database through a DNS name rather than an IP address. This didn’t change anything.
-
using “endpoint_mode: dnsrr” in our compose file. This didn’t allow us to have ports exposed in our container, which we absolutely need.
-
changing the sysctl setting on each of the hosts to increase available sockets and faster reusability of those sockets. This didn’t change anything.
Docker compose file (in reality it holds many more services that are all like this one, they all have the same issue with the database):
version: "3.7"
services:
ovinto-api:
image: <private_registry>/ovinto-api:ovinto-dev
hostname: ovinto-api
ports:
- 8203:8090
deploy:
resources:
limits:
memory: 768M
reservations:
memory: 256M
replicas: 1
update_config:
order: start-first
networks:
- ovinto-route-calculation
extra_hosts:
- "ovinto-mongodb-dev:<ip>"
- "ovinto-mysql-dev:<ip>"
- "ovinto-old-dev:<ip>"
- "ovinto-rabbitmq-dev:<ip>"
- "ovinto-redis-dev:<ip>"
- "ovinto-swarm-dev:<ip>"
- "ovinto-swarm-dev:<ip>"
- "ovinto-swarm-dev:<ip>"
networks:
ovinto-api:
driver: overlay
Has anyone ever had this problem before? Or do you have an idea of what might be giving us this issue? Any advice is appreciated