Hello everybody,
Our team is working on a new project that is still not deployed on production and we have decided to give docker swarm a try.
We have successfully created a swarm with several containers and we can say that, in general, is working fine. Taking into account that we are still on the first dev stages, our configuration so far is really simple (docker-compose.yml): one network, one volume, 6 services 1 replica each (with the intention to have more replicas in the future), all services are rest services, only 1 node (with the intention to have more nodes in the future).
The only strange we’ve noticed, and that is worrying us, is that after the swarm has been idle for a while (e.g. no incoming request on any container on a dev server) we experience high latency while reaching the different services in our swarm, but only for the first request on each of the services (only the first request is slow after being idle). So for example:
Swarm: Service A, Service B
Idle 30 mins --> no incoming requests for any of the services during 30 mins
Incoming request (non-cacheable request) to Service A --> responds after (aprox) 20 secs
Incoming request (non-cacheable request) to Service A --> responds immediately
... All further hits to Service A respond immediately
Incoming request (non-cacheable request) to Service B --> responds after (aprox) 20 secs
Incoming request (non-cacheable request) to Service B --> responds immediately
... All further hits to Service B respond immediately
We entered the containers (execute bash) and noticed that:
- Ping to other containers IP responds immediately
- Ping to other containers name (dns) has the mentioned delay (high latency)
- The container name (dns) resolution happens immediately (so it is not latency at dns level)
More details:
- Host OS: Ubuntu 16.04
- Docker version: 17.06.0-ce
Anybody has an idea of what could be going on here?