One container cannot see another


This is my first post to the community, I hope I’ve managed to abide by the guidelines and provide sufficient information. The summary is that after upgrading docker, our containers can no longer “see” the oauth container, which for some reason the container attempts to resolve by using the ingress ip address. More details…

Current Platform

Our Spring Cloud based platform has several containers that are launched in a swarm via “docker service” and run in a Docker Engine 1.12 environment. The containers are configured to talk to an oauth container for authentication and authorization purposes:

     url: https://auth:9999

Upgraded Platform

We are in the process of converting the platform to launch in a swarm via “docker stack deploy” and also with docker-compose files as opposed to laboriously specifying the options at the command line. The target docker environment is Docker engine: 18.09.2 .

However the new configuration suffers from a strange failure: the containers are no longer able to access the oauth container, we get the stack trace below from spring cloud. The strange thing is that it somehow given the ingress ip address for the container which is

 ... 110 common frames omitted
Caused by: java.lang.RuntimeException: org.apache.http.conn.ConnectTimeoutException: Connect to [/] failed: connect timed out
 at rx.exceptions.Exceptions.propagate(
 at rx.observables.BlockingObservable.blockForSingle(
 at rx.observables.BlockingObservable.single(
 ... 170 common frames omitted

More Information

From docker inspect on any of our containers two overlay networks can be seen (many details omitted):

"Networks": {
	"app_default": {
		"IPAMConfig": {
			"IPv4Address": ""
	"ingress": {
		"IPAMConfig": {
			"IPv4Address": ""

If only docker had dispensed the app_default ip address of to spring cloud then all would have been well because I’ve confirmed by opening a shell into the container and could successfully connect with telnet 9999. The same telnet command did not work for telnet 9999, however ping did work. So it would appear accessing the port is a problem. Our Operations department are adamant that there are no firewall rules working against us.

One more important point is that as another experiment we kept our platform the same and only upgraded docker and noticed the failure started when we upgraded to just a few version beyond Docker Engine 1.12 - I’m afraid I don’t have the exact version information.


We don’t have much experience of overlay networks so any pointers would be gratefully received.

For example, why are there two overlay networks present on each of our containers? I thought the ingress network was the only one needed since its job is to facilitate docker daemon communication.

Also how come the ingress ip address was being selected and passed through to our containers rather than the app_default?

Many thanks