Docker Community Forums

Share and learn in the Docker community.

[Resolved] Ever since I upgraded docker engine to 1.11 my links are on and off


(Mwaaas) #1

I upgraded docker version to 1.11.1-cs1 my links do not work, any one with a similar problem


(Pukkasoftware) #2

Yes, I’m experiencing something similar. Specifically it appears that referring to the service name (e.g. “app”) doesn’t resolve to the correct IP address of the container (e.g. “app-1”) which according to the documentation should be Round-Robin on the containers in the service.

I wrote to support about this and am waiting back on details. It’s totally taken our entire stack offline and we can’t run our app any longer.


(Dvirf) #3

we have the same problem


(Fernando Mayo) #4

Are you using alpine as the base image? There’s an issue with alpine + docker 1.11 that we are working to solve as we speak


(Mwaaas) #5

@fermayo thought a container once it works in development it will work it will work in production, didn’t know there is another kind of dependencies with the docker version


(Iteamnetworkdc) #6

@fermayo What is the status of the Alpine fix?


(Fernando Mayo) #7

@mwaaas @iteamnetworkdc Fix is in production. You will have to redeploy any alpine-based services. Can you please confirm it works for you?


(Iteamnetworkdc) #8

@fermayo Works great! Thanks for the quick turn around and for responding. Much appreciated! :punch:


(Fernando Mayo) #9

Awesome. Thanks for confirming.


(Sam) #10

Tracking this issue as nginx:alpine seems to be affected by the new changes.

Even redeploying just now, nginx reports being unable to find a linked hostname (in my case, ‘web-staging’).

nginx:latest works normally (based on Debian), but nginx:alpine (same nginx version, but on Alpine) does not work (terminates on startup). Exact same config.

nginx: [emerg] host not found in upstream "<stack-name>"

The /etc/resolv.conf file contains:


options ndots:1 ndots:0 

(Fernando Mayo) #11

nginx:alpine is running Alpine 3.3 which does not have the fix for DNS search (it was introduced in Alpine 3.4).


(Sam) #12

Thanks for the info. Thought it was a Docker Cloud issue but indeed, awaiting upstream nginx maintainers to publish based on Alpine 3.4. These delays kind of explain why many people publish their own images rather than use official variants which are often slow to update.


(Pukkasoftware) #13

I am not using Alpine Nginx and still experiencing this issue tonight even after redeploy. I’m using base from nginx image which is based on NGINX_VERSION=1.9.2-1~jessie.

I updated my nginx to the latest NGINX_VERSION=1.11.1-1~jessie and still no dice. The service name resolves to a bogus IP:

root@nginx-app-1:/# ping app
PING app (127.0.53.53): 56 data bytes
64 bytes from 127.0.53.53: icmp_seq=0 ttl=64 time=0.031 ms
64 bytes from 127.0.53.53: icmp_seq=1 ttl=64 time=0.050 ms


(Pukkasoftware) #14

More details… something is borked here:

root@nginx-app-1:/# cat /etc/resolv.conf
search f6387388-7647-4305-a667-4f89a3e85c74.local.dockerapp.io
nameserver 127.0.0.11
options ndots:1 ndots:0

root@nginx-app-1:/# ping app
PING app (127.0.53.53): 56 data bytes
64 bytes from 127.0.53.53: icmp_seq=0 ttl=64 time=0.058 ms
64 bytes from 127.0.53.53: icmp_seq=1 ttl=64 time=0.048 ms

root@nginx-app-1:/# telnet app 8500
Trying 127.0.53.53…
telnet: Unable to connect to remote host: Connection refused

root@nginx-app-1:/# ping app.f6387388-7647-4305-a667-4f89a3e85c74.local.dockerap
PING app.f6387388-7647-4305-a667-4f89a3e85c74.local.dockerapp.io (10.7.0.12): 56 data bytes
64 bytes from 10.7.0.12: icmp_seq=0 ttl=64 time=0.063 ms
64 bytes from 10.7.0.12: icmp_seq=1 ttl=64 time=0.070 ms

root@nginx-app-1:/# telnet app.f6387388-7647-4305-a667-4f89a3e85c74.local.docker
Trying 10.7.0.12…
Connected to app.f6387388-7647-4305-a667-4f89a3e85c74.local.dockerapp.io.
Escape character is ‘^]’.
^]
telnet> close
Connection closed.

So the service name “app” resolves to 127.0.53.53 address but if I append the domain name which is in resolv.conf, it works and returns the proper 10.7.0.12 host running my application server (which I can successfully telnet to).

Further work tonight has this apparently working on Docker Cloud. It boils down to “app” is a seemingly reserved service name which always resolves to 127.0.53.53 (a bogus IP). I have changed my service names to “appserver” and it works fine. When I ping an undefined host like “dbserver”, I get “host not found”. But even with app undefined, I still get ping responses from 127.0.53.53.


NewUI: Link Aliases no longer working
(Chris Hiestand) #15

I’m not sure if it’s related, but I used to have that problem until I upgraded to docker-compose file format version 2.


(Fernando Mayo) #16

The “127.0.53.53” IP that you are seeing is a placeholder for the “.app” TLD: https://www.icann.org/resources/pages/name-collision-2013-12-06-en

This means that when you pinged “app”, the DNS lookup of “app.f6387388-7647-4305-a667-4f89a3e85c74.local.dockerapp.io” did not resolve and it tried to resolve “app” directly, to which the DNS resolvers returned “127.0.53.53”.

I have tried to reproduce this scenario in both alpine and non-alpine based images (the scenario where “app” is defined in your stack, but you still get 127.0.53.53), and couldn’t reproduce it. Which nginx image were you using?


(Pukkasoftware) #17

We are using the official Nginx image with our custom configuration baked in. As indicated above, we’ve tried 1.9.2-1 and 1.11.1-1 both on jessie.

This is a problem that ping app would try to instead ping .app. Perhaps that is the actual problem here but as I indicated above ping app.f6387388-7647-4305-a667-4f89a3e85c74.local.dockerapp.io did work.

I get this behavior locally as well in my Nginx container after changing the name. ping app still returns back the 127.0.53.53 IP address. I wouldn’t expect ping com to try and ping .com… not sure if that is a resolver issue or what?


(Fernando Mayo) #18

The official nginx:alpine image uses Alpine 3.3 which does not have the DNS search fix (that ships with Alpine 3.4). Alpine 3.3 and below are not compatible with our service discovery.


(Pukkasoftware) #19

We are not using Alpine, we are using “nginx”, the Jessie-based distro: https://github.com/nginxinc/docker-nginx/blob/11fc019b2be3ad51ba5d097b1857a099c4056213/mainline/jessie/Dockerfile


(Ahansson89) #20

I have the same issue with nginx jessie based distro. I will try to spin up an alpine nginx container tomorrow and see if that solves it.