DNS update for a running Docker container

[rwanwork (Ray)]

Dear all,

I’m really new with Docker, but I’ve been tasked with looking after a running container and I’ve hit a brick wall.

At first, it and the host machine (Ubuntu 18.04) was running fine. Let’s say its hostname was www.example.com and it had an IP address of 123.123.123.123 .

And then, my organisation required us to change to a private IP address. Let’s say 192.168.123.123 .

On the host machine, I released and got the new private IP address fine via DHCP. I can ssh in and, if I installed Apache on it, then I can go to www.example.com with no problems.

In our case, several Docker containers are running. One is running Nginx and another is our app in question (written using Ruby on Rails). Previously, http/https traffic would be filtered using Nginx and sent off to the app.

Now, I’m completely unable to access this app. When I go to the site, I’m getting an Nginx problem about an upstream connection. There is no problem with other Docker containers on the computer… When I tail the logs for the app, it is still trying to access 123.123.123.123. But, on the host machine, I can ping , etc. and I would get the new local IP address.

Perhaps there are other issues that I am facing. But, I think I shouldn’t see 123.123.123.123 within the running Docker containers’ log files any more. I thought it would “figure it out” after some time has passed, but half a day has passed and it still doesn’t use the new IP address.

So far, I’ve tried:

• Flushing the host computers’ DNS (sudo systemd-resolve --flush-caches for Ubuntu 18.04)
• Manually adding the new local address to the host computer’s /etc/hosts file
• Stopping and restarting the Docker container

Personally, I don’t know Ruby on Rails. So, I don’t have the knowledge to go into the code and fix it

Any suggestions would be appreciated! There might be other issues at play that are preventing me from accessing the application, but so far, the fact that the application is still trying to access the old IP address seems to be the most noticeable problem.

Thank you in advance!

Ray

[meyay (Metin Y.)]

You might want to check the nginx.conf of your nginx server. Seems the host ip is configured as upstream for whatever reason.

If nginx is running as native service on the host, didn’t they use 127.0.0.1 instead of the host ip?
If nginx is running as inside a container, it makes even less sense to use the host ip - docker’s user definied network have build-in dns-based service discovery for that.

[rwanwork (Ray)]

Hmmmm, I didn’t design this architecture so I don’t know if it all makes sense.

So, nginx is not running on the host computer. It is definitely another Docker container.

I think I see what you mean, though. The nginx container should not be aware of the global IP addresses (whether it was the old 123.123.123.123 or the new 192.168.123.123). It should know the Ruby on Rails application by a Docker bridge and just send packets to some IP address within that bridge (i.e., a 172.17.X.Y address).

Is this what you mean?

In any case, what you’re saying which “doesn’t make sense” seems to be what’s happening. I guess when the client makes a request to [some link], Nginx sends the client to the Container. But the Container’s pages refers to itself using that same link…

If I removed the Nginx container and installed Nginx on the host system, would this solve my problem? This would remove “one layer” in this complex system. The host system knows about the new IP address. Hmmmm…let me give this a try… Thank you!

[meyay (Metin Y.)]

@rwanwork

The nginx container should not be aware of the global IP addresses (whether it was the old 123.123.123.123 or the new 192.168.123.123).

yep, i shouldn’t need to care about any of the host ips.

@rwanwork

It should know the Ruby on Rails application by a Docker bridge and just send packets to some IP address within that bridge (i.e., a 172.17.X.Y address).

Almost correct. It should not need to care about the container ip either. It should rely for the container or service name, which get registered in the buildin dns-based service discovery and should be used to resolve the current container ip. Of course this only works in a user defined network (= not the default bridge).

@rwanwork

But the Container’s pages refers to itself using that same link…

It was setup to use the published host ports of the target containers. Instead nginx is usually used as central entry point into a container network, and all nginx to container communication takes place using a container network.

@rwanwork

If I removed the Nginx container and installed Nginx on the host system, would this solve my problem?

Please don’t!

Can you share the docker ru command or compos file content used to start nginx and one more container + the nginx.conf? We can use it as an example on how to configure it properly.

[rwanwork (Ray)]

@meyay

@rwanwork

But the Container’s pages refers to itself using that same link…

It was setup to use the published host ports of the target containers. Instead nginx is usually used as central entry point into a container network, and all nginx to container communication takes place using a container network.

Oh…I see! So this set up does make sense to you? I have no documentation from the developer explaining why he chose to do what he did.

@meyay

@rwanwork

If I removed the Nginx container and installed Nginx on the host system, would this solve my problem?

Please don’t!

Can you share the docker ru command or compos file content used to start nginx and one more container + the nginx.conf? We can use it as an example on how to configure it properly.

Oh, ok! Thank you for offering to take a look!

So, this is the docker run command for the Nginx:

docker run --name my-nginx -v /srv/nginx/conf:/etc/nginx/conf.d:Z -v /srv/nginx/www:/var/www -v /srv/nginx/log:/var/log/nginx -v /srv/nginx/sys:/sys -v /cert:/cert -p 80:80 -p 443:443 -d nginx

This is the nginx.conf file. I’ve simplified it to just the application that’s causing me problems. Basically, there are other applications and there are two server groups for each of them, one for port 80 and 443.

myapp.org, myserver.org, and another.myapp.org are domain names that point to the same computer. All 3 names are known outside the host computer.

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
  server {
    listen 80;
    server_name myapp.org; 

    rewrite ^(.*) https://$server_name$request_uri permanent
  }

  server {
    listen 443 ssl;
    server_name myapp.org;

    ssl_certificate      /cert/fullchain.pem;
    ssl_certificate_key  /cert/privkey.pem;
    error_log  /var/log/nginx/myapp-error.log;        

    location / {
      proxy_redirect off;
      proxy_pass http://myserver.org:12000;
      proxy_set_header Host $host;
      proxy_set_header X-real-ip $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }

  server {
    listen 80 default;

    rewrite ^(.*) http://another.myapp.org permanent;
  }
}

The application myapp.org is started using this command:

docker run --restart=always -p 12000:8080 -v /srv/myapp/attachment:/app/public/attachment -v /srv/myapp/system:/app/public/system -v /srv/myapp/log:/app/log --name myapp -d myappdb unicorn_rails -E production -c /app/config/unicron.rb

The Dockerfile (not sure if that is important) looks like this:

# Base image with ruby 2.2.0
FROM ruby:2.3

# Install required libraries and dependencies
RUN apt-get update && apt-get install -qy nodejs postgresql-client sqlite3 --no-install-recommends && rm -rf /var/lib/apt/lists/*

# Set Rails version
ENV RAILS_VERSION 4.2.3 

# Install Rails
RUN gem install rails --version "$RAILS_VERSION"

# Create directory from where the code will run 
RUN mkdir /app  
WORKDIR /app

# Make webserver reachable to the outside world
EXPOSE 8080 

# Set ENV variables
ENV PORT=8080

# Install the necessary gems 
ADD Gemfile /app/Gemfile  
ADD Gemfile.lock /app/Gemfile.lock  
RUN bundle install --without development test

# Add rails project (from same dir as Dockerfile) to project directory
ADD ./ /app
VOLUME /app/db
VOLUME /app/log

# Run rake tasks
RUN RAILS_ENV=production rake assets:precompile

# Start the web app

There are other issues which I have mentioned, but didn’t properly emphasize. The host machine is an Ubuntu 18.04 machine. So, arguably, some of the software is a bit old. Updating is a problem for another day…

Another problem (which I don’t understand) is if I do a docker restart nginx, it restarts fine but it no longer listens to ports 80 and 443. I don’t know what this means as I don’t seen an error with the nginx.conf. (i.e., when I do a port scan with nmap after restarting Nginx, it says that the ports are closed).

Any help would be appreciated! Thank you!

Ray

[meyay (Metin Y.)]

@rwanwork

Oh…I see! So this set up does make sense to you?

The first one? No The second one indeed does!

@rwanwork

      proxy_pass http://myserver.org:12000;

I would have expected to see the ip here. Instead, it seems to use a domain name. So either the nginx server was not restarted after changing the dns entry and host ip (nginx is known to cache resolved dns entries until is stopped), or the host uses a different dns server, or in case of the default bridge network has a local entry in /etc/hosts.

Since nginx is running in a container, you not simply use proxy_pass http://localhost:12000 (which would have solved it, if nginx would be service on the host). Though, as the containers are attached to the default bridge, you can not use service discovery.

If you’d create a container network and attach the containers to the network, you could use the container name instead of myserver.org, but it requires an additional step to mitigate the indefinitely caching of resolved dns entries.

# create the user defined network
docker network create web

docker run \
  --name my-nginx \
  --network web \
  --restart=always \
  -v /srv/nginx/conf:/etc/nginx/conf.d:Z \
  -v /srv/nginx/www:/var/www \
  -v /srv/nginx/log:/var/log/nginx \
  -v /srv/nginx/sys:/sys \
  -v /cert:/cert \
  -p 80:80 \
  -p 443:443 \
  -d \
  nginx

docker run \
  --name myapp \
  --network web \
  --restart=always \
  -v /srv/myapp/attachment:/app/public/attachment \
  -v /srv/myapp/system:/app/public/system \
  -v /srv/myapp/log:/app/log \
  -p 12000:8080 \
  -d 
  myappdb unicorn_rails -E production -c /app/config/unicron.rb

This allows to use proxy_pass http://myapp:8080; instead. It uses the container name, and application port from inside the container. Furthermore, it allows removing the published port on the container, as it’s already accessible through nginx.

You can mitigate the dns caching issue by applying the settings discussed in the posed to your nginx.conf: NGINX swarm redeploy timeouts - #5 by meyay

That should do the trick.

You should consider migrating your docker run commands into docker compose files. As a positive side effect you can version the compose files in git and keep track of changes.

Update: I forget to address the restart problem. A container restart is nothing else than docker stop && docker start, thus it shouldn’t behave any differently.

[meyay (Metin Y.)]

@meyay

You should consider migrating your docker run commands into docker compose files. As a positive side effect you can version the compose files in git and keep track of changes.

a docker-compose.yml would look like this:

services:

  my-nginx:
    image: nginx
    container_name: my-nginx
    restart: always
    volumes:
    - /srv/nginx/conf:/etc/nginx/conf.d:Z
    - /srv/nginx/www:/var/www
    - /srv/nginx/log:/var/log/nginx
    - /srv/nginx/sys:/sys
    - /cert:/cert
    ports:
    - 80:80
    - 443:443

  myapp:
    image: myappdb
    container_name: myapp
    restart: always
    volumes:
    - /srv/myapp/attachment:/app/public/attachment
    - /srv/myapp/system:/app/public/system
    - /srv/myapp/log:/app/log
    ports:
    - 12000:8080
    command: unicorn_rails -E production -c /app/config/unicron.rb

and can be deployed and redeployed with docker compose up -d (-d = detached)

Note: docker compose always creates a default network, which is considered as a user defined network, and attaches all containers in the compose project to this network. This happens implicitly. Though, it can be configured explicitly as well.

[rwanwork (Ray)]

Hi! Thank you so much for your help! Your instructions were very clear! Unfortunately, I’m not yet successful, but thanks to you, I feel like I’m much closer.

@meyay

@rwanwork

      proxy_pass http://myserver.org:12000;

I would have expected to see the ip here. Instead, it seems to use a domain name. So either the nginx server was not restarted after changing the dns entry and host ip (nginx is known to cache resolved dns entries until is stopped), or the host uses a different dns server, or in case of the default bridge network has a local entry in /etc/hosts.

It is indeed the domain name. And I was not aware that Nginx is caching its DNS entries.

I guess the problem was with Nginx not telling the app the correct IP address. It wasn’t a problem with the app “not knowing” the new IP address (how I described my problem at first). I see now!

@meyay

This allows to use proxy_pass http://myapp:8080; instead. It uses the container name, and application port from inside the container. Furthermore, it allows removing the published port on the container, as it’s already accessible through nginx.

You can mitigate the dns caching issue by applying the settings discussed in the posed to your nginx.conf: NGINX swarm redeploy timeouts - #5 by meyay

That should do the trick.

Thank you for the clear example and for the link to the DNS caching issue. I have added that in.

@meyay

You should consider migrating your docker run commands into docker compose files. As a positive side effect you can version the compose files in git and keep track of changes.

Update: I forget to address the restart problem. A container restart is nothing else than docker stop && docker start, thus it shouldn’t behave any differently.

I have also used your example docker compose file and am using it now. The network “docker_default” gets created. Thank you!

As for the Nginx container, it wasn’t starting correctly. In the end, I gave up and realised that it is just a “standard” Nginx image, albeit a few years too old. Rather than figuring it out, I can just pull down the latest version of Nginx and use that.

I did this and Nginx worked – I don’t know what’s wrong with the earlier Nginx image. It obviously used to work, but getting a new one means I don’t need to try to figure out what’s wrong with the old one.

The app is another problem; I’m unable to build the image again. Yes – a problem for another day.

In any case, like I said, it still doesn’t work. Both images come up and stay up. Log files indicate both are up.

When I access the site, I get the following Nginx error log:

2023/08/01 19:52:04 [error] 13#13: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: [my address], server: myapp.org, request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.27.0.2:8080/favicon.ico", host: "myapp.org", referrer: "https://myapp.org/"

The number 12000 is in the docker-compose.yml file, but not in the nginx.conf file. As you suggested, I have:

proxy_pass http://myapp:8080/;

Also, if I do a docker inspect on myapp, it does have 172.27.0.2 as its network.

Does this mean that Nginx is forwarding correctly, but it is the app that is not giving the correct response? The (custom-made) app’s logs don’t indicate any errors – it should be running.

I guess what I’m saying is that it feels like it is now an internal problem of the myapp image (despite it not giving any errors). Is there anything obvious that I’ve missed?

I’m going to think about it some more tonight…see if I have missed anything you said. Thank you for your patience with this!

Ray

Edit: Just before I signed off, I did a docker ps and noticed this:

Ports                                        Names
0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp     nginx
0.0.0.0:12000->8080/tcp                      myapp

I think I’ve done something wrong. The second line used to say 12000/tcp, I think.

[meyay (Metin Y.)]

@rwanwork

I guess the problem was with Nginx not telling the app the correct IP address. It wasn’t a problem with the app “not knowing” the new IP address (how I described my problem at first). I see now!

Spot on!

@rwanwork

… it is just a “standard” Nginx image, albeit a few years too old.

Usually patch management with containers is done by replacing existing containers with new containers based on their current image. Most official images are updated on a regular basis, to include either the version updates or patches. As long as configurations are configured using environment variables or are persisted in volumes, this is the way to go. It doesn’t make sense to patch existing containers → they are designed to be disposable.

@rwanwork

Does this mean that Nginx is forwarding correctly, but it is the app that is not giving the correct response? The (custom-made) app’s logs don’t indicate any errors – it should be running.

Looks like the response is starving. If you kept the published port 12000:8080, you could check on the docker host, if accessing the url using curl -H "host: myapp.org" http://localhost:12000/favicon.ico works. If it works, it should also work with curl -H "host: myapp.org" https://localhost/favicon.ico

@rwanwork

I think I’ve done something wrong. The second line used to say 12000/tcp, I think.

The line 0.0.0.0:12000->8080/tcp indicates that host port 12000 is bound to every ip on the host and forwards the traffic to the container port 8080. Since the Dockerfile indicates port 8080 as application port, this should be correct. You are free to modify the left-hand side of the mapping as you want/need, though the right-hand side must be the port your application is listening on inside the container.

[rwanwork (Ray)]

@meyay

@rwanwork

… it is just a “standard” Nginx image, albeit a few years too old.

Usually patch management with containers is done by replacing existing containers with new containers based on their current image. Most official images are updated on a regular basis, to include either the version updates or patches. As long as configurations are configured using environment variables or are persisted in volumes, this is the way to go. It doesn’t make sense to patch existing containers → they are designed to be disposable.

Ah! Ok! That’s good to know!

Today, I did manage to find out why Nginx kept stopping and starting. It was because the configuration file (i.e., the one that was there previously) was wrong – it was referring to a Docker container that was no longer there. In other words, the Nginx container had been running a long time (a year or more) and after the container had started, it seems like someone (i.e., not me and long gone from the organisation) had changed the Nginx configuration file and/or removed a container. Thus the configuration file currently there didn’t match the current state of the Nginx instance that had been running.

There were no log files, so I didn’t know. When I went up to the latest version of Nginx, log files were being made and the issue with the configuration file was obvious…

@meyay

@rwanwork

Does this mean that Nginx is forwarding correctly, but it is the app that is not giving the correct response? The (custom-made) app’s logs don’t indicate any errors – it should be running.

Looks like the response is starving. If you kept the published port 12000:8080, you could check on the docker host, if accessing the url using curl -H "host: myapp.org" http://localhost:12000/favicon.ico works. If it works, it should also work with curl -H "host: myapp.org" https://localhost/favicon.ico

So…this didn’t work. I ran the command with -k also so that it didn’t compare the certificate with “localhost”. I got this:

http://localhost – 301 Moved Permanently
https://localhost – 504 Gateway Timeout
http://localhost:12000 – Recv failure: Connection reset by peer
https://localhost:12000 – OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection

Purely out of desperation, I changed proxy pass from 8080 to 808 (without changing the application). Indeed, the “Connection timed out” changed to “Connection refused”. I’m not sure if this is a very scientific test, but it seems like it is reaching the application.

Again, out of desperation, I even removed Nginx completely and had the application listen to port 80 (i.e., 80 → 8080), and that didn’t work.

I’m really tempted to say that this is a problem with the application. But I still feel like I’m missing something. I might not have the working source code for it (i.e., I can’t rebuild the image), but the image is there and it can be stopped and started.

These are two server groups in the nginx.conf (I’m not using a default.conf) file. All the other containers have now been removed from the configuration (but the containers are still running).

Sorry that it is taking me so long, but there isn’t an obvious mistake that I’m making, is there? Not sure if you can spot an obvious mistake I’m making…

  server {
    listen 80;
    server_name myapp.org;
    error_log  /var/log/nginx/database.log;

    rewrite ^(.*) https://$server_name$request_uri permanent;
  }

  server {
    listen 443 ssl;
    server_name myapp.org;

    # set DNS resolver as Docker internal DNS                                   
    resolver 127.0.0.11 valid=10s;
    resolver_timeout 5s;

    ssl_certificate      /cert/fullchain.pem;
    ssl_certificate_key  /cert/privkey.pem;
    error_log  /var/log/nginx/database-ssl.log;

    location / {
      proxy_redirect off;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      ##  https://stackoverflow.com/questions/18740635/nginx-upstream-timed-out\
-110-connection-timed-out-while-reading-response-hea                            
      proxy_http_version 1.1;
      proxy_set_header Connection "";

      proxy_pass http://myapp-container-name:8080/;
    }
  }

I was getting 502 errors this morning. Now I’m getting 504…I guess 504 is worse? A brief Google gave me this definition:

A 502 bad gateway means that the server acting as the gateway received an invalid response from the main server. A 504 gateway timeout means that the server acting as the gateway didn’t receive a response at all from the main server.

[meyay (Metin Y.)]

@rwanwork

http://localhost:12000 – Recv failure: Connection reset by peer

This should not happen, as the port is directly forwarded to the container port. Something is obviously off with your application. What is when you docker exec into the container and run curl -L http://localhost:8080 inside the container?

If this is not working, then you need to sort out that problem before. nginx can not reverse proxy to a not properly working target.

@rwanwork

http://localhost – 301 Moved Permanently

If you have a redirect, then you need to use curl -L -H "host: myapp.org" http://localhost instead.

[fatdollar]

@meyay

What is when you docker exec into the container and run curl -L http://localhost:8080 inside the container?

Can you explain this a little more? Which container is it running in? nginx?

[meyay (Metin Y.)]

My bad. I thought from the context of the url http://localhost:12000, it would be clear it’s the application.

[laodantong]

in the app container, put the correct address under /etc/hosts, so the app does not use DNS to translate it.

[rwanwork (Ray)]

@meyay

@rwanwork

http://localhost:12000 – Recv failure: Connection reset by peer

This should not happen, as the port is directly forwarded to the container port. Something is obviously off with your application. What is when you docker exec into the container and run curl -L http://localhost:8080 inside the container?

If this is not working, then you need to sort out that problem before. nginx can not reverse proxy to a not properly working target.

Sorry for being silent the last couple of days!

In fact, what you said above helped me greatly! It seems there were at least kthree problems that I was facing:

1. Nginx caching the old IP address
2. Nginx configuration file (left over by previous colleague) wasn’t correct; so restarting it caused it to enter into a restart loop
3. Application had some unknown problem

Your advice helped me realise the first two problems. After you suggested docker exec, I verified the application had a serious problem. It was running without any error messages, but something was still off and it wasn’t giving an error message to tell me what it was.

I have solved my problem by moving contents of the Docker container to a VM by matching the directory structure of the app. It did have issues at first, but in this state, it gave better error messages. After fiddling with it for 2 days, I got it back up and working.

Thank you for all your advice! It was really helpful and I am sure, with very little Docker experience but being forced to look after a system made by someone else, I couldn’t have done it without your help!

Thank you!!

Ray

[rwanwork (Ray)]

Ah! I connected to it using docker exec and see what you mean. I will keep that in mind in the future. Thank you!

[meyay (Metin Y.)]

Welcome!

@laodantong

in the app container, put the correct address under /etc/hosts, so the app does not use DNS to translate it.

I am not sure what sense this post makes. Containers in user defined network do not use the hosts /etc/hosts. They very much depend on dns name resolution from the buildin dns, which uses the dns servers from the host’s /etc/resolv.conf as upstream.

[laodantong]

[meyay] meyay https://forums.docker.com/u/meyay Metin Y. Leader
August 5

Welcome!

Sine dns-based service dis

laodantong:

in the app container, put the correct address under /etc/hosts, so the app
does not use DNS to translate it.

I am not sure what sense this post makes. Containers in user defined network do
not use the hosts /etc/hosts. They very much depend on dns name resolution from
the buildin dns, which uses the dns servers from /etc/resolv.conf as upstream.

but in said case the DNS was returning a wrong address, outside the scope of the
internal addresses the container was allowed to use. Other solution is to use
DNS views in the DNS server, returning different answers as a function of the
DNS client address.

[meyay (Metin Y.)]

Please re-read the whole topic. I feel your answer ignores the context of the problem, its exact cause and the solution that already applies for it.

Generally there is no need to tinker with /etc/resolv.conf in the docker context. The only useful scenario that comes to mind is on a developer machine where you want to inject domain names into your host files, so you can use a reverse proxy or application that only reacts to the domain name.

Even if you want to inject extra domain names into a container, the correct solution would be to either use the -add-host argument with docker run or extra_hosts in a compose file.