Network disconnect does not seem to disconnect?

General Discussions General
#1

I have been messing about with docker and networking, but I have a question
It seems to me that disconnect does not do what I would think that it does…
To illustrate I set up a system with a rudimentary http server connected solely to a private network and a proxy connected to the private network.
I try to wget from the webserver via the hosts localhost interface and it fails as expected.
Then I connect the proxy to the public network, and the wget succeeds
The I disconnect the proxy from the public network
Now I would expect the wget to fail, but in stead it succeeds, only the proxy now receives the wget from localhost via the private network.
How did the internal private network suddenly become connected to hosts localhost?
What am I doing wrong?
HC

  1. I start with an empty docker installation
# docker version
Client:
Version:          20.10.12
API version:      1.41
Go version:       go1.17.5
Git commit:       e91ed5707e
Built:            Mon Dec 13 22:31:40 2021
OS/Arch:          linux/amd64
Context:          default
Experimental:     true
Server:
Engine:
 Version:         20.10.12
 API version:     1.41 (minimum version 1.12)
 Go version:      go1.17.5
 Git commit:      459d0dfbbb
 Built:           Mon Dec 13 22:30:43 2021
 OS/Arch:         linux/amd64
 Experimental:    false
containerd:
 Version:         v1.5.9
 GitCommit:       1407cab509ff0d96baa4f0eb6ff9980270e6e620.m
runc:
 Version:         1.1.0
 GitCommit:       v1.1.0-0-g067aaf85
docker-init:
 Version:         0.19.0
 GitCommit:       de40ad0

# uname -a
Linux docker01 5.16.1-arch1-1 #1 SMP PREEMPT Sun, 16 Jan 2022 11:39:23 +0000 x86_64 GNU/Linux
  1. Create a network bridge: public - it becomes 172.22.0.0/16
docker network create --driver bridge public
docker network inspect public

[
   {
       "Name": "public",
       "Id": "718c5333dfcc617d1f6c4e1dd24272bc3ebf57926fcc9783b82ba984cf566d24",
       "Created": "2022-01-20T16:55:27.723524281+01:00",
       "Scope": "local",
       "Driver": "bridge",
       "EnableIPv6": false,
       "IPAM": {
           "Driver": "default",
           "Options": {},
           "Config": [
               {
                   "Subnet": "172.22.0.0/16",
                   "Gateway": "172.22.0.1"
               }
           ]
       },
       "Internal": false,
       "Attachable": false,
       "Ingress": false,
       "ConfigFrom": {
           "Network": ""
       },
       "ConfigOnly": false,
       "Containers": {},
       "Options": {},
       "Labels": {}
   }
]
  1. Create an “internal” network bridge: private - it becomes 172.23.0.0/16
docker network create --driver bridge --internal private
docker network inspect private

[
   {
       "Name": "private",
       "Id": "e69cf1803518c702fbf1f7544e83b4b7251367163e4f4dca1670834ec1bdd0e5",
       "Created": "2022-01-20T16:55:27.980620123+01:00",
       "Scope": "local",
       "Driver": "bridge",
       "EnableIPv6": false,
       "IPAM": {
           "Driver": "default",
           "Options": {},
           "Config": [
               {
                   "Subnet": "172.23.0.0/16",
                   "Gateway": "172.23.0.1"
               }
           ]
       },
       "Internal": true,
       "Attachable": false,
       "Ingress": false,
       "ConfigFrom": {
           "Network": ""
       },
       "ConfigOnly": false,
       "Containers": {},
       "Options": {},
       "Labels": {}
   }
]
  1. Create and start a rudimentory web server listening on port 8080 connected to the private network
    docker create --name httpd --network private --publish 8080:8080 alpine/socat -d -d TCP4-LISTEN:8080,crlf,reuseaddr,fork SYSTEM:"echo HTTP/1.0 200;echo Server\: socat hack;echo Content-Type\: text/plain;echo;echo ok" ; docker start httpd
  2. Create and start a simple proxy listening on 8081 and forwarding to our rudimetory http server
    docker create --name proxy --network private --publish 8081:8081 alpine/socat -d -d tcp4-listen:8081,fork,reuseaddr tcp4-connect:httpd:8080 ; docker start proxy
  3. In two other windows keep an eye on the logs
docker logs -f httpd
               2022/01/20 16:00:55 socat[1] N listening on AF=2 0.0.0.0:8080
docker logs -f proxy
               2022/01/20 16:01:49 socat[1] N listening on AF=2 0.0.0.0:8081
  1. In another window attempt to wget from the httpd server via the proxy server
    wget -v localhost:8081 -O -
    This fails as expected :slight_smile:
  2. Connect the proxy to the public network
    docker network connect proxy public
  3. In the window from 6) attempt to wget from the httpd server via the proxy server
    wget -v localhost:8081 -O -
    This succedes as expected :slight_smile:

Log from proxy

               2022/01/20 16:01:49 socat[1] N listening on AF=2 0.0.0.0:8081
               2022/01/20 16:02:28 socat[1] N accepting connection from AF=2 172.22.0.1:48298 on AF=2
 172.22.0.2:8081
               2022/01/20 16:02:28 socat[1] N forked off child process 6
               2022/01/20 16:02:28 socat[1] N listening on AF=2 0.0.0.0:8081
               2022/01/20 16:02:28 socat[6] N opening connection to AF=2 172.23.0.2:8080
               2022/01/20 16:02:28 socat[6] N successfully connected from local address AF=2 172.23.0.3:41038
               2022/01/20 16:02:28 socat[6] N starting data transfer loop with FDs [6,6] and [5,5]
               2022/01/20 16:02:28 socat[6] N socket 2 (fd 5) is at EOF
               2022/01/20 16:02:28 socat[6] N socket 1 (fd 6) is at EOF
               2022/01/20 16:02:28 socat[6] N exiting with status 0
               2022/01/20 16:02:28 socat[1] N childdied(): handling signal 17

Log from httpd

               2022/01/20 16:02:28 socat[1] N accepting connection from AF=2 172.23.0.3:41038 on AF=2 172.23.0.2:8080
               2022/01/20 16:02:28 socat[1] N forked off child process 6
               2022/01/20 16:02:28 socat[1] N listening on AF=2 0.0.0.0:8080
               2022/01/20 16:02:28 socat[6] N forking off child, using socket for reading and writing
               2022/01/20 16:02:28 socat[6] N forked off child process 7
               2022/01/20 16:02:28 socat[6] N forked off child process 7
               2022/01/20 16:02:28 socat[6] N starting data transfer loop with FDs [6,6] and [5,5]
               2022/01/20 16:02:28 socat[6] N socket 2 (fd 5) is at EOF
               2022/01/20 16:02:28 socat[6] N childdied(): handling signal 17
               2022/01/20 16:02:28 socat[6] E write(5, 0x7fef98f1d000, 122): Broken pipe
               2022/01/20 16:02:28 socat[6] N exit(1)
               2022/01/20 16:02:28 socat[1] N childdied(): handling signal 17

Notice that the proxy accepts from the public network interface 172.22.0.2:8081, then connects to the httpd server on the private network 172.23.0.2:8080
9) Disconnect proxy from the public network
docker network disconnect proxy public
10) In the window from 6) & 8) attempt to wget from the httpd server via the proxy server
wget -v localhost:8081 -O -

I would expect this to fail, but in stead it succedes :open_mouth:

Log from proxy

2022/01/20 16:03:34 socat[1] N accepting connection from AF=2 172.23.0.1:56558 on AF=2 172.23.0.3:8081
2022/01/20 16:03:34 socat[1] N forked off child process 7
2022/01/20 16:03:34 socat[1] N listening on AF=2 0.0.0.0:8081
2022/01/20 16:03:34 socat[7] N opening connection to AF=2 172.23.0.2:8080
2022/01/20 16:03:34 socat[7] N successfully connected from local address AF=2 172.23.0.3:41040
2022/01/20 16:03:34 socat[7] N starting data transfer loop with FDs [6,6] and [5,5]
2022/01/20 16:03:34 socat[7] N socket 2 (fd 5) is at EOF
2022/01/20 16:03:34 socat[7] N socket 1 (fd 6) is at EOF
2022/01/20 16:03:34 socat[7] N exiting with status 0
2022/01/20 16:03:34 socat[1] N childdied(): handling signal 17

Log from httpd

2022/01/20 16:03:34 socat[1] N accepting connection from AF=2 172.23.0.3:41040 on AF=2 172.23.0.2:8080
2022/01/20 16:03:34 socat[1] N forked off child process 9
2022/01/20 16:03:34 socat[1] N listening on AF=2 0.0.0.0:8080
2022/01/20 16:03:34 socat[9] N forking off child, using socket for reading and writing
2022/01/20 16:03:34 socat[9] N forked off child process 10
2022/01/20 16:03:34 socat[9] N forked off child process 10
2022/01/20 16:03:34 socat[9] N starting data transfer loop with FDs [6,6] and [5,5]
2022/01/20 16:03:34 socat[9] W read(5, 0x7fef98f1d000, 8192): Connection reset by peer
2022/01/20 16:03:34 socat[9] N socket 2 to socket 1 is in error
2022/01/20 16:03:34 socat[9] N socket 2 (fd 5) is at EOF
2022/01/20 16:03:34 socat[9] N childdied(): handling signal 17
2022/01/20 16:03:34 socat[9] N socket 1 (fd 6) is at EOF
2022/01/20 16:03:34 socat[9] N socket 2 (fd 5) is at EOF
2022/01/20 16:03:34 socat[9] N exiting with status 0
2022/01/20 16:03:34 socat[1] N childdied(): handling signal 17

This time the proxy suddenly and unexpectedly accepts from the private network interface 172.23.0.2 ??? then, as expected connects to the httpd server on the private network 172.23.0.2:8080

How did that “internal” private network suddenly become exposed and accesible on the host localhost ?
What am I doing wrong…?
There must be something I didn’t understand :slight_smile:

Any help will be much appreciated :slight_smile:

HC

#2

So I did a second run, same result, but with a lot of network inspection and iptables inspection included

https://dedibox4.hobsoft.net/owncloud/index.php/s/ALjzqeanTO0aWeU

I’m hoping someone can help me shed light on this :slight_smile:

HC

#3

I don’t have time to work on a solution right now, but I edited your first post to make your question more readable. I am not sure I used the code blocks properly, I just guessed what was a code and what were your thoughts. Please, always use the </> button to insert codes and check your post after sending it. If you have huge letters, strange indentations and very long texts where we can’t tell which is a code and which line is a comment from you there is a good chance you won’t get an answer.

It is very important to compose a clear, understandable question!

I will try to come back to your post later today.

#4

That txt is really long so I scrolled down a little and closed. You could simplify the test since it doesn’t matter if you have a connection between a proxy and a server. One container is enough. I tested with httpd:2.4

docker run --rm -d --name httpd2 --network p2 -p 8082:80 httpd:2.4

When it has only an internal network I cannot access the HTTPD server on port 8082.

wget -v localhost:8082 -O -

When I connect a non-internal network, it works.
When I disconnect the non-internal network, it stilll works.

I also checked the iptables rules:

  • test-before.txt: before connecting the non-internal network
  • test-between.txt: after connecting but before disconnecting the non-internal network
  • test-after.txt: After disconnecting the non-internal network

test-before.txt and test-after.txt was basically the same and yet curl worked on localhost:8082

Then I noticed that something was different. Even if I defined the port forward initially, docker ps showed that there was no port forward. After connecting the non-internal network, I could see the port forward and it remained there after disconnecting the non-internal network.

When I restarted the container, the portforward disappeared from the list and curl didn’t work again.

I would say that it is a bug. But it doesn’t have to do anything with the communication between your proxy and httpd (those containers are communicating on the private network) except that when you have internal network, dns resolution doesn’t work either, so socat can’t forward traffic using “httpd” as hostname but it works with an ip address.