Understanding How Host File Blocking Interferes with Docker Communication (127.0.0.1 Issue)

Docker Desktop
, , ,
1

Hi there,

I’m working with Docker to build a Flutter app through act-cli, but I’m encountering an issue where Gradle cannot connect to 127.0.0.1 during the build process. Initially, this seemed like a network issue with Docker’s default network configuration.

But then I checked the Gradle log and figured out some endpoint link to the port 127.0.0.1

2024-12-14T05:04:39.937+0000 [DEBUG] [org.gradle.launcher.daemon.bootstrap.DaemonMain] Assuming the daemon was started with following jvm opts: [--add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.lang.invoke=ALL-UNNAMED, --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.nio.charset=ALL-UNNAMED, --add-opens=java.base/java.net=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED, -Xmx1000m, -Dfile.encoding=UTF-8, -Duser.country=US, -Duser.language=en, -Duser.variant]
2024-12-14T05:04:40.004+0000 [INFO] [org.gradle.launcher.daemon.server.Daemon] start() called on daemon - DefaultDaemonContext[uid=476a22cb-6485-4d7c-a6b1-b142c4c80e15,javaHome=/usr/lib/jvm/java-17-openjdk-amd64,javaVersion=17,javaVendor=Ubuntu,daemonRegistryDir=/root/.gradle/daemon,pid=533,idleTimeout=120000,priority=NORMAL,applyInstrumentationAgent=true,nativeServicesMode=ENABLED,daemonOpts=--add-opens=java.base/java.util=ALL-UNNAMED,--add-opens=java.base/java.lang=ALL-UNNAMED,--add-opens=java.base/java.lang.invoke=ALL-UNNAMED,--add-opens=java.prefs/java.util.prefs=ALL-UNNAMED,--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED,--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED,--add-opens=java.base/java.nio.charset=ALL-UNNAMED,--add-opens=java.base/java.net=ALL-UNNAMED,--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED,-Xmx1000m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]
2024-12-14T05:04:40.018+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding IP addresses for network interface docker0
2024-12-14T05:04:40.018+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Is this a loopback interface? false
2024-12-14T05:04:40.018+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding remote address /172.17.0.1
2024-12-14T05:04:40.019+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding IP addresses for network interface br-1ac2cbf1abc1
2024-12-14T05:04:40.019+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Is this a loopback interface? false
2024-12-14T05:04:40.019+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding remote address /172.18.0.1
2024-12-14T05:04:40.019+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding IP addresses for network interface eth0
2024-12-14T05:04:40.019+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Is this a loopback interface? false
2024-12-14T05:04:40.020+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding remote address /fdc4:f303:9324:0:0:0:0:3%eth0
2024-12-14T05:04:40.020+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding remote address /fe80:0:0:0:c91:dfff:fef2:8afa%eth0
2024-12-14T05:04:40.020+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding remote address /192.168.65.3
2024-12-14T05:04:40.020+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding IP addresses for network interface services1
2024-12-14T05:04:40.020+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Is this a loopback interface? false
2024-12-14T05:04:40.020+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding remote address /fe80:0:0:0:309a:5ff:fe7f:5efe%services1
2024-12-14T05:04:40.020+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding remote address /fdc4:f303:9324:0:0:0:0:6%services1
2024-12-14T05:04:40.020+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding remote address /192.168.65.6
2024-12-14T05:04:40.021+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding IP addresses for network interface lo
2024-12-14T05:04:40.021+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Is this a loopback interface? true
2024-12-14T05:04:40.021+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding loopback address /0:0:0:0:0:0:0:1%lo
2024-12-14T05:04:40.021+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding loopback address /127.0.0.1
2024-12-14T05:04:40.022+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.TcpIncomingConnector] Listening on [f0396ef3-6600-4fa1-9305-cebee6fdd858 port:35595, addresses:[localhost/127.0.0.1]].
2024-12-14T05:04:40.028+0000 [DEBUG] [org.gradle.launcher.daemon.server.Daemon] Daemon starting at: Sat Dec 14 05:04:40 GMT 2024, with address: [f0396ef3-6600-4fa1-9305-cebee6fdd858 port:35595, addresses:[localhost/127.0.0.1]]
2024-12-14T05:04:40.029+0000 [INFO] [org.gradle.launcher.daemon.server.DaemonRegistryUpdater] Advertising the daemon address to the clients: [f0396ef3-6600-4fa1-9305-cebee6fdd858 port:35595, addresses:[localhost/127.0.0.1]]
2024-12-14T05:04:40.029+0000 [DEBUG] [org.gradle.launcher.daemon.server.DaemonRegistryUpdater] Advertised daemon context: DefaultDaemonContext[uid=476a22cb-6485-4d7c-a6b1-b142c4c80e15,javaHome=/usr/lib/jvm/java-17-openjdk-amd64,javaVersion=17,javaVendor=Ubuntu,daemonRegistryDir=/root/.gradle/daemon,pid=533,idleTimeout=120000,priority=NORMAL,applyInstrumentationAgent=true,nativeServicesMode=ENABLED,daemonOpts=--add-opens=java.base/java.util=ALL-UNNAMED,--add-opens=java.base/java.lang=ALL-UNNAMED,--add-opens=java.base/java.lang.invoke=ALL-UNNAMED,--add-opens=java.prefs/java.util.prefs=ALL-UNNAMED,--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED,--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED,--add-opens=java.base/java.nio.charset=ALL-UNNAMED,--add-opens=java.base/java.net=ALL-UNNAMED,--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED,-Xmx1000m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]
2024-12-14T05:04:40.030+0000 [DEBUG] [org.gradle.launcher.daemon.registry.PersistentDaemonRegistry] Storing daemon address: [f0396ef3-6600-4fa1-9305-cebee6fdd858 port:35595, addresses:[localhost/127.0.0.1]], context: DefaultDaemonContext[uid=476a22cb-6485-4d7c-a6b1-b142c4c80e15,javaHome=/usr/lib/jvm/java-17-openjdk-amd64,javaVersion=17,javaVendor=Ubuntu,daemonRegistryDir=/root/.gradle/daemon,pid=533,idleTimeout=120000,priority=NORMAL,applyInstrumentationAgent=true,nativeServicesMode=ENABLED,daemonOpts=--add-opens=java.base/java.util=ALL-UNNAMED,--add-opens=java.base/java.lang=ALL-UNNAMED,--add-opens=java.base/java.lang.invoke=ALL-UNNAMED,--add-opens=java.prefs/java.util.prefs=ALL-UNNAMED,--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED,--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED,--add-opens=java.base/java.nio.charset=ALL-UNNAMED,--add-opens=java.base/java.net=ALL-UNNAMED,--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED,-Xmx1000m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]
...
(more)

The critical error is that the Gradle daemon couldn’t been connected

What went wrong:
Could not connect to the Gradle daemon.
Daemon uid: 908223a2-5fc1-4129-9952-a4c2edad0ed7 with diagnostics:
Daemon pid: 14720

After a very long trial and errors, with various combination on config I solved the problem by turn-off the network host on Docker Desktop ( I am using Window 10 and Docker Desktop).

Then my intuition suggest me to check the host file (in C:\Windows\System32\drivers\etc)

Looked like I intentionally put two record to block distracting site using 127.0.0.1

127.0.0.1 9gag.com
127.0.0.1 facebook.com

By removing the two lines, the problem had been solved. But I want to study & understand why it happened from the first place.

In summary, here’s the sequence of what happened:

  1. I couldn’t run flutter build because Gradle couldn’t connect to 127.0.0.1.
  2. The problem was resolved when I disabled Docker’s host network mode (--network host).
  3. I then found that the issue was fixed (even on network host) when I removed two blocked entries in my hosts file that were redirecting certain domains to 127.0.0.1.

I want to deeply understand how these blocked lines in the hosts file (redirecting domain names to 127.0.0.1) are interfering with Docker’s networking and communication. Specifically: How does routing 127.0.0.1 for certain domains affect Docker containers and their ability to access services running on the host (like Gradle)?

Any insights on the relationship between Docker’s network stack and the host file would be really helpful!

Edit 12/15/2024
It’s more likely that I was wrong. I tested again with the two line removed and then readded later then it turns out the error STILL HAPPENED. So that mean the only way to fix it is to TURN-OFF the NETWORK HOST. So sorry for the confusion I caused!

2

The host file can be used to add/override name resolution. It neither blocks anything, nor forwards anything. It does not make any sense that it’s part of the problem, or that removing those lines is part of the solution.

3

I also agree with you, albeit I still don’t understand why it cause the issue from the first place. That why I ask here.

Btw, here is the .workflow file. The actual code to test could be the Flutter sample project. I just made a sample and run a quick test. The error happened again so I am confident that this bug can be reproduced reliably

name: Bug

on:
  workflow_dispatch:


jobs:
  build:
    name: Build Flutter Project
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/cirruslabs/flutter:3.24.5
    steps:
        - name: Checkout flutter-hello-world
          uses: actions/checkout@v4

        - name: Install Node.js
          run: |
            curl -sL https://deb.nodesource.com/setup_20.x | bash -
            apt-get install -y nodejs
        


        - name: Set up JDK 17
          uses: actions/setup-java@v3
          with:
              java-version: '17'
              distribution: 'temurin'

        - name: Setup Android SDK
          uses: android-actions/setup-android@v3

        - name: Build SampleApplication
          run: flutter build apk --release

I just re-added the line 127.0.0.1 9gag.com and the error occurs again

FAILURE: Build failed with an exception.

* What went wrong:
Could not connect to the Gradle daemon.
Daemon uid: 9e27ccdb-95a8-4e89-b82a-3d2a9ce6ff02 with diagnostics:
Daemon pid: 984
  log file: /root/.gradle/daemon/8.3/daemon-984.out.log
----- Last  20 lines from daemon log file - daemon-984.out.log -----
2024-12-14T12:51:14.335+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding remote address /fdc4:f303:9324:0:0:0:0:6%services1
2024-12-14T12:51:14.336+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding remote address /fe80:0:0:0:a4d6:eff:feb8:6f75%services1
2024-12-14T12:51:14.336+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding remote address /192.168.65.6
2024-12-14T12:51:14.336+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding IP addresses for network interface lo
2024-12-14T12:51:14.336+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Is this a loopback interface? true
2024-12-14T12:51:14.336+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding loopback address /0:0:0:0:0:0:0:1%lo
2024-12-14T12:51:14.336+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.InetAddresses] Adding loopback address /127.0.0.1
2024-12-14T12:51:14.338+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.TcpIncomingConnector] Listening on [8cb624c8-e7c2-4a8d-97f8-e40ce91adf14 port:43741, addresses:[localhost/127.0.0.1]].
2024-12-14T12:51:14.500+0000 [DEBUG] [org.gradle.launcher.daemon.server.Daemon] Daemon starting at: Sat Dec 14 12:51:14 GMT 2024, with address: [8cb624c8-e7c2-4a8d-97f8-e40ce91adf14 port:43741, addresses:[localhost/127.0.0.1]]
2024-12-14T12:51:14.500+0000 [INFO] [org.gradle.launcher.daemon.server.DaemonRegistryUpdater] Advertising the daemon address to the clients: [8cb624c8-e7c2-4a8d-97f8-e40ce91adf14 port:43741, addresses:[localhost/127.0.0.1]]
2024-12-14T12:51:14.500+0000 [DEBUG] [org.gradle.launcher.daemon.server.DaemonRegistryUpdater] Advertised daemon context: DefaultDaemonContext[uid=9e27ccdb-95a8-4e89-b82a-3d2a9ce6ff02,javaHome=/opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/17.0.13-11/x64,daemonRegistryDir=/root/.gradle/daemon,pid=984,idleTimeout=10800000,priority=NORMAL,applyInstrumentationAgent=true,daemonOpts=-XX:MaxMetaspaceSize=2G,-XX:+HeapDumpOnOutOfMemoryError,--add-opens=java.base/java.util=ALL-UNNAMED,--add-opens=java.base/java.lang=ALL-UNNAMED,--add-opens=java.base/java.lang.invoke=ALL-UNNAMED,--add-opens=java.prefs/java.util.prefs=ALL-UNNAMED,--add-opens=java.base/java.nio.charset=ALL-UNNAMED,--add-opens=java.base/java.net=ALL-UNNAMED,--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED,-Xmx4G,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]
2024-12-14T12:51:14.501+0000 [DEBUG] [org.gradle.launcher.daemon.registry.PersistentDaemonRegistry] Storing daemon address: [8cb624c8-e7c2-4a8d-97f8-e40ce91adf14 port:43741, addresses:[localhost/127.0.0.1]], context: DefaultDaemonContext[uid=9e27ccdb-95a8-4e89-b82a-3d2a9ce6ff02,javaHome=/opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/17.0.13-11/x64,daemonRegistryDir=/root/.gradle/daemon,pid=984,idleTimeout=10800000,priority=NORMAL,applyInstrumentationAgent=true,daemonOpts=-XX:MaxMetaspaceSize=2G,-XX:+HeapDumpOnOutOfMemoryError,--add-opens=java.base/java.util=ALL-UNNAMED,--add-opens=java.base/java.lang=ALL-UNNAMED,--add-opens=java.base/java.lang.invoke=ALL-UNNAMED,--add-opens=java.prefs/java.util.prefs=ALL-UNNAMED,--add-opens=java.base/java.nio.charset=ALL-UNNAMED,--add-opens=java.base/java.net=ALL-UNNAMED,--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED,-Xmx4G,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]
2024-12-14T12:51:14.507+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire exclusive lock on daemon addresses registry.
2024-12-14T12:51:14.510+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2024-12-14T12:51:14.513+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2024-12-14T12:51:14.513+0000 [LIFECYCLE] [org.gradle.launcher.daemon.server.Daemon] Daemon server started.
2024-12-14T12:51:14.513+0000 [DEBUG] [org.gradle.launcher.daemon.bootstrap.DaemonStartupCommunication] Completed writing the daemon greeting. Closing streams...
2024-12-14T12:51:14.516+0000 [DEBUG] [org.gradle.launcher.daemon.server.Daemon] stopOnExpiration() called on daemon
2024-12-14T12:51:14.517+0000 [DEBUG] [org.gradle.launcher.daemon.server.Daemon] awaitExpiration() called on daemon
2024-12-14T12:51:14.517+0000 [DEBUG] [org.gradle.launcher.daemon.server.DaemonStateCoordinator] daemon is running. Sleeping until state changes.
----- End of the daemon log -----


* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.
| Running Gradle task 'assembleRelease'...                           70.7s
| Gradle task assembleRelease failed with exit code 1
4

I also added my host file below

# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

# localhost name resolution is handled within DNS itself.
#	127.0.0.1       localhost
#	::1             localhost


127.0.0.1 9gag.com
# Added by Docker Desktop
192.168.100.6 host.docker.internal
192.168.100.6 gateway.docker.internal
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section

# Setup for tailscale VPN ...

I am using VPN service (Tailscale) BUT I already tested its and I am confirm that they are not related to the bug. Turning VPN on/off doesn’t matter to this behavior.

5

In Dcoker Desktop, there is no real host network. It is just forwarding ports from the host to and from the virtual machine of Docker Desktop. And it seems it also does something with the hosts files.

It also has to do something with the WSL2 backend or just Windows, because I could not reproduce any issue with the host file on my Mac, but I could on Windows. Although I’m not sure which one is the issue. Not being able to reproduce it on macOS, or being able to do it on Windows. The hosts file seems to be used for the name resolution in the containers in Docker Desktop for Windows, but not on Docker Desktop for Mac.

If I add a host to my hosts file on Windows, and try to curl that host pointing to a loopback interface (which would originally be a real website on a public IP), the site is not available. Which could be the expected result when using the host network even though it is not a real one. When I tried the same on macOS, I could always access the website regardless of my hosts file.

Does your Gradle do anything with any of the sites you added to your hosts file? If not, I don’t udnerstand it either. Unless you have an older Docker Desktop which had an additional bug, but any other site works from my Docker Desktop v4.37

One thing I can imagine, if your Gradle daemon (I don’t use it, so I don’t know if it can happen) tries to detect the hosts names assigned to the loopback interface and gets domain names that are not localhost, so it can’t handle it. Why, I don’t know.

6

HI there,

So sorry for the confusion I caused, I re-tested with the two lines removed, then with and without VPN, with NETWORK_HOST mode set on. The bug still happened.

So yes, as @meyay argued, it’s not related to the HOST file, albeit I will investigate how/why the network host-mode causing issue for Gradle.

7

I am not sure if you already wrote about which command is executed in which context:

  • PowerShell on the Windows host?
  • Terminal in a WSL2 distribution with enabled wsl integration in Docker Desktop?
  • Inside a container?

I assume you run the command flutter build in a PowerShell Terminal on the host.

The argument --network host only make sense, if the “host network driver” is enabled in Docker Desktop, see: Host network driver | Docker Docs.

8

Sorry for the confusion I made,

It is flutter build from the .workflow file, so it should run inside the docker container. From my understanding, the act-cli create a container, then run flutter build inside that docker container.

It’s my wrong to say that I disabled Docker’s host network mode by using flag (--network host ). In reality what I did is toggle off the Network Host mode in Docker Desktop, (in Setting → Resources → Network)

9

A single command without a precise context, isn’t what will help me to get a better understanding of the situation. Please share all relevant details that allow someone not being in your headspace to actually understand what you do, where you do it and how exactly you do it.

Regarding the host network driver: Is it safe to assume that you read the link and know how a container needs to be started to actually work with the host network driver of Docker Desktop?