Docker compose does not start container for second environment on host Windows Server 2022

Hi, I am having problem with starting container from second compose.yml and moreover only on docker host with OS Windows Server 2022. I will describe the situation.

I need two same web applications on diferent URL adresses with own databases. So I have created two compose.yml files (compose-env01.yml and compose-env02.yml) for running two environments (env01 and env02). Each environment include two containers (web and mssql DB).

After build compose-env01.yml and compose-env02.yml are the images for web same for env01 and env02. Also images for mssql are the same for env01 and env02:
See Figure 1 below
Differents betweens compose-env01.yml and compose-env02.yml are only in using ports, as you can see:

compose-env01.yml:

version: "3.8"
services:
  web:
    build: ./web-mw
    isolation: 'default'
    hostname: web
    deploy:
      resources:
        limits:
          memory: 8G
    ports:
      - "7001:443"
    environment:
      "#CONFSETTINGS_PORT_MW#": "7001"
      "#CONFSETTINGS_MSSQLSERVERPORT#": "14331"
    depends_on:
      - mssql
  mssql: 
    build: ./database
    isolation: 'default'
    hostname: mssql
    ports:
      - "14011:14331"
    expose:
      - "14331"
    volumes:
      - .\volumes\db:C:\sqlbak
    storage_opt:
      size: '25G'
    environment:
      SA_PASSWORD: "Your_password"
      ACCEPT_EULA: "Y"
      MSSQL_PID: "Developer"
      ChangeMSSQLServerPort: "TRUE"
      MSSQLServerPort: "14331"

compose-env02.yml:

version: "3.8"
services:
  web:
    build: ./web-mw
    isolation: 'default'
    hostname: web
    deploy:
      resources:
        limits:
          memory: 8G
    ports:
      - "7002:443"
    environment:
      "#CONFSETTINGS_PORT_MW#": "7002"
      "#CONFSETTINGS_MSSQLSERVERPORT#": "14332"
    depends_on:
      - mssql
  mssql: 
    build: ./database
    isolation: 'default'
    hostname: mssql
    ports:
      - "14012:14332"
    expose:
      - "14332"
    volumes:
      - .\volumes\db:C:\sqlbak
    storage_opt:
      size: '25G'
    environment:
      SA_PASSWORD: "Your_password"
      ACCEPT_EULA: "Y"
      MSSQL_PID: "Developer"
      ChangeMSSQLServerPort: "TRUE"
      MSSQLServerPort: "14332"

When I run containers from env01 using command “docker compose -p env01 -f compose-env01.yml up -d”, both containers start correctly. Then I want to run containers from env02 using “docker compose -p env02 -f compose-env02.yml up -d”. But now will start only container env02-mssql-1 and container env02-web-1 remains in the state Starting:
See Figure 2 below
And this is the problem! This only happens on docker host with OS Windows Server 2022. The same configuration on docker host with OS Windows Server 2019 is OK and all containers from env01 and env02 start correctly.

I was trying run docker compose up also with parameter “–verbose” and with “debug”: true in daemon.json, but it didn’t give me any further information. When starting containers from compose-env02.yml, the container env02-mssql-1 started correctly and the container env02-web-1 didn’t even start to run and no further informations was displayed. So after that I canceled the start of the containers from compose-env02.yml. The state of containers from both environments was now this:
See Figure 3 below

And now a very interesting information:
Now when I shut down the containers from env01, the container env02-web-1 started Immediately automatically after that, as you can see:
See Figure 4 below
Given that, I think it will be probably some network issue. Interestingly, however, the same configuration works on docker host Windows Server 2019.

Figures (I am the new user here, so I can not upload more then one Figure):

Using versions are:
Docker version 24.0.4, build 3713ee1
Docker Compose version v2.20.2

So could you please help me anybody with this problem?

This indeed doesn’t add up. Must be a bug either in the docker compose cli plugin, or the docker engine. Is it safe to assume that you are running Windows Containers?

Since you already specify the project name, both deployments are not able to interfere with each other.

It almost looks like the engine thinks the node does not have enough resources, which doesn’t make sense either, since you didn’t specify deploy.resources.reservations.memory.

Have you checked docker container inspect? What if there is a bug that sets deploy.resources.limits.memory as deploy.resources.reservations.memory, in absence of deploy.resources.reservations.memory.

Just out of curiosity: is there a reason you set a different port for mssql using an environment, then changing the container port to that port and additionally expose it? None of this is necessary.

Hi, thank you for fast reply @meyay. Yes your assume is right. We are running Windows Containers. I also think it will be some kind of bug in docker. But I don’t understand that on docker host with Windows Server 2019 with the same docker cli and docker engine everything works fine.

Yes I checked docker container inspect. In addition, now I tried to configure memory limits on 4GB for all of the containers and I tried run both environments again (RAM on the host is 24 GB). The result is the same as in my previous post. In addition, I have another similar configuration with two compose.yml files but with three containers in each compose.yml file and also in this case the situation is the same. All three containers from the first environment are started, and from the second environment, only the container with mssql DB is started again, and when the second container is starting, it remains in the Starting state.

And to your question about the port for mssql: we have been using web applications with MSSQL DB installed in docker for several years, and during that time I noticed problems when the port for MSSQL was left as the default (1433) for both environments. It sometimes happened that the container with the web application from the env02 environment connected to the MSSQL DB in the container from the env01 environment. I don’t understand how this is possible, but it really happened. Maybe it’s a bug too. So that’s the reason why I set a different port for MSSQL for each of the containers for each environments. Now this doesn’t happen anymore.

Thank you for shedding some light on why you change the port. Maybe it was a bug with Windows Containers.

I must admit, I am clueless when it comes to Windows Server and Windows Containers. Since Docker on Windows Server uses Moby instead of Docker, they (=Moby) are the ones to provide support. Moby is the upstream project for all Docker versions, but there is no Docker Windows version other than Docker Desktop.

I suggest you raise an issue in the Moby Github project.

Update: I found the Moby forum

I installed the docker engine, docker cli and docker compose on the docker host Windows Server 2022 using chocolatey - see e.g. Chocolatey Software | Docker Engine 24.0.4. So I have not installed Docker Desktop on Windows Server. Even so, is Docker on Windows Servers really uses Moby instead of Docker?

Please see for yourself which operating systems and docker flavors Docker itself releases and supports: Install Docker Engine | Docker Documentation

Yes, I know this official docker web site, but there is no information about installing docker on Windows Server OS. So I will try to post this issue on the Moby forum. Thank you for your time!

Personally, I would raise an issue in the Moby project, as it looks more like a bug than an issue that can be sorted out in a forum discussion.

Yes you are right. I posted this issue on forums.mobyproject.org. So I’ll see.

1 Like

Given that, I think it will be probably some network issue

You could test that theory - remove the port mapping from the env02 web service and see if it starts then. Since you have specified different port mappings I would suspect it is not a network issue but that test would help narrow your focus.

I tried remove the port mapping on the env02 web service (“7002:443”) and you are right! Then container env02-web-1 has started. But what should I do now? I need to run a web application through that port. I checked that the port 7002 is on the docker host free using netstat -ano | findstr "7002". Couldn’t this really be a bug when it works on docker host with Windows Server 2019?

Update:
I tried use for port mapping any other free port on the host, but without success. If the port mapping is configure to any free port on the host, the env02-web-1 container will not start.

That is perplexing as it does seem from your netstat test that the port should be available for use. I have two other troubleshooting ideas, again just to try and isolate what might be going on:

(Idea 1) Try to deploy env01 and env02 compose file in reverse order. What happens then?


(Idea 2) Change your env02 compose file to use an ephemeral port (i.e. let Docker choose the host port for you). In other words rather than specifying - "7002:443" just specify - "443" in the ports section. What happens then? Does the env02 web container start and if so what host port is assigned to it? Perhaps if this works you could then update your config to use the ephemeral port that docker chose.

To idea 1:
I tried to deploy compose files in reverse order before a few days, so first env02 and than env01. The situation was the same. Both containers from env02 started correctly and from env01 started only env01-mssql-1 and container env01-web-1 remains in the state Starting.

To idea 2:
I don’t know if this is a good solution. I need to know what port is being used because starting the containers is an automated process in Azure DevOps pipelines, and after the containers are started, GUI tests are run that launch the web application on the URL with the chosen port. But I can try it tomorrow. Thanks for the idea.

Regarding idea 2 I’m not saying it is a good idea for production use, but just an idea to see if docker on its own is able to choose an available port and get the env02 container started with the env01 already running, but I agree it has limited value - mostly just a troubleshooting data point.

So, I tried to use an ephemeral port (Idea 2) for env02, but without success. I also tried to use an ephemeral port in the compose files for both environments env01 and env02, and also without success. Always both containers from env01 start correctly and from env02 start only container env02-mssql-1 and container env02-web-1 remains in the state Starting.

I had an idea if there is a bug in mapping the same internal ports from containers in multiple environments. So I tried use "7001:443" in compose file for env01 and "7002:444" in compose file for env02. But the result was the same again. So now I’m confused. A container with web application (env02-web-1) from env02 will only start if no port mapping is specified for that container in compose file for env02. In Dockerfile for web application I am using FROM mcr.microsoft.com/dotnet/framework/aspnet:4.8 for pull image.

Given the facts found, I think there is indeed a bug when is using docker on the host with OS Windows Server 2022. But I can’t determine exactly where is the bug.

Looks like you might not be alone:

It seems that people with the problem you’ve described can start the same containers without an issue using the docker run command, but when they use the docker compose command they encounter the problem too.

Interesting comment in that GitHub issue I linked to:

it’s possible to start two times a web-container in the same project with docker compose

So maybe that is a workaround you could use - consolidate your compose file into a single file that starts both environments? Worth a try!

Thank you for this informations. Now I know, that I am not alone with this problem. I will try to configure in one compose file.

Update:
So the workaround work for me. When I configured all of the containers in one compose file, all containers started and everythings worked.

I just have to decide now if I will use a docker host with OS Windows Server 2022 in combination with this workaround. Or I will be using a docker host with OS Windows Server 2019 and standard using multiple environments from multiple compose files for now. Because to use this workaround, it means for me to modify a lot of things in our processes. I can only hope that someone fixes this bug soon.

Anyway, thank you so much @rleapm for your help!

1 Like

Just to add my voice of support here.

I have a similar issue on Windows Server 2022. A second instance of my application basically hangs until the service is bounced.

My application uses a single yaml file that takes args to specify ports if desired, but otherwise lets compose (or perhaps it’s docker) pick them.
Nothing I have tried has helped.

This is using Windows Containers on 2022.
Using Hyper-V on 2019 it was fine.

It’s so broken that for now I have given up on using docker compose on Windows, and am looking at using Kubernetes to deploy the app. Kompose is proving useful here, although it doesn’t look like can support arguments in the same way compose does. Very frustrating.

I really hope this issue can get some focus, since many of us are forced to host on Windows if running .NET framework apps, and running on Server 2022 and containers is much more efficient than Server 2019 and Hyper-V (since cannot share much of the executive, so memory reqs go through the roof).

Cheers,
Westy