My windows CI containers are very slow to a ridiculous point and I need to know how to fix it.
The setup
I am running a Jenkins CI/CD server locally where the agents are custom docker containers with all of my build, lint tools .etc preinstalled in the image (it’s at about 20GB currently). My linux agents and controller give me no problems and run as expected with performance that closely matches that of native execution on the host, however this has definitely not been the case with my windows agents. Building one of my C++ projects on my laptop locally from clean repo takes about a minute and a half (16 CPUs, 32GB of ram), but on the container it can take upwards of half an hour! I’ve tried allocating more or less CPUs in the docker-compose, more RAM but none of it seems to be making a difference. I’ve posted my dockerfile and docker-compose. Hopefully someone can help me to find the issue. Note that sometimes the connection between the agent and the controller is a bit shaky / low throughput, I don’t know if that could be causing issues as jenkins is holding up stuff waiting for a packet from the controller?
The Dockerfile
# Set Shell to PowerShell for all subsequent RUN commands
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop';"]
# 1. Install Chocolatey
RUN Set-ExecutionPolicy Bypass -Scope Process -Force; \
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; \
iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
# 2. Install basic build dependencies
RUN choco install -y git cmake ninja curl openjdk make python312
# 3. Install VS 2026 Build Tools
RUN mkdir C:\TEMP
COPY .vsconfig C:/TEMP/.vsconfig
SHELL ["cmd", "/S", "/C"]
RUN curl -SL --output C:\TEMP\vs_buildtools.exe https://aka.ms/vs/stable/vs_buildtools.exe ^ \
&& C:\TEMP\vs_buildtools.exe ^ \
--quiet ^ \
--wait ^ \
--norestart ^ \
--nocache ^ \
--installPath "C:\BuildTools" ^ \
--config "C:\TEMP\.vsconfig" ^ \
--remove Microsoft.VisualStudio.Component.Windows10SDK.10240 ^ \
--remove Microsoft.VisualStudio.Component.Windows10SDK.10586 ^ \
--remove Microsoft.VisualStudio.Component.Windows10SDK.14393 ^ \
--remove Microsoft.VisualStudio.Component.Windows81SDK ^ \
&& del C:\TEMP\vs_buildtools.exe
RUN reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\rc.exe" /v MitigationOptions /t REG_QWORD /d 2 /f
# Verify build tools installation
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop';"]
RUN if (-not (Test-Path 'C:\BuildTools')) { throw 'Build Tools installation failed' }
# Install rust
ENV CARGO_HOME="C:\cargo" \
RUSTUP_HOME="C:\rustup"
RUN Invoke-WebRequest -Uri "https://win.rustup.rs/x86_64" -OutFile "C:\TEMP\rustup-init.exe"
RUN C:\TEMP\rustup-init.exe -y --default-toolchain stable --profile default
RUN [Environment]::SetEnvironmentVariable('Path', 'C:\cargo\bin;' + $env:Path, [EnvironmentVariableTarget]::Machine)
RUN rustup component add rustfmt clippy
RUN del C:\TEMP\rustup-init.exe
# Install esp-idf
ADD https://github.com/espressif/idf-im-ui/releases/download/v0.12.2/eim-cli-windows-x64.exe eim-cli/bin/eim.exe
RUN [Environment]::SetEnvironmentVariable('Path', 'C:\eim-cli\bin;' + $env:Path, [EnvironmentVariableTarget]::Machine)
# 4. Setup Jenkins User
RUN net user jenkins /add /passwordreq:no; \
net localgroup Administrators jenkins /add; \
New-Item -ItemType Directory -Path C:\Users\jenkins -Force; \
icacls C:\\Users\\jenkins /grant 'jenkins:(OI)(CI)F' /T
RUN takeown /f "C:\cargo" /r /d y
RUN icacls "C:\cargo" /grant "jenkins:F" /t
RUN takeown /f "C:\rustup" /r /d y
RUN icacls "C:\rustup" /grant "jenkins:F" /t
# 5. Environment Configuration
ENV JENKINS_URL="" \
AGENT_NAME="" \
AGENT_SECRET="" \
AGENT_WORKDIR="C:\Users\jenkins"
USER jenkins
WORKDIR C:\\Users\\jenkins
COPY eim-config.toml eim-config.toml
RUN eim install --config eim-config.toml --non-interactive true
RUN eim select v5.1.7
RUN Set-Content -Path C:\shell.bat -Value '@echo off'; ` \
Add-Content -Path C:\shell.bat -Value 'call "C:\BuildTools\VC\Auxiliary\Build\vcvarsall.bat" x64 && %*';
SHELL ["C:\\shell.bat", "cmd", "/S", "/C"]
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop';"]
ENTRYPOINT ["powershell", "-NoLogo", "-Command", \
"$ErrorActionPreference = 'Stop'; \
Write-Host 'Fetching agent.jar from' $env:JENKINS_URL; \
Invoke-WebRequest -UseBasicParsing -Uri \"$($env:JENKINS_URL)/jnlpJars/agent.jar\" -OutFile 'agent.jar'; \
Write-Host 'Starting Jenkins Agent...'; \
java -jar agent.jar -url $env:JENKINS_URL -secret $env:AGENT_SECRET -name $env:AGENT_NAME -webSocket -workDir $env:AGENT_WORKDIR"]
So you basically shared how you create the image for your Jenkins agent (the shared Dockerfile is incomplete btw) and start a container using it. It’s been 10 years since I saw Jenkins being used in the wild - so I am far from being up-to-date when it comes to Jenkins.
It says nothing about the CI job you actually use to build your cpp application.
The number of Windows container users is rather small compared to the number of Linux container users, so it might take a while until you get an answer here.
@vrapolinario do I recall correct that there was a resource limitation on Windows Containers with Docker Desktop? I remember we had a topic about it long time ago, but I don’t recall the details. Do I remember it correct?
Ah sorry I missed the from line in my copy paste. I’ve added it here below:
FROM mcr.microsoft.com/windows/server:ltsc2025
Here is my Jenkinsfile for the project, I’ve just stripped out some URLs to internal tools. Yeah I know Jenkins is a bit of an unusual tool but I need some custom tools (like esp-idf) installed into my runners and I’m a bit scared of github beginning to charge for self-hosted runners. I’ve found github actions to also be MUCH slower than Jenkins (at least most of the time). The particularly offending stage is the Build ESP32 firmware one, on a linux runner it takes about a minute to a minute and a half while on windows it can take upwards of half an hour. From looking at the logs while it’s running, it’s the main compilation progress that is slow. It seems to be particularly linker / disk heavy as it’s about a thousand C object files being compiled and then linked together, so maybe that is the bottleneck?
pipeline {
agent none
stages {
stage("Build ESP32 firmware") {
matrix {
axes {
axis {
name 'OS'
values 'windows', 'linux'
}
}
agent { label "${OS}" }
stages {
stage("Checkout") {
steps {
checkout scm
}
}
stage("Build") {
steps {
dir('esp32') {
script {
if (isUnix()) {
sh "eim run \"idf.py clean build\" v6.0.1"
} else {
bat "eim run \"idf.py clean build\" v6.0.1"
}
}
}
}
}
stage("Archive") {
steps {
archiveArtifacts artifacts: 'esp32/build/*.bin, esp32/build/*.map', fingerprint: true
}
}
}
}
}
stage("Build and push postgres container") {
agent { label 'docker-linux' }
steps {
checkout scm
sh "docker build -t septic-db:latest -f backend/Postgres.dockerfile backend"
script {
if (env.BRANCH_NAME == 'master') {
sh "docker tag septic-db:latest INTERNAL-REGISTRY/septic-db:latest"
sh "docker push INTERNAL-REGISTRY/septic-db:latest"
}
}
}
}
stage("Build frontend") {
matrix {
axes {
axis {
name 'OS'
values 'windows', 'linux'
}
}
agent { label "${OS}" }
tools {
nodejs 'node24'
}
stages {
stage("Checkout") {
steps {
checkout scm
}
}
stage("Install dependencies") {
steps {
script {
if (isUnix()) {
sh "npm install"
} else {
bat "npm install"
}
}
}
}
stage("Build") {
steps {
script {
if (isUnix()) {
sh "npm run tauri build"
} else {
bat "npm run tauri build"
}
}
}
}
stage("Archive") {
steps {
archiveArtifacts artifacts: 'src-tauri/target/release/bundle/**/*', fingerprint: true
}
}
}
}
}
stage("Compile backend") {
matrix {
axes {
axis {
name 'OS'
values 'windows', 'linux'
}
axis {
name 'CONFIGURATION'
values 'Debug', 'Release'
}
}
agent { label "${OS}" }
stages {
stage("Checkout") {
steps {
checkout scm
}
}
stage("Download dependencies") {
steps {
dir("backend") {
script {
if (isUnix()) {
sh 'dotnet restore'
} else {
bat 'dotnet restore'
}
}
}
}
}
stage("Build project") {
steps {
dir("backend") {
script {
if (isUnix()) {
sh "dotnet build --configuration ${CONFIGURATION} /m:1 /p:UseSharedCompilation=false /nodeReuse:false"
} else {
bat "dotnet build --configuration ${CONFIGURATION} /m:1 /p:UseSharedCompilation=false /nodeReuse:false"
}
}
}
}
}
stage("Archive artifacts") {
steps {
dir("backend") {
script {
def artifactPath = "**/bin/Release/**"
archiveArtifacts artifacts: "${artifactPath}", allowEmptyArchive: true, fingerprint: true
}
}
}
}
}
}
}
stage("Build and push backend container") {
agent { label 'docker-linux' }
steps {
checkout scm
sh "docker build -t septic -f backend/Backend.dockerfile backend"
script {
if (env.BRANCH_NAME == 'master') {
sh "docker tag septic INTERNAL-REGISTRY/septic"
sh "docker push INTERNAL-REGISTRY/septic"
}
}
}
}
stage("Run backend tests") {
agent { label 'docker-linux-dind' }
steps {
checkout scm
dir("backend") {
sh 'dotnet restore'
sh 'dotnet build --configuration Release /m:1 /p:UseSharedCompilation=false /nodeReuse:false'
warnError('Tests failed, but they are sometimes flaky so this could just be a bad run. You should run the tests yourself on a computer') {
sh 'dotnet test --configuration Release'
}
}
}
}
stage("MQTT plugin track") {
stages {
stage('Build MQTT plugin') {
options {
throttle(['RamIntensiveJob'])
}
matrix {
axes {
axis {
name 'OS'
values 'windows', 'linux'
}
axis {
name 'BUILD_TYPE'
values 'debug', 'release'
}
}
agent { label "${OS}" }
stages {
stage('Checkout') {
steps {
checkout scm
script {
if (isUnix()) {
sh "git submodule sync --recursive"
sh "git submodule update --init --recursive"
} else {
bat "git submodule sync --recursive"
bat "git submodule update --init --recursive"
}
}
}
}
stage('Configure') {
steps {
dir("mqtt") {
script {
if (env.OS == 'windows') {
bat """
call "C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvarsall.bat" x64
cmake --preset x64-${BUILD_TYPE}-win
"""
} else {
sh "cmake --preset x64-${BUILD_TYPE}-linux"
}
}
}
}
}
stage('Build') {
steps {
dir("mqtt") {
script {
if (env.OS == 'windows') {
bat """
call "C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvarsall.bat" x64
cmake --build out/build/x64-${BUILD_TYPE}-win
"""
} else {
sh "cmake --build out/build/x64-${BUILD_TYPE}-linux"
}
}
}
}
}
stage('Archive') {
steps {
dir("mqtt") {
script {
if (env.OS == 'windows') {
cleanCPPBuildDir("out/build/x64-${BUILD_TYPE}-win", "package-${BUILD_TYPE}-win", BUILD_TYPE == "debug")
archiveArtifacts artifacts: "package-${BUILD_TYPE}-win/septic_plugin.dll", fingerprint: true
} else {
cleanCPPBuildDir("out/build/x64-${BUILD_TYPE}-linux", "package-${BUILD_TYPE}-linux", BUILD_TYPE == "debug")
archiveArtifacts artifacts: "package-${BUILD_TYPE}-linux/septic_plugin.so", fingerprint: true
}
}
}
}
}
stage("Stash"){
steps {
dir("mqtt"){
script {
if(env.BUILD_TYPE == 'release' && env.OS == 'linux'){
stash name: 'septic-plugin', includes: 'package-release-linux/septic_plugin.so'
}
}
}
}
}
}
}
}
stage("Build MQTT broker container") {
agent { label 'docker-linux' }
steps {
checkout scm
sh 'mkdir -p mqtt/out/build/x64-release-linux'
dir('mqtt/out/build/x64-release-linux') {
unstash 'septic-plugin'
}
dir("mqtt") {
sh "docker build -t septic-mosquitto:latest ."
script {
if (env.BRANCH_NAME == 'master') {
sh "docker tag septic-mosquitto INTERNAL-REGISTRY/septic-mosquitto"
sh "docker push INTERNAL-REGISTRY/septic-mosquitto"
}
}
}
}
}
}
}
}
}
In particular, the Build ESP32 firmware stage is the big offender, on linux it takes
This seems to me like it could be some networking issue, more than hardware resources. Have you inspected the network route/traffic to see it could be impacting it - since you mentioned networking…