Connectivity issue in Containers under Docker Swarm clusters

Based on a requirement we had to move our old infrastructure to our enterprise scale landing zone in Azure, we moved our Docker swarm cluster with same configuration to new foundation setup and connectivity to our App URL’s are intermittent. We suspect, it could be some DNS issue within our containers which is not able to connect DNS regularly.We have not changed any configuration to our container images. We tried checking traces on the container level but didn’t get much details. Hence any kind support would be very much appreciated

1 Like

What kind of support do you expect when you write 5 sentences, have vague errors and share zero config.

My best guess would be an MTU issue, try pinging within the Docker overlay network with a payload > 1500 bytes.

hi @bluepuma77 ,
Thanks for your response . I will upload the details(architecture and configuration) asap

Configuration details
network

docker inspect frontend
[
    {
        "Name": "frontend",
        "Id": "mj6ssbj9lwbv0x8p7ij19878f72",
        "Created": "2024-01-08T13:01:56.175311851+01:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "356b18c444514048348f2cf496a395027428c9317b9530258b5f3e9ce3fc1cba": {
                "Name": "abc-frontend.1.xxtv29s33ph9vjn2qj4gwxdlw",
                "EndpointID": "ab72b043f479fb4a0e20560397car4dbe2005b6f30875d8381dca95bc7267ebc",
                "MacAddress": "02:42:0a:00:00:08",
                "IPv4Address": "10.0.0.8/24",
                "IPv6Address": ""
            },
            "89045d9ff6889083a8ee93483e358bfed91dcc2f0e7a295411e9949d2a0702d9": {
                "Name": "observability.ms7upvp5b1tdert5oikfjq8og.rz91wd09unda9c7e9mxwrflsu",
                "EndpointID": "e8fa8f50b7917ceada1bb2639c9fd124e9471a0d62c2cf137c880a363e27a2fe",
                "MacAddress": "02:42:0a:00:00:09",
                "IPv4Address": "10.0.0.9/24",
                "IPv6Address": ""
            },
            "a4612d99c588d0f44518ae694db13655c3685d7e947caea6825d6160385f0dfe": {
                "Name": "abc-contracts.1.vfiaoar2gh2uig8i73eonb7tl",
                "EndpointID": "fc5f7bd03e0f3531b1bc295rf5f7fcc8615c488902abe5838fc2b360bff54e3e",
                "MacAddress": "02:42:0a:00:00:0c",
                "IPv4Address": "10.0.0.12/24",
                "IPv6Address": ""
            },
            "lb-frontend": {
                "Name": "frontend-endpoint",
                "EndpointID": "73ac5ebfc0d2161316798181de15ba320c9eb579fd11c023ee002d509fdd4e77",
                "MacAddress": "02:42:0a:00:00:0d",
                "IPv4Address": "10.0.0.13/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "0ba0417fd379",
                "IP": "XX.XX.XX.XX"
            },
            {
                "Name": "15e294704fd4",
                "IP": "XX.XX.XX.XX"
            },
            {
                "Name": "ef37ab323d29",
                "IP": "XX.XX.XX.XX"
            },
            {
                "Name": "1f6a198ded8d",
                "IP": "XX.XX.XX.XX"
            },
            {
                "Name": "cc6701d6decc",
                "IP": "XX.XX.XX.XX"
            },
            {
                "Name": "99cd04ecd3f5",
                "IP": "XX.XX.XX.XX"
            },
            {
                "Name": "550645b082b1",
                "IP": "XX.XX.XX.XX"
            },
            {
                "Name": "5b2df15fcf64",
                "IP": "XX.XX.XX.XX"
            }
        ]
    }
]

Service

docker service inspect abc-fe_abc-fe
[
    {
        "ID": "h648oxtgt1tjad906tbwc9njm",
        "Version": {
            "Index": 53842370
        },
        "CreatedAt": "2023-04-27T11:32:01.0637873Z",
        "UpdatedAt": "2024-02-19T08:50:14.19927354Z",
        "Spec": {
            "Name": "abc-fe_abc-fe",
            "Labels": {
                "com.docker.stack.image": "abc-fe:1.1999.0",
                "com.docker.stack.namespace": "abc-fe",
                "traefik.docker.network": "proxy",
                "traefik.frontend.headers.STSIncludeSubdomains": "true",
                "traefik.frontend.headers.STSPreload": "true",
                "traefik.frontend.headers.STSSeconds": "31536000",
                "traefik.frontend.headers.frameDeny": "true",
                "traefik.frontend.rule": "Host:test-abc.dummy.com",
                "traefik.http.middlewares.testHeader.headers.customrequestheaders.Pragma": "no-cache",
                "traefik.http.middlewares.testHeader.headers.customrequestheaders.browserXssFilter": "true",
                "traefik.http.middlewares.testHeader.headers.customrequestheaders.contentTypeNosniff": "true",
                "traefik.port": "80"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "abc-fe:1.1999.0@sha256:sdasda77a0",
                    "Labels": {
                        "com.docker.stack.namespace": "abc-fe"
                    },
                    "Env": [

                    ],
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/etc/localtime",
                            "Target": "/etc/localtime",
                            "ReadOnly": true
                        }
                    ],
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {},
                    "Secrets": [

                    ],
                    "Isolation": "default"
                },
                "Resources": {
                    "Limits": {
                        "NanoCPUs": 2000000000,
                        "MemoryBytes": 1073741824
                    },
                    "Reservations": {
                        "MemoryBytes": 1073741824
                    }
                },
                "RestartPolicy": {
                    "Condition": "on-failure",
                    "Delay": 5000000000,
                    "MaxAttempts": 0,
                    "Window": 380000000000
                },
                "Placement": {
                    "Constraints": [
                        "node.labels.private == true"
                    ],
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "mj6ssbj9lwbv0srwt8p7ij328f72",
                        "Aliases": [
                            "abc-fe"
                        ]
                    },
                    {
                        "Target": "mj6ssbj9lwbv0srwt8p7ij328f72",
                        "Aliases": [
                            "abc-fe"
                        ]
                    }
                ],
                "ForceUpdate": 3,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "Delay": 10000000000,
                "FailureAction": "rollback",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip"
            }
        },
        "PreviousSpec": {
            "Name": "abc-fe_abc-fe",
            "Labels": {
                "com.docker.stack.image": "abc-fe:1.1998.0",
                "com.docker.stack.namespace": "abc-fe",
                "traefik.docker.network": "proxy",
                "traefik.frontend.headers.STSIncludeSubdomains": "true",
                "traefik.frontend.headers.STSPreload": "true",
                "traefik.frontend.headers.STSSeconds": "31536000",
                "traefik.frontend.headers.frameDeny": "true",
                "traefik.frontend.rule": "Host:test-abc.dummy.com",
                "traefik.http.middlewares.testHeader.headers.customrequestheaders.Pragma": "no-cache",
                "traefik.http.middlewares.testHeader.headers.customrequestheaders.browserXssFilter": "true",
                "traefik.http.middlewares.testHeader.headers.customrequestheaders.contentTypeNosniff": "true",
                "traefik.port": "80"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "abc-fe:1.1998.0@sha256:ggdjjjb6fc",
                    "Labels": {
                        "com.docker.stack.namespace": "abc-fe"
                    },
                    "Env": [

                    ],
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/etc/localtime",
                            "Target": "/etc/localtime",
                            "ReadOnly": true
                        }
                    ],
                    "Secrets": [

                    ],
                    "Isolation": "default"
                },
                "Resources": {
                    "Limits": {
                        "NanoCPUs": 2000000000,
                        "MemoryBytes": 1073741824
                    },
                    "Reservations": {
                        "MemoryBytes": 1073741824
                    }
                },
                "RestartPolicy": {
                    "Condition": "on-failure",
                    "Delay": 5000000000,
                    "MaxAttempts": 0,
                    "Window": 380000000000
                },
                "Placement": {
                    "Constraints": [
                        "node.labels.private == true"
                    ],
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "mj6ssbj9lwjhjjksiiwx8p7ij198f72",
                        "Aliases": [
                            "abc-fe"
                        ]
                    },
                    {
                        "Target": "s6st369977999gy9gmw5241",
                        "Aliases": [
                            "abc-fe"
                        ]
                    }
                ],
                "ForceUpdate": 3,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "Delay": 10000000000,
                "FailureAction": "rollback",
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip"
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip"
            },
            "VirtualIPs": [
                {
                    "NetworkID": "mj6ssbj9l876tgsquwbseap7ij198f72",
                    "Addr": "10.0.0.202/24"
                },
                {
                    "NetworkID": "s6st3622yterwsdtyavd1gy9gmw5241",
                    "Addr": "10.0.3.44/24"
                }
            ]
        },
        "UpdateStatus": {
            "State": "completed",
            "StartedAt": "2024-02-19T08:48:45.442411538Z",
            "CompletedAt": "2024-02-19T08:50:14.19923734Z",
            "Message": "update completed"
        }
    }
]

hi @bluepuma77 , I have enriched the details .
Thanks

Ping results within overlay network

Usually we share docker-compose.yml files here, they are a bit more human readable that docker inspect JSON.