Containers can't communicate to each other

Hi there.
I believe this topic was raised not once. But I couldn’t find any reasonable solution. So here it is…
I have deployed Zabbix in containers. I have only one host where 3 containers run: zabbix-server, zabbix-web-nginx-mysql, zabbix-agent. All 3 are defined in a single docker-compose.yaml:

version: '3.5'
networks:
 zbx_net:
  driver: bridge

services:
 zabbix-server:
  image: zabbix/zabbix-server-mysql:alpine-5.2-latest
  ports:
   - "10051:10051"
  volumes:
   - /etc/localtime:/etc/localtime:ro
   - /etc/timezone:/etc/timezone:ro
   - ./zbx_env/usr/lib/zabbix/alertscripts:/usr/lib/zabbix/alertscripts:ro
   - ./zbx_env/usr/lib/zabbix/externalscripts:/usr/lib/zabbix/externalscripts:ro
   - ./zbx_env/var/lib/zabbix/export:/var/lib/zabbix/export:rw
   - ./zbx_env/var/lib/zabbix/modules:/var/lib/zabbix/modules:ro
   - ./zbx_env/var/lib/zabbix/enc:/var/lib/zabbix/enc:ro
   - ./zbx_env/var/lib/zabbix/ssh_keys:/var/lib/zabbix/ssh_keys:ro
   - ./zbx_env/var/lib/zabbix/mibs:/var/lib/zabbix/mibs:ro
  ulimits:
   nproc: 65535
   nofile:
    soft: 20000
    hard: 40000
  env_file:
   - .env_db_mysql
   - .env_srv
  secrets:
   - MYSQL_USER
   - MYSQL_PASSWORD
  networks:
   zbx_net:
    aliases:
    - zabbix-server
    - zabbix-server-mysql
    - zabbix-server-alpine-mysql
    - zabbix-server-mysql-alpine
  stop_grace_period: 30s
  sysctls:
   - net.ipv4.ip_local_port_range=1024 65000
   - net.ipv4.conf.all.accept_redirects=0
   - net.ipv4.conf.all.secure_redirects=0
   - net.ipv4.conf.all.send_redirects=0
  labels:
   com.zabbix.description: "Zabbix server with MySQL database support"
   com.zabbix.company: "Zabbix LLC"
   com.zabbix.component: "zabbix-server"
   com.zabbix.dbtype: "mysql"
   com.zabbix.os: "alpine"

 zabbix-web-nginx-mysql:
  image: zabbix/zabbix-web-nginx-mysql:alpine-5.2-latest
  ports:
   - "8081:8080"
   - "8443:8443"
  volumes:
   - /etc/localtime:/etc/localtime:ro
   - /etc/timezone:/etc/timezone:ro
   - ./zbx_env/etc/ssl/nginx:/etc/ssl/nginx:ro
   - ./zbx_env/usr/share/zabbix/modules/:/usr/share/zabbix/modules/:ro
  env_file:
   - .env_db_mysql
   - .env_web
  secrets:
   - MYSQL_USER
   - MYSQL_PASSWORD
  depends_on:
   - zabbix-server
  healthcheck:
   test: ["CMD", "curl", "-f", "http://localhost:8080/"]
   interval: 10s
   timeout: 5s
   retries: 3
   start_period: 30s
  networks:
   zbx_net:
    aliases:
     - zabbix-web-nginx-mysql
     - zabbix-web-nginx-alpine-mysql
     - zabbix-web-nginx-mysql-alpine
  stop_grace_period: 10s
  sysctls:
   - net.core.somaxconn=65535
  labels:
   com.zabbix.description: "Zabbix frontend on Nginx web-server with MySQL database support"
   com.zabbix.company: "Zabbix LLC"
   com.zabbix.component: "zabbix-frontend"
   com.zabbix.webserver: "nginx"
   com.zabbix.dbtype: "mysql"
   com.zabbix.os: "alpine"

 zabbix-agent:
  image: zabbix/zabbix-agent:alpine-5.2-latest
  ports:
   - "10050:10050"
  volumes:
   - /etc/localtime:/etc/localtime:ro
   - /etc/timezone:/etc/timezone:ro
   - ./zbx_env/etc/zabbix/zabbix_agentd.d:/etc/zabbix/zabbix_agentd.d:ro
   - ./zbx_env/var/lib/zabbix/modules:/var/lib/zabbix/modules:ro
   - ./zbx_env/var/lib/zabbix/enc:/var/lib/zabbix/enc:ro
   - ./zbx_env/var/lib/zabbix/ssh_keys:/var/lib/zabbix/ssh_keys:ro
  env_file:
   - .env_agent
  privileged: true
  pid: "host"
  networks:
   zbx_net:
      aliases:
       - zabbix-agent
       - zabbix-agent-passive
       - zabbix-agent-alpine
  stop_grace_period: 5s
  labels:
   com.zabbix.description: "Zabbix agent"
   com.zabbix.company: "Zabbix LLC"
   com.zabbix.component: "zabbix-agentd"
   com.zabbix.os: "alpine"
secrets:
  MYSQL_USER:
    file: ./.MYSQL_USER
  MYSQL_PASSWORD:
    file: ./.MYSQL_PASSWORD

All 3 containers are in same user-defined network. So according to Docker Compose documentation it should be enough that they are able to communicate. However if try to nc zabbix-server 10051 from zabbix-agent container the connection fails and I don’t see any connection attempt at zabbix-server.
I checked whether it can be caused by the firewall. I use nftables and created a rule allowing all traffic between 172.0.0.0/8 and 172.0.0.0/8. It helps partially in a way that I can connect to :10051 (like 172.26.0.1:10051). Although the zabbix-agent log still shows error that connection is refused so I think it still doesn’t work properly.

Anyway my primary concern is ability of containers to communicate to each other directly as it’s supposed to be. Also I tried to connect from zabbix-agent to zabbix-web-nginx-mysql:8081 and it also didn’t connect. So it’s my understanding that inter-container communication doesn’t work at all.

If I look at the zabbix_zbx_net network I can see all 3 containers are there:

[
    {
        "Name": "zabbix_zbx_net",
        "Id": "def0d254c1077d3874c74ebd6f93a9a9895683a2cc97ffe53a0fa2524649f790",
        "Created": "2020-12-22T10:02:59.55942359+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.27.0.0/16",
                    "Gateway": "172.27.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "95ed96686a7607b5c8aa22bc86f69916dcc117ae118859a5254d3e001df70de9": {
                "Name": "zabbix_zabbix-server_1",
                "EndpointID": "98e7c97886308716b5bf85bf5c8a4bb9655df9e3d79d34a05c8b9d6bca10ae15",
                "MacAddress": "02:42:ac:1b:00:03",
                "IPv4Address": "172.27.0.3/16",
                "IPv6Address": ""
            },
            "ab9216585795561226e608dc5f8a074de3d551f4e09f4caba48a111ec2d89c2b": {
                "Name": "zabbix_zabbix-web-nginx-mysql_1",
                "EndpointID": "0159042e1b64b7ac7f5ca3d675b7a855fa7d22aa42b4765877f4f09723f73307",
                "MacAddress": "02:42:ac:1b:00:04",
                "IPv4Address": "172.27.0.4/16",
                "IPv6Address": ""
            },
            "d17b6314e3f0ade5af7f7bf770fa72f82677e3d6fe82b62bea3ae05567ceb836": {
                "Name": "zabbix_zabbix-agent_1",
                "EndpointID": "8799fe8af03e945fb020d39e66912a20501c08d7b715b3f4aeed531f57392c65",
                "MacAddress": "02:42:ac:1b:00:02",
                "IPv4Address": "172.27.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "zbx_net",
            "com.docker.compose.project": "zabbix",
            "com.docker.compose.version": "1.25.0"
        }
    }
]

So I don’t know what else can be checked.