Existing network not visible when docker compose up

General
1 
Kernel: 6.2.0-33-generic
OS: Ubuntu 22.04.1
Docker: 24.0.6 (from Ubuntu repo)
Compose variable ELASTALERT_INSTANCE_NAME = "elastalert_blueteam01"

I have an issue which is puzzling me somewhat :slight_smile:

I have a compose file with the following networks:

  elastalert_${ELASTALERT_INSTANCE_NAME}:
    driver: bridge
  es_network:
    name: ${ES_NETWORK}
    external: true

One of the services (‘elastalert_blueteam01’ in this case) uses both:

      - es_network
      - elastalert_${ELASTALERT_INSTANCE_NAME}

The other (‘frontend’) uses only one:
- elastalert_${ELASTALERT_INSTANCE_NAME}

The ‘es_network’ is running an Elastic-stack (Elasticsearch stack) and is only used by ‘elastalert’ in this compose file. However, the ‘frontend’ service uses the ‘elastalert’ network above.

If I issue a docker network ls command it shows like this:

NETWORK ID     NAME                    DRIVER    SCOPE
84660b1762b6   elastalert_blueteam01   bridge    local

and a docker network ls gives:

[
    {
        "Name": "elastalert_blueteam01",
        "Id": "84660b1762b6031a25a387798883b7627949ca1ab997008991f666460abd2bd9",
        "Created": "2023-10-06T09:45:46.639444919+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.21.0.0/16",
                    "Gateway": "172.21.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]

When I issue docker compose up I get an
service "elastalert" refers to undefined network elastalert_blueteam01: invalid compose project
error. I thought compose allowed “dynamic” network mappings based on env-variables. What am I missing here?

Although this is not a “show stopper” for me it would be great to be able to “hide” the traffic between elastalert_xxx and frontend services. Keeping it on the bridged network I may use the same ports although I might deploy additional copies (with their own “elastalert_xxxx” bridged network). If I can’t solve this I’ll have to perform some portmappings with different mappings/deployment. Doable but not so funny and very error prone although I’ll provision with, say, ansible.

Really hope there are some more seasoned Docker gurus out there :slight_smile:

2

Unless there was an update for Docker Compose, I don’t think you can use a variable in a parameter name without using envsubst for example in a bashs hell to render your yaml file. You could do something like this:

elastalert:
  name: ${ELASTALERT_INSTANCE_NAME}
  driver: bridge

And I don’t know if it was intentional, but you have a variable with the value “elastalert_blueteam01” and tried to append it to “elastalert_” restulting “elastalert_elastalert_blueteam01”

3

Hi!

Many thanks for your answer! I ignored some parts of the text which makes it look like I tried to concatenate ‘elastalert_’ twice - sorry for that, it’s not the case - the network name is rendered correctly which is clear in the error message from docker compose. I find this a little strange as docker compose up is able to interpolate the concatenation of ‘elastalert_’ and ${ELASTALERT_INSTANCE} but fails to create/refer to an existing network. An ‘envsubst’ could definitely work albeit it feels a little “rough” :slight_smile:

Again, many thanks for your reply. I really want to understand the how docker uses the compose file and this is something I’d really like to understand.

4

Oh, sorry, I missed that part. For an existing network you need the external: true parameter.

elastalert:
  external: true
  name: ${ELASTALERT_INSTANCE_NAME}
  driver: bridge

Compose will always create its own network for the project otherwise. Did you use that parameter too?

5

No problem - I sometimes jump ahead in my reasoning and forget parts to write :slight_smile: The problem is that the ‘elastalert_xxxxx’ network should be created by this compose file if it doesn’t exist and re-used if it has been created already. docker network ls reports that it has been created in an earlier run but it fails to re-use it as it thinks it is not present. The ‘elastalert_xxx’ network must only exist within the context of this compose file as I only want the containers created by the file should be able to communicate.

I hope this makes sense.

6

Can you share your actual compose file? If you want a compose project to use a network, you don’t need to add the prefix in the compose file. Since you mentioned you didn’t share some parts that could help to understand the issue, I’m still not sure I understand it correctly.

You showed that elastalert_blueteam01 network existed, but you also used the prefix in the compose file ( and in the variable). Even if you removed the prefix from the variable name, you still need to remove it from the compose file or at leasst set the name parameter which I used only with the external flag


7

Hi! I think I might have found a workaround. I’ll look into it next week and keep you updated. Many thanks for your help!

Have a nice weekend!