Handling dynamic container creation, attaching to hostname, networking

General
1

Hello everyone,

I am trying to create a platform where users can dynamically deploy 1 or more docker containers, and those containers get one hostname, and also map the ports of all containers to that hostname, where that hostname is available on the internet. All this needs to happen on one docker engine. We need to be able to dynamically create new deployments and also dynamically take them down.

Specifically, users are able to specify the image(s) they would like to deploy from our private registry and the ports to be exposed on each container (they would also include environment variables but this is not related).

We would like to use the user’s UUID to create a custom FQDN, for example $UUID.containers.example.com, and have all containers requested by that user being served on that FQDN. So if for example they asked for a postgres database and a frontend, and requested that port 80 should be forwarded on the frontend and port 5432 from the database container should be forwarded, then they would both be available at $UUID.containers.example.com:80 and $UUID.containers.example.com:5432 accordingly. Of course, before trying to launch the containers, we will check if there is a port binding issue and let the user know.

I’ve read that when setting the same hostname on two containers, then docker will automatically redirect requests depending on the port to the container that exposes that port, so I tried to use this to my advantage.

Now the issue is how to dynamically relate those FQDNs to the containers, and map their ports. Currently, I tried using coredns and this Corefile:

containers.example.com {
    forward . 127.0.0.11:53
    log
}
. {
    forward . 8.8.8.8 8.8.4.4
    log
}

with a wildcard DNS record on containers.example.com pointing to the docker host. This basically forwards requests for containers.example.com to the Docker DNS server. The issue that seems to exist is that if I run a sample docker container using
docker run -d --name nginx-sample1 --hostname uuid1.containers.example.com -p 80:80 nginx
and then another one using
docker run -d --name nginx-sample2 --hostname uuid2.containers.example.com -p 80:80 nginx
Of course I can’t bind both containers to the same port. Also, if i only bind one to that port, and leave the other container with no bound ports, requests will be routed to the one with the bound port. Hence I think this attempt is a no-go at solving this issue.

I am really not sure how I should approach this.

Also, ideally, later down the line, there is the question of allowing users to VPN into the subnet of the containers to allow them to perform debugging etc. Auth is currently a non-issue which we plan on tackling once we have a good solution for this issue. We would also like to limit the amount of containers per user to just what they requested, so we would ideally like to figure out a solution that does not need a reverse proxy per user.

I am not sure if I should be tackling this as a completely infrastructure issue or if I should try to use docker’s inbuilt tools for this.

Using traefik, or any other reverse proxy, is out of the question, because users need to be allowed to bind to any port they would like, and I cannot define every port to be an entrypoint on traefik. I can maybe force some exceptions for port bindings to specific ports.

How should I tackle this problem? I am quite a novice in networking and any thoughts of using docker’s networks seem to be restricted by the need to use a reverse proxy, which I cannot use because of the issue of allowing all ports to be used. Maybe some sort of mode like macvlan or ipvlan is actually useful in this case?

Thank you so much for reading and I hope we can solve this!

2

I don’t think that is possible in a general way. Domain names are resolved to IPs and client connect to IPs. Only when you use http the original hostname is passed in headers and when you use underlying TLS the domain name is passed (HostSNI), but you need to have access to the certs to decrypt it, so no end-to-end encryption. If it’s any other protocol you can’t determine the target hostname. Even SSH doesn’t work.

3

After a bit of deliberation, we’ve decided that for each user’s deployment, we are going to put a reverse proxy to collect all container ports to one container. We are also experimenting with ipvlan L3 to give IPs to each container. We are also deploying a DNS server where we will dynamically add new A records that point the host names to the IP of the reverse proxy. Does this sound problematic/is there a better way to approach this?

Thanks for your help @bluepuma77 !

4

I think it kind of depends on what you are trying to achieve. What kind of containers do you want to deploy, what kind of protocols will be used on what ports?

5

the problem with all this is that it could literally be anything. as long as it’s an image from the registry, and the users provided a port to be exposed, it should work. I’d be okay with excluding UDP, but I doubt that makes a difference.

I also have an idea of using nginx-proxy in network-mode host to allow for access to all ports, and then registering the new containers as they come with their VIRTUAL_HOST and VIRTUAL_PORT. Does that sound logical?

6

But if it’s plain TCP connections coming in, you will not be able to determine the target. If it’s plain SSH connections coming in, you will not be able to determine the target.

If it’s a TLS connections coming in, you might be able to do a person-in-the-middle thing, but that won’t work with all services, they can’t use tlsChallenge themselves, they can’t use mTLS, they can’t do real end-to-end encryption.

So you can offer http/s including TLS termination and forwarding of requests with http/s, but more will get complicated with a single IP for all customers.

Also be aware of websockets, which will keep the connection open for a very long time, so you need to check Traefik capacity limits.