Docker Community Forums

Share and learn in the Docker community.

ELB keeps sporadically resetting its listener ports

(Kevinmcgarry) #1

Docker for AWS 17.04.0-ce-aws1

I created a workaround for my prior issue where the ELB created and maintained by the Docker for AWS template would reset its listeners each time I would deploy a change to the swarm. Essentially I have my services/containers listening on a set of ports that are different from my ELB listeners as I want my customers to terminate their SSL traffic at the ELB. So for example,

my internal api (running in a container in the swarm) is listening/exposed on port 2112 while the ELB is listening on SSL port 443 and it hands off the unencrypted traffic to the swarm at port 2112. Of course the problem is every time I make a change to the swarm, the ELB listeners are reset to the internal ports that that the services are listening (and the external ports are deleted). I was able to work around this by configuring my CI server to make api calls and reconfigure the ELB after each new deployment. THIS WORKS GREAT. BUT …

The ELB is resetting itself after a few days to the original listener ports (deleting the external ports/ssl certs) WITHOUT ANY CHANGES being made to the swarm. I am just noticing this now as our deployments are scaling down as we approach beta testing phase.

For example, I made a deployment to the swarm this past Friday and everything worked great. Sunday morning, I get an alert that the apis are no longer reachable. After connecting to the AWS console, I see that the SSL ports that I configured are gone and the listener ports are back to their defaults. Nothing changed after my last deployment - no new services, no scaling, nothing. This also happened a couple of weeks ago but I just figured someone perhaps made an unschedule change.

I looked through all my logs but can’t see anything in there that would explain why the ELB all of a sudden reset its listener ports. Nothing on my end made a call to it. Its like for some reason the ELB reverted to the DEFAULT CloudFormation Template desired state and removed all my configurations.

And I can’t just shift over to using a completely different statically configured ELB as I need the ability to for the ASG to be able to automatically add and remove worker nodes in the ELB on the fly.

Again, everything works great except for the sporadic resetting of my ELB configured listener ports.

I am going to keep an eye on it (as I won’t be touching the stack for a few days) and hopefully I can pin down how long it takes before they reset.

(Kevinmcgarry) #2

on a side but related note - how can we decouple the auto-updating of the listener ports on the ELB from the service startup process in the swarm? Meaning, the most important automated process with the ELB should be the registering/deregistering of the EC2 instances (manager and worker nodes) as they are added/removed by the AutoScaling Group. This process should definitely stay in place. On the other hand, I would like to remove the auto-updating of the ELB listener ports so that the admin of the cluster would be able to statically configure that info.

Is there any way to adjust the cloudformation template to allow this right now?

(Mgpoly) #3

Right: I want to start a private/internal/non-public swarm that can only be accessed through VPN. I don’t want the automatic public IPs and I don’t wan’t the holes punched through the LB.

Looks like the l4controller container controlls LB configuration. Maybe that can be modified. From the FAQ: