Swarm mode and AWS ELB

I am setting up a swarm on AWS instances and traffic will be coming through an existing ELB. For this to work with swarm mode do I need to add all of the node IP addresses to the Target Zone on the ELB? In my testing, I can see that a drained manager will not forward on traffic to the nodes for a particular service so it seems like I’ll have to point to all of the worker nodes directly.