Getting pretty frustrated trying to build a swarm

Assuming that you have set everything up correctly, yes – with TLS enabled the proxy from the manager to the joined nodes should be encrypted, and identity verified, with TLS.