Are you in the happy position to have a Docker Enterprie subscription? Docker EE includes UCP, which provides a user and ressource managent that allows to restrict access to ressources on a fine grain level even for swarm stack deployments. Each user downloads a client bundle with the connection details to controll the swarm thru UCP.
Without Docker Enterprise, the answer is: nope, everyone who can control docker on a master node will be able to controll all objects, including secrets.
Kuberentes has fine grained RBAC security build in. Swarm byitself does not.