A node can only be removed from a Swarm if it’s in the Down state so… sure it would be safe to remove them as no task will be running on them. A node in the Unknown state could mean that it never got added to the Swarm so it would be safe to remove it too.
Your question about it those nodes are recoverable. Well, a node is considered to be down either if it can be reachable because there is a network issue or because it was powered off / shutted down. If it’s a network issue you may recover those nodes, if it’s a token issue you may have to try to rejoin them to the swarm.