I am not sure if it realy is the solution, years ago I when I was using ngnix as a reverse, I don’t remember having that sort of problem with a configuration like this:
Though, it might be possible that nginx had a different behavior back in those days. If I remember correct it had to use proxypass with variables that pointed to the target, as using proxypass with upstream was problematic.
If it’s not the solution, then I would strongy suggest to use Traefik instead. Traefik listens on docker events and registers/unregisters reverse proxy rules depeding of container labels of the target container and its lifecycle.