The down sides are that the containers are still taking up hard drive space and as you pointed out, you cannot remove their images until the exited containers are removed. Also if they have volumes you can’t remove the volumes of exited containers either so that’s more wasted storage space unless you are naming those volumes and reusing them.
It might be a good idea to clean these both up every one in a while with:
docker rm $(docker ps -a -q -f status=exited)
docker volume prune
Note that this will also remove containers that are just stopped.
I keep a set of database containers for development stopped so that I can quickly start them when needed with
docker start redis or
docker start mysql and stop them just as easily when no longer needed. If you keep stopped containers around then don’t use the
rm command above because it will remove those as well.