I have the very same problem. In my case only installing the version 20.10.1 fixed the issue - because it does not assign IPv6 ports by default.
I have not found out any other way to disable IPv6 port assignment yet. My case is, that I require IPv6 available in the container itself, but the port being bound to only IPv4. Since I’m using Jenkins I can not use the syntax 0.0.0.0:80:80 for that, because I need the port bound to an arbitrary host port.
The problem accumulates due to using the classic swarm which maps the IPv6 ports to IPv4 ports, resulting in port overlaps.
Running Docker version 20.10.8, build 3967b7d on Debian 10 does not show the mentioned problem - all ports are available for IPv4 + IPv6 the way I expected it