Docker Networking Question UDP Ports being re-written on outbound

I have a Dedicated Server Application in a docker container that both lists itself on a public directory of servers and listens to accept clients connecting via the public directory of servers.

Communication is designed to have clients connecting on a specific UDP port to access the server. We have a firewall rule to allow that port inbound from the internet directly to this server and docker has the port published.

Where we have been seeing the issue is on outbound connections to the directory. Specifically the traffic is supposed to leave the Docker VM on the same port as the inbound connections it is listening for with a destination of the same port on the remote end. This is different behaviour than more applications that use a random local port and specific destination port. The reason for this is to allow the server to record the IP and port combination of that connection and list it in the directory.

It seems to be the docker host doing the remapping of the outbound traffic because the outbound will stay on that random port through reboots but as soon as I switch from a published port to using the host network and then back to bridged it will switch the outbound udp port to a new random port number.

Is there a way to change this behaviour? or is this due to the fact that docker is essentially working like a firewall and doing it’s typical NAT and PAT operations? Is there a way to make this work by connecting to the HOST network directly instead of using the published ports?