Using host networking in swarm mode results in random source port

Hello all,
I have a SIP based application running in a container depending on port 5060/udp. Because of how SIP stuff works, the source IP is really important. I have the application in a container running in host networking mode. Traffic arrives on 5060, which relays to the container and the app on port 5060. However the reply of that traffic comes from my application with a source port of 5060, but when it leaves it host, it is no longer on 5060. This causes the NAT near the client to fail and not be able to route the traffic.

Is there anyway to make docker use the same source port instead of randomizing it? I suspect this has something to do with the masquerading?

Thanks

Docker version 20.10.12, build 20.10.12-0ubuntu2~20.04.1
docker-compose version 1.25.0, build unknown
Linux ind-sbc-1 5.4.0-128-generic #144-Ubuntu SMP Tue Sep 20 11:00:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 20.04.5 LTS