Why container's outer udp source port changes with VXLAN and UDP?

I am using vxlan with docker containers.
Packets from containers are sent to loadbalancer which has two connected components.

But sometimes, packets from containers have different outer(UDP) source port in transportation.
So udp loadbalancer cannot loadbalances this packets normally because they have different source port.

For example when tcp handshaking

[SYN] container port: 33922, mapped host port(udp source port): 49550
[SYN+ACK] from loadbalancer (component source port: 40034)
[ACK] container port: 33922, mapped host port(udp source port): 48173 ← changed from 49550 why?
[RST] from loadbalancer (component source port: 49844) ← because of wrong loadbalance, other component receives [ACK] packet.
[TCP Retransmission][SYN+ACK] from loadbalancer (component source port: 40034) ← didnot receive [ACK] packet.

There is one suspicious point.
It took more than 1 second for receiving [SYN+ACK] packet from lb after sending [SYN] packet.

I am wondering when and why mapped port(on host) changes?