How to connect to ikev2 vpn from docker container using bridge net mode?

I’m trying to connect to a VPN from inside a docker container via ikev2, Ubuntu host.

If I try the following:

docker run -i -t --privileged --net host --entrypoint /bin/bash ikev

It then do ipsec start and ipsec up vpn it connects without an issue. Problem is this will impact routes on the host as well so I wish to run it from the net bridge mode, but if I run:

docker run -i -t --privileged --entrypoint /bin/bash ikev

I can no longer connect to the vpn, I basically can't even connect to the ikev2 vpn server anymore:

initiating IKE_SA vpn[1] to X.X.X.X
generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
sending packet: from Y.Y.Y.Y[500] to X.X.X.X[500] (454 bytes)
retransmit 1 of request with message ID 0
sending packet: from Y.Y.Y.Y[500] to X.X.X.X[500] (454 bytes)
retransmit 2 of request with message ID 0
[...]

I have tried from both container with nmap to figure out if something is filtered but got same results from both --net host and bridge:

PROTOCOL STATE         SERVICE
50       open|filtered esp
51       open|filtered ah
PORT     STATE         SERVICE
500/udp  open          isakmp
4500/udp open|filtered nat-t-ike

What should I forward or do from the host so that the container can connect to the ikev2 server ?