Docker overlay host to container network not working

General
1

I am trying to create overlay docker network without swarm Link using consul multi host in aws ec2 instance AMI. I have created a overlay network and able to communicate with the containers . However when I am trying to reach from host to container specific port getting connection refused error . Host is not having any firewall. How I can reach the container port on overlay network .

Strange fact

  1. If I put the container without overlay network I am able to ping the container port from same host machine as well as other overlay network connected machine .
[ec2-user@ip-172-31-3-222 ~]$ sudo docker run --name mynginx1 -p 5080:80 -d nginx

33a120ec5616fde53cfe4242055379c708a37451ce6d7ff12427ec7cdfce2f69

Working :

[ec2-user@ip-172-31-3-222 ~]$ telnet 172.31.3.222 5080
Trying 172.31.3.222...
Connected to 172.31.3.222.
Escape character is '^]'.
  1. If I put the container overlay network I am not able to ping the container from host machine as well as other host machine. .
[ec2-user@ip-172-31-3-222 ~]$ sudo docker run --net my-overlay --name mynginxoverlay -p 5081:80 -d nginx

af30418ad4c930ee4adfc231880c7855f972e2765df9facba074955f200e164c
[ec2-user@ip-172-31-3-222 ~]$ telnet 172.31.3.222 5081
Trying 172.31.3.222…

My overlay network:

docker network create -d overlay --subnet=172.31.0.0/20 my-overlay

details:

[ec2-user@ip-172-31-7-118 ~]$ sudo docker inspect my-overlay
[
    {
        "Name": "my-overlay",
        "Id": "f4f2c67649ee108c9c88f224687a0ac1590a63f7ac06544cfe6dbd88aa2d773a",
        "Created": "2019-06-22T15:09:13.739759915Z",
        "Scope": "global",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.31.0.0/20"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "ep-2bf37ede7b867f3b4059755d482da6e98557896d5c64c06c7e8655b4892e01a1": {
                "Name": "mynginxoverlay",
                "EndpointID": "2bf37ede7b867f3b4059755d482da6e98557896d5c64c06c7e8655b4892e01a1",
                "MacAddress": "02:42:ac:1f:00:02",
                "IPv4Address": "172.31.0.2/20",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }