SSH forwarding in docker build fails on CentOS

I am trying to use SSH Forward feature in docker build as mentioned in 18.09 release notes. But not able to run it successfully. My host machine has CentOS 7.

Docker -v
Docker version 18.09.0, build 4d60db4

Below is the build command:
docker build --ssh default .

Command inside Dockerfile:

RUN — mount=type=ssh git clone<my-org>/<my-project.git>

Throws below error : Permission denied (publickey).
#12 2.653 fatal: Could not read from remote repository.
#12 2.653
#12 2.653 Please make sure you have the correct access rights
#12 2.653 and the repository exists.

I have cross checked that SSH-agent is running on my host machine and also tried passing key in docker build.

Can someone please point out where I am doing wrong.