How to avoid transferring everything as build context

Here is the directory structure posted above
mydir/bin/heavy1.zip
mydir/bin/heavy2.zip

mydir/bin/heavy10.zip
mydir/scripts/script1.sh
mydir/scripts/script2.sh

mydir/scripts/script10.sh
mydir/scripts/dockerfile

Here is docker file content
FROM some_base
COPY bin/heavy1.zip /tmp

Here is docker build run from directory “mydir”
docker build -t test -f scripts/Dockerfile .