How to avoid transferring everything as build context

How moving the docker file one level up will prevent from sending whole mydir to the docker daemon?

I don’t insist to keep any exact structure. I am ready to consider any working solution which will not send all the binaries to the docker daemon. So far the only option I see is where I keep the binaries somewhere outside of current directory (outside of build context) and copy the binary required for each particular image somewhere in the “build context” and then remove after that every time.