Is rebuilding your image from scratch not possible?
If there are a lot of build artifacts, it’s not totally crazy to do something like:
FROM debian
# Since this script builds and removes everything in a single layer,
# no build artifacts will be persisted
COPY ./build-and-rm-artifacts.sh /build.sh
RUN /build.sh
ENTRYPOINT ["my-executable"]