Same for me on Docker for Mac beta Version 1.12.0-beta-rc2-beta16 (build: 9493). Removing all images and containers manually didn’t help, Reset to Factory defaults worked.
I fixed it by the error suggested: “director not empty”, then go delete everything under that directory.
You can do by running those command to clean up the folder:
$ docker-machine ssh
docker@default$ sudo su
root@default$ cd /mnt/sda1/var/lib/docker/image/aufs/layerdb/sha256/ddef881aeaffef528aa8fecd090a9b68dd4fabb6f653c3c1b689c133b07e38b0
root@default$ rm -rf ./*