I just tried following that same example:
git clone 'https://github.com/everett-toews/docker-dns-round-robin.git'
cd docker-dns-round-robin
docker-compose up -d
docker run --rm -it --net=dockerdnsroundrobin_backend azukiapp/dig dig apps.
docker-compose scale app=3
docker run --rm -it --net=dockerdnsroundrobin_backend azukiapp/dig dig apps.
(the azukiapp/dig image is one I found randomly on hub)
Before the scale, dig returned one address:
;; ANSWER SECTION:
apps. 600 IN A 172.21.0.3
After the scale, it returned all three containers:
;; ANSWER SECTION:
apps. 600 IN A 172.21.0.5
apps. 600 IN A 172.21.0.3
apps. 600 IN A 172.21.0.4
Since in my testing, I hit the nginx before I called scale, I ran into the same behavior that you did until I restarted the lb service with docker-compose restart lb. Then I was seeing responses from each container. Nginx will cache a DNS response for as long as the TTL. In this case, that is 600 seconds.
nginx can be configured to override this with a different value: http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver