How to specify max log json file size in docker compose?

It seems to me you had a typo the first try. According to the docs, the setting should be called log_opt instead of log-opt.

I tried the following YML (docker 1.12.1, docker-compose 1.8.0) and it works:

elasticsearch:
  image: elasticsearch
  log_opt:
    max-size: 50m