How to limit or rotate Docker container logs?

No, Docker Compose does not have different configuration but it does not affect already created containers.
The other problem could be that your docker does not read that configuration file. On a linux host I could change log driver options using that config file but I could not on Windows in a WSL machine.