Enabling CFS support in Ubuntu

Docker Engine General
1

I need to restrict CPU in containers, using latest Ubuntu on host (17.0.4).

$ docker info
  Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
  Images: 1
  Server Version: 17.06.0-ce
  Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  Logging Driver: json-file
  Cgroup Driver: cgroupfs
  Plugins: 
  Volume: local
  Network: bridge host macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
  Swarm: inactive
  Runtimes: runc
  Default Runtime: runc
  Init Binary: docker-init
  containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
  runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
  init version: 949e6fa
  Security Options:
    seccomp
    Profile: default
  Kernel Version: 4.9.20-std-1
  Operating System: Ubuntu 17.04
  OSType: linux
  Architecture: x86_64
  CPUs: 4
  Total Memory: 7.753GiB
  ID: 2JJV:EM37:VCY5:HVKQ:MUNV:VV5N:H247:XE5Q:VBFD:6VES:P62H:YXPJ
  Docker Root Dir: /var/lib/docker
  Debug Mode (client): false
  Debug Mode (server): false
  Registry: https://index.docker.io/v1/
  Experimental: false
  Insecure Registries:
    127.0.0.0/8
  Live Restore Enabled: false

  WARNING: No cpu cfs quota support
  WARNING: No cpu cfs period support

I’ve looked around, but I haven’t found any resources on how to actually enable CFS quote/period support on Ubuntu. Whenever I try to use --cpus or --cpu-shares I get an error:

NanoCPUs can not be set, as your kernel does not support CPU cfs period/quota or the cgroup is not mounted.

The host runs the latest Ubuntu with 4.9.20 kernel, so it should have support.
Any instructions or a good tutorial on how to mount the cgroup mentioned?
I have the same issue on latest Debian as well, help on Debian would do as well.