NVIDIA AGX JetPack 4.5 with NVMe SSD - docker daemon not able to run

Hello,

I’m trying to use docker with my AGX that is using an NVMe SSD with JetPack 4.5 to do some AI/ML work. I am unable to launch docker and I thought I would reach out to see if anyone here has some ideas. Previously, before installing the NVMe SSD, docker worked just fine.

Any help or suggestions are greatly appreciated!!

These are the logs when trying to install docker.

agx@agx-desktop:~$  sudo apt-get install docker-ce docker-ce-cli containerd.io
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  apt-clone archdetect-deb bogl-bterm busybox-static cryptsetup-bin dpkg-repack gir1.2-timezonemap-1.0 gir1.2-xkl-1.0 grub-common gyp kde-window-manager kpackagetool5
  kwayland-data kwin-common kwin-data kwin-x11 libc-ares2 libdebian-installer4 libhttp-parser2.7.1 libjs-async libjs-inherits libjs-node-uuid libjs-underscore
  libkdecorations2-5v5 libkdecorations2private5v5 libkf5activities5 libkf5declarative-data libkf5declarative5 libkf5globalaccelprivate5 libkf5idletime5
  libkf5kcmutils-data libkf5kcmutils5 libkf5package-data libkf5package5 libkf5plasma5 libkf5quickaddons5 libkf5waylandclient5 libkf5waylandserver5 libkscreenlocker5
  libkwin4-effect-builtins1 libkwineffects11 libkwinglutils11 libkwinxrenderutils11 libqgsttools-p1 libqt5designer5 libqt5help5 libqt5multimedia5
  libqt5multimedia5-plugins libqt5multimediaquick-p5 libqt5multimediawidgets5 libqt5quickwidgets5 libuv1-dev libxcb-composite0 libxcb-cursor0 libxcb-damage0
  node-abbrev node-ansi node-ansi-color-table node-archy node-async node-balanced-match node-block-stream node-brace-expansion node-combined-stream node-concat-map
  node-cookie-jar node-delayed-stream node-forever-agent node-form-data node-fs.realpath node-fstream node-fstream-ignore node-github-url-from-git node-glob
  node-graceful-fs node-hosted-git-info node-inflight node-inherits node-ini node-isexe node-json-stringify-safe node-lockfile node-lru-cache node-mime node-minimatch
  node-mkdirp node-mute-stream node-node-uuid node-nopt node-npmlog node-once node-osenv node-path-is-absolute node-pseudomap node-qs node-read node-request node-retry
  node-rimraf node-semver node-sha node-slide node-spdx-correct node-spdx-expression-parse node-spdx-license-ids node-tar node-tunnel-agent node-underscore
  node-validate-npm-package-license node-which node-wrappy node-yallist nodejs-doc os-prober python3-dbus.mainloop.pyqt5 python3-icu python3-pam python3-pyqt5
  python3-pyqt5.qtsvg python3-pyqt5.qtwebkit python3-sip qml-module-org-kde-kquickcontrolsaddons qml-module-qtmultimedia qml-module-qtquick2 rdate tasksel tasksel-data
  ubuntu-fan
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  containerd.io docker-ce docker-ce-cli
0 upgraded, 3 newly installed, 0 to remove and 13 not upgraded.
Need to get 0 B/75.9 MB of archives.
After this operation, 373 MB of additional disk space will be used.
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package containerd.io.
(Reading database ... 209740 files and directories currently installed.)
Preparing to unpack .../containerd.io_1.4.4-1_arm64.deb ...
Unpacking containerd.io (1.4.4-1) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../docker-ce-cli_5%3a20.10.6~3-0~ubuntu-bionic_arm64.deb ...
Unpacking docker-ce-cli (5:20.10.6~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../docker-ce_5%3a20.10.6~3-0~ubuntu-bionic_arm64.deb ...
Unpacking docker-ce (5:20.10.6~3-0~ubuntu-bionic) ...
Setting up containerd.io (1.4.4-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up docker-ce-cli (5:20.10.6~3-0~ubuntu-bionic) ...
Setting up docker-ce (5:20.10.6~3-0~ubuntu-bionic) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Thu 2021-05-06 09:04:02 EDT; 14ms ago
     Docs: https://docs.docker.com
  Process: 23944 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
 Main PID: 23944 (code=exited, status=1/FAILURE)
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for systemd (237-3ubuntu10.46) ...
Errors were encountered while processing:
 docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)
agx@agx-desktop:~$ 

agx@agx-desktop:~$ sudo systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2021-05-06 09:04:11 EDT; 1min 22s ago
     Docs: https://docs.docker.com
  Process: 24312 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
 Main PID: 24312 (code=exited, status=1/FAILURE)

May 06 09:04:09 agx-desktop systemd[1]: Failed to start Docker Application Container Engine.
May 06 09:04:11 agx-desktop systemd[1]: docker.service: Service hold-off time over, scheduling restart.
May 06 09:04:11 agx-desktop systemd[1]: docker.service: Scheduled restart job, restart counter is at 4.
May 06 09:04:11 agx-desktop systemd[1]: Stopped Docker Application Container Engine.
May 06 09:04:11 agx-desktop systemd[1]: docker.service: Start request repeated too quickly.
May 06 09:04:11 agx-desktop systemd[1]: docker.service: Failed with result 'exit-code'.
May 06 09:04:11 agx-desktop systemd[1]: Failed to start Docker Application Container Engine.
agx@agx-desktop:~$ 

agx@agx-desktop:~$ journalctl -xe
May 06 09:04:22 agx-desktop compiz[8409]: nvdc: failed to open '/dev/tegra_dc_ctrl'.
May 06 09:04:22 agx-desktop compiz[8409]: Error: Can't initialize nvrm channel
May 06 09:04:22 agx-desktop compiz[8409]: nvdc: open: Operation not permitted
May 06 09:04:22 agx-desktop compiz[8409]: nvdc: failed to open '/dev/tegra_dc_ctrl'.
May 06 09:04:31 agx-desktop dbus-daemon[5105]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' re
May 06 09:04:31 agx-desktop systemd[1]: Starting Hostname Service...
-- Subject: Unit systemd-hostnamed.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit systemd-hostnamed.service has begun starting up.
May 06 09:04:31 agx-desktop dbus-daemon[5105]: [system] Successfully activated service 'org.freedesktop.hostname1'
May 06 09:04:31 agx-desktop systemd[1]: Started Hostname Service.
-- Subject: Unit systemd-hostnamed.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit systemd-hostnamed.service has finished starting up.
-- 
-- The start-up result is RESULT.
May 06 09:05:34 agx-desktop sudo[24514]:      agx : TTY=pts/7 ; PWD=/home/agx ; USER=root ; COMMAND=/bin/systemctl status docker.service
May 06 09:05:34 agx-desktop sudo[24514]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 06 09:05:34 agx-desktop sudo[24514]: pam_unix(sudo:session): session closed for user root