We have a few servers running Docker CE version 18.09.2 and calico/node v2.6.11 on Centos 3.10.0-862.14.4.el7.x86_64. I could run containers within a calico network without any issues. I recently upgraded a server to centos version 3.10.0-957.12.1.el7.x86_64 and now whenever I run a container within a calico network on that server, I get an error similar to this:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:407: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\"2019-05-07T23:45:01Z\\\\\\\" level=fatal msg=\\\\\\\"failed to add interface temp447b85d6ad0 to sandbox: error setting interface \\\\\\\\\\\\\\\"temp447b85d6ad0\\\\\\\\\\\\\\\" routes to [\\\\\\\\\\\\\\\"169.254.1.1/32\\\\\\\\\\\\\\\" \\\\\\\\\\\\\\\"fe80::4c42:4eff:fe54:355e/128\\\\\\\\\\\\\\\"]: permission denied\\\\\\\"\\\\n\\\"\"": unknown.
time="2019-05-07T23:45:02Z" level=error msg="error waiting for container: context canceled"
The only error message I see within /var/log/messages is
May 8 16:25:33 tupsjc03xdcslv001 dockerd[8402]: time="2019-05-08T16:25:33.091558151Z" level=error msg="stream copy error: reading from a closed fifo"
If I switch this to network=host, it works. This happens for all images I have tried (including hello-world). I updated docker to 18.09.5 and calico/node to v2.6.12 with no changes.
Any ideas on what is going on? I would love to know exactly what command is running as part of “prestart hook 0” but I don’t know where to find this.