My docker container freeze after about ~ 14 min or so.
Machine details are :
Docker version 1.8.2-el7.centos, build a01dc02/1.8.2
Linux ip-172-19-3-15 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Sep 22 19:06:58 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.0.1406 (Core)
Here is the snippet from /var/log/messages for the container that freeze :
Feb 26 15:46:30 ip-172-19-3-15 docker: time=“2016-02-26T15:46:30.402703868-08:00” level=info msg="POST /containers/4b189ef0da2bd3de1904fbd3e5afa1e65b4aa798e64b7611e5a2ccc58d99c75f/start"
Feb 26 15:46:30 ip-172-19-3-15 kernel: bio: create slab at 2
Feb 26 15:46:30 ip-172-19-3-15 kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts:
Feb 26 15:46:30 ip-172-19-3-15 kernel: device veth7117dfd entered promiscuous mode
Feb 26 15:46:30 ip-172-19-3-15 kernel: IPv6: ADDRCONF(NETDEV_UP): veth7117dfd: link is not ready
Feb 26 15:46:30 ip-172-19-3-15 kernel: docker0: port 1(veth7117dfd) entered forwarding state
Feb 26 15:46:30 ip-172-19-3-15 kernel: docker0: port 1(veth7117dfd) entered forwarding state
Feb 26 15:46:30 ip-172-19-3-15 kernel: docker0: port 1(veth7117dfd) entered disabled state
Feb 26 15:46:30 ip-172-19-3-15 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth7117dfd: link becomes ready
Feb 26 15:46:30 ip-172-19-3-15 kernel: docker0: port 1(veth7117dfd) entered forwarding state
Feb 26 15:46:30 ip-172-19-3-15 kernel: docker0: port 1(veth7117dfd) entered forwarding state
Feb 26 15:46:30 ip-172-19-3-15 systemd: Starting docker container 4b189ef0da2bd3de1904fbd3e5afa1e65b4aa798e64b7611e5a2ccc58d99c75f.
Feb 26 15:46:30 ip-172-19-3-15 systemd: Started docker container 4b189ef0da2bd3de1904fbd3e5afa1e65b4aa798e64b7611e5a2ccc58d99c75f.
Feb 26 15:46:30 ip-172-19-3-15 docker: time=“2016-02-26T15:46:30.640870861-08:00” level=info msg="GET /containers/4b189ef0da2bd3de1904fbd3e5afa1e65b4aa798e64b7611e5a2ccc58d99c75f/json"
Feb 26 15:46:45 ip-172-19-3-15 kernel: docker0: port 1(veth7117dfd) entered forwarding state
Feb 26 15:48:39 ip-172-19-3-15 docker: time=“2016-02-26T15:48:39.975510421-08:00” level=info msg="GET /containers/json?all=0&size=0"
Feb 26 15:48:39 ip-172-19-3-15 docker: time=“2016-02-26T15:48:39.978988087-08:00” level=info msg="GET /containers/json?all=0&size=0"
Feb 26 15:48:49 ip-172-19-3-15 docker: time=“2016-02-26T15:48:49.979294873-08:00” level=info msg="GET /containers/json?all=0&size=0"
Feb 26 15:48:49 ip-172-19-3-15 docker: time=“2016-02-26T15:48:49.982720914-08:00” level=info msg="GET /containers/json?all=0&size=0"
Feb 26 15:48:59 ip-172-19-3-15 docker: time=“2016-02-26T15:48:59.979253428-08:00” level=info msg="GET /containers/json?all=0&size=0"
Feb 26 15:48:59 ip-172-19-3-15 docker: time=“2016-02-26T15:48:59.982480352-08:00” level=info msg="GET /containers/json?all=0&size=0"
Feb 26 15:53:59 ip-172-19-3-15 docker: time=“2016-02-26T15:53:59.978788831-08:00” level=info msg="GET /containers/json?all=0&size=0"
Feb 26 15:53:59 ip-172-19-3-15 docker: time=“2016-02-26T15:53:59.982364046-08:00” level=info msg="GET /containers/json?all=0&size=0"
Feb 26 16:00:41 ip-172-19-3-15 docker: time=“2016-02-26T16:00:41.348431463-08:00” level=info msg="POST /containers/4b189ef0da2bd3de1904fbd3e5afa1e65b4aa798e64b7611e5a2ccc58d99c75f/stop?t=10"
Feb 26 16:00:41 ip-172-19-3-15 systemd: Stopping docker container 4b189ef0da2bd3de1904fbd3e5afa1e65b4aa798e64b7611e5a2ccc58d99c75f.
Feb 26 16:00:41 ip-172-19-3-15 systemd: Stopped docker container 4b189ef0da2bd3de1904fbd3e5afa1e65b4aa798e64b7611e5a2ccc58d99c75f.
Feb 26 16:00:41 ip-172-19-3-15 kernel: docker0: port 1(veth7117dfd) entered disabled state
Feb 26 16:00:41 ip-172-19-3-15 kernel: docker0: port 1(veth7117dfd) entered disabled state
Feb 26 16:00:41 ip-172-19-3-15 kernel: device veth7117dfd left promiscuous mode
Feb 26 16:00:41 ip-172-19-3-15 kernel: docker0: port 1(veth7117dfd) entered disabled state
Feb 26 16:00:41 ip-172-19-3-15 docker: time=“2016-02-26T16:00:41.572901416-08:00” level=info msg=“DELETE /containers/4b189ef0da2bd3de1904fbd3e5afa1e65b4aa798e64b7611e5a2ccc58d99c75f?v=0&force=0”
Here is the “normal” container run :
Feb 26 06:56:08 ip-172-19-3-1 docker: time=“2016-02-26T06:56:08.035448189-08:00” level=info msg="POST /containers/7ef9c70e2f9fd8f64c2712c0ce19f19d518ff1ffc9505e4803ee37aeaf084060/start"
Feb 26 06:56:08 ip-172-19-3-1 kernel: bio: create slab at 2
Feb 26 06:56:08 ip-172-19-3-1 kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts:
Feb 26 06:56:08 ip-172-19-3-1 kernel: device veth0f34b5d entered promiscuous mode
Feb 26 06:56:08 ip-172-19-3-1 kernel: IPv6: ADDRCONF(NETDEV_UP): veth0f34b5d: link is not ready
Feb 26 06:56:08 ip-172-19-3-1 kernel: docker0: port 1(veth0f34b5d) entered forwarding state
Feb 26 06:56:08 ip-172-19-3-1 kernel: docker0: port 1(veth0f34b5d) entered forwarding state
Feb 26 06:56:08 ip-172-19-3-1 kernel: docker0: port 1(veth0f34b5d) entered disabled state
Feb 26 06:56:08 ip-172-19-3-1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0f34b5d: link becomes ready
Feb 26 06:56:08 ip-172-19-3-1 kernel: docker0: port 1(veth0f34b5d) entered forwarding state
Feb 26 06:56:08 ip-172-19-3-1 kernel: docker0: port 1(veth0f34b5d) entered forwarding state
Feb 26 06:56:08 ip-172-19-3-1 systemd: Starting docker container 7ef9c70e2f9fd8f64c2712c0ce19f19d518ff1ffc9505e4803ee37aeaf084060.
Feb 26 06:56:08 ip-172-19-3-1 systemd: Started docker container 7ef9c70e2f9fd8f64c2712c0ce19f19d518ff1ffc9505e4803ee37aeaf084060.
Feb 26 06:56:08 ip-172-19-3-1 docker: time=“2016-02-26T06:56:08.223917393-08:00” level=info msg="GET /containers/7ef9c70e2f9fd8f64c2712c0ce19f19d518ff1ffc9505e4803ee37aeaf084060/json"
Feb 26 06:56:23 ip-172-19-3-1 kernel: docker0: port 1(veth0f34b5d) entered forwarding state
Feb 26 07:01:01 ip-172-19-3-1 systemd: Created slice user-0.slice.
Feb 26 07:01:01 ip-172-19-3-1 systemd: Starting Session 15 of user root.
Feb 26 07:01:01 ip-172-19-3-1 systemd: Started Session 15 of user root.
Feb 26 07:06:46 ip-172-19-3-1 docker: time=“2016-02-26T07:06:46.889776644-08:00” level=info msg="POST /containers/7ef9c70e2f9fd8f64c2712c0ce19f19d518ff1ffc9505e4803ee37aeaf084060/stop?t=10"
Feb 26 07:06:46 ip-172-19-3-1 systemd: Stopping docker container 7ef9c70e2f9fd8f64c2712c0ce19f19d518ff1ffc9505e4803ee37aeaf084060.
Feb 26 07:06:46 ip-172-19-3-1 systemd: Stopped docker container 7ef9c70e2f9fd8f64c2712c0ce19f19d518ff1ffc9505e4803ee37aeaf084060.
Feb 26 07:06:46 ip-172-19-3-1 kernel: docker0: port 1(veth0f34b5d) entered disabled state
Feb 26 07:06:46 ip-172-19-3-1 kernel: docker0: port 1(veth0f34b5d) entered disabled state
Feb 26 07:06:46 ip-172-19-3-1 kernel: device veth0f34b5d left promiscuous mode
Feb 26 07:06:46 ip-172-19-3-1 kernel: docker0: port 1(veth0f34b5d) entered disabled state
Feb 26 07:06:46 ip-172-19-3-1 docker: time=“2016-02-26T07:06:46.976097638-08:00” level=info msg=“DELETE /containers/7ef9c70e2f9fd8f64c2712c0ce19f19d518ff1ffc9505e4803ee37aeaf084060?v=0&force=0”
==========================
Only diff i see between the one that freeze and one that dont is those three lines in bold.
Another thing worth mentioning is that we use docker to do our maven build , so on the docker host :
- container is spined up
- java process runs in this container with allot of network activity like pulling allot of files via http.
- this goes on for about 1 -2 min
- container is stopped and then deleted.
We do the above process on same host for about ~ 50 times a day.
any helps will be much appreciated.
-raj