Same container but different behaviour on different host

General Discussions General
1

My image contains a java application and is builded from this docker file:

FROM cogniteev/oracle-java:java8
MAINTAINER me@opencanarias.es
RUN mkdir /opt/siscloudrepositoryapy
WORKDIR /opt/siscloudrepositoryapy
COPY ./target/siscloudreposirotyapi*.jar /opt/siscloudrepositoryapy/
COPY ./startup.sh /opt/siscloudrepositoryapy/
RUN chmod +x /opt/siscloudrepositoryapy/startup.sh
RUN mkdir /opt/siscloudrepositoryapy/lib
COPY ./target/lib/*.jar /opt/siscloudrepositoryapy/lib/
EXPOSE 4567
CMD /opt/siscloudrepositoryapy/startup.sh

I export the image using “docker save tarfile.tar”. On the host I import the image using command “docker load -i tarfile.tar”.

I have no problems to run (docker run -it -p 4567:4567 IMAGEID bash) the container but when we start using it. On some hosts works perfectly but on others doesn’t.

To test the container I do (lynx http://localhost:4567/api/me@opencanarias.es/projects). It works on Host1, does not work on Host2.

This is all the info I have…

Host 1 - debian 8 - KVM - Virtual Machine

VERSION

Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.5.1
 Git commit:   f4bf5c7
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.5.1
 Git commit:   f4bf5c7
 Built:        
 OS/Arch:      linux/amd64

INFO

Containers: 2
Images: 18
Storage Driver: devicemapper
 Pool Name: docker-8:1-2-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.559 GB
 Data Space Total: 107.4 GB
 Data Space Available: 5.744 GB
 Metadata Space Used: 2.548 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.145 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.110 (2015-10-30)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.0-040200-generic
Operating System: Debian GNU/Linux stretch/sid (containerized)
CPUs: 1
Total Memory: 993.1 MiB
Name: debian8
ID: KTIV:MOFC:KS3G:SQNY:3BRW:SQFT:YX62:E33Z:FANE:2YM7:OMJ3:Q3VD

IMAGES

REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
<none>              <none>              906974ea0c0a        33 hours ago        655.1 MB

CONTAINERS

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
d208f42677c8        9069                "bash"              8 minutes ago       Up 8 minutes        0.0.0.0:4567->4567/tcp   thirsty_heisenberg

UNAME -A

Linux debian8 4.2.0-040200-generic #201508301530 SMP Sun Aug 30 19:31:40 UTC 2015 x86_64 GNU/Linux

Host 2 - debian 8 - GCE - Virtual Machine

VERSION

Client:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:10:10 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:10:10 UTC 2015
 OS/Arch:      linux/amd64

INFO

Containers: 12
Images: 40
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.0-0.bpo.1-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 1
Total Memory: 3.618 GiB
Name: nodo1
ID: VJ2B:3FLU:QQJR:W7KF:H2C7:3WQH:GQ2G:2FF2:7MC5:IFSB:Z3K6:SYGH

IMAGES

REPOSITORY                           TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
xxxxxx/javatest                     latest              906974ea0c0a        33 hours ago        655.1 MB

CONTAINERS

CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS                    NAMES
c363d08c9676        9069                                        "bash"                   24 hours ago        Up 24 hours         0.0.0.0:4567->4567/tcp   nostalgic_kowalevski

UNAME -A

Linux nodo1 4.2.0-0.bpo.1-amd64 #1 SMP Debian 4.2.3-2~bpo8+1 (2015-10-20) x86_64 GNU/Linux

When it doesn’t work this is the problem:

Selección_008.png798×539 21.8 KB

And this is the console output:

java.lang.NoSuchMethodError: javax.servlet.http.HttpServletResponse.getHeaders(Ljava/lang/String;)Ljava/util/Collection;
	at spark.utils.GzipUtils.checkAndWrap(GzipUtils.java:64)
	at spark.webserver.MatcherFilter.doFilter(MatcherFilter.java:251)
	at spark.webserver.JettyHandler.doHandle(JettyHandler.java:61)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
	at org.eclipse.jetty.server.Server.handle(Server.java:517)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:302)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:245)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
[qtp1186902720-14] WARN org.eclipse.jetty.server.HttpChannel - //pro-container01.opencanarias.com:4567/api/bcuesta@opencanarias.es/projects
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletResponse.getHeaders(Ljava/lang/String;)Ljava/util/Collection;
	at spark.utils.GzipUtils.checkAndWrap(GzipUtils.java:64)
	at spark.webserver.MatcherFilter.doFilter(MatcherFilter.java:251)
	at spark.webserver.JettyHandler.doHandle(JettyHandler.java:61)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
	at org.eclipse.jetty.server.Server.handle(Server.java:517)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:302)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:245)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
[qtp1186902720-15] INFO spark.webserver.MatcherFilter - The requested route [/favicon.ico] has not been mapped in Spark
[qtp1186902720-15] WARN org.eclipse.jetty.server.HttpChannel - //pro-container01.opencanarias.com:4567/favicon.ico
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletResponse.getHeaders(Ljava/lang/String;)Ljava/util/Collection;
	at spark.utils.GzipUtils.checkAndWrap(GzipUtils.java:64)
	at spark.webserver.MatcherFilter.doFilter(MatcherFilter.java:251)
	at spark.webserver.JettyHandler.doHandle(JettyHandler.java:61)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
	at org.eclipse.jetty.server.Server.handle(Server.java:517)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:302)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:245)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
[qtp1186902720-15] WARN org.eclipse.jetty.server.HttpChannel - //pro-container01.opencanarias.com:4567/favicon.ico
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletResponse.getHeaders(Ljava/lang/String;)Ljava/util/Collection;
	at spark.utils.GzipUtils.checkAndWrap(GzipUtils.java:64)
	at spark.webserver.MatcherFilter.doFilter(MatcherFilter.java:251)
	at spark.webserver.JettyHandler.doHandle(JettyHandler.java:61)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
	at org.eclipse.jetty.server.Server.handle(Server.java:517)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:302)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:245)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)

I have the same problem on a CentOS 6.7 virtual machine.

My developer team argue that the same container doesn’t work every where the same way!!!.

Thanks in advance.

2

Well not sure on the java exception (I could make a wild guess that tomcat did not start properly as servlet was probably not running) but as for Centos 6.7 I was looking at this just today myself. Need centos 7/kernel 3.10

from: https://docs.docker.com/engine/installation/centos/
CentOS

Docker runs on CentOS 7.X. An installation on other binary compatible EL7
distributions such as Scientific Linux might succeed, but Docker does not test
or support Docker on these distributions.

This page instructs you to install using Docker-managed release packages and
installation mechanisms. Using these packages ensures you get the latest release
of Docker. If you wish to install using CentOS-managed packages, consult your
CentOS documentation.

Prerequisites

Docker requires a 64-bit installation regardless of your CentOS version. Also,
your kernel must be 3.10 at minimum, which CentOS 7 runs.

3

Thanks for your reply.

As you can see on my post, I’ve tested the container on two Debian 8 Hosts, one on Google Compute Engine and the other one on a KVM virtual machine. Both hosts using 4.2 kernel. The container works on one host and not on the other one.

Also, on the picture you can see there’s a reply from server, so the servlet it’s running.

4

sgonzalez,

I’m following you why I said I was only really commenting on the Centos 6.7 issue.

But your right I missed the java.lang.NoSuchMethodError, thus had to have the servlet running. I have not done any serious Java development in a few years now I’m just starting to evaluate docker for a project. But this looks like maybe different java versions or classpath issue between the machines/VM/host running the container. Which if docker was totally self-contained should be NA (ie write once run anywhere like java). But I noticed something that has me confused with docker if that was truly the case. I started 2 containers on the same VM and I see 2 Java process running, one for each container. Well then it has to use the VM’s java to run it. Well each VM could have a different java version and a different classpath to find the libraries.

So I would go on each of the VMs and get the java version and the classpath and see if any difference.

from my VM:
ps -fe| grep java
root 3451 1690 39 09:13 pts/28 00:00:52 /usr/bin/java -Dnop -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dstringchararrayaccessor.disabled=true -server -Djavax.net.ssl.trustStore=/usr/local/owf/apache-tomcat/certs/keystore.jks -Djavax.net.ssl.keyStore=/usr/local/owf/apache-tomcat/certs/keystore.jks -Djavax.net.ssl.keyStorePassword=XXXXX -Djavax.net.ssl.trustStorePassword=XXXXX -Xmx1024m -Xms512m -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent -Djava.endorsed.dirs=/usr/local/owf/apache-tomcat/endorsed -classpath /usr/local/owf/apache-tomcat/bin/bootstrap.jar:/usr/local/owf/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/owf/apache-tomcat -Dcatalina.home=/usr/local/owf/apache-tomcat -Djava.io.tmpdir=/usr/local/owf/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
root 3539 1690 31 09:14 pts/29 00:00:33 /usr/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/local/tomcat/endorsed -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start

Also what does the Dockerfile look like? I’m assuming it starts off with something like this?
FROM java:8-jre

v/r,

Frank

5

You can see my dockerfile on my first post. I tried to build the container from two images:

FROM cogniteev/oracle-java:java8 → Here we have java inside the container

and

FROM ubuntu:wily
RUN apt-get install -y software-properties-common
RUN add-apt-repository -y ppa:webupd8team/java
RUN apt-get update
RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections

RUN apt-get install -y oracle-java8-installer

Both containers wroks the same way and both fail with class not found.

But this is very strange, as you said before, when I run the container I can a java process inside the container and a java process (pid 13450) on my vm.

The first console it’s inside the container, the second console it’s on my vm:

1654×1018 220 KB

I can’t figured out WHY there is a java process outside the container. If I have a mysql container I can’t see a mysqld process on my vm, just on the container.

I think this is a bug.

6

Looks like there are two javax.servlet.http.HttpServletResponse interfaces in the classpath of your app. This particular method was added in version 3 of servlet api (version 2 doesn’t have it). For some reason, classloader picks up the wrong interface on VM’s where you have failures. Try to start your java process with -verbose flag to see where (what jar) HttpServletResponse is loaded from and may be update the dependencies.

However, I would like to know why is the java process is visible outside of container as well…

7

Zeppelinux,
I actually posted this in another place and got a good response/makes sense.