Web server does not respond in spite of the fact all K8S indicators are OK

Docker Desktop
1

Hi, I have K8S with docker desktop, I apply a deployment as well as a service and forward port 80 to port 8080. Issuing the following commands show that everything is OK
1 - get pods

NAME READY STATUS RESTARTS AGE
stable-deployment-7cc45c64fb-5vgp6 1/1 Running 0 31m
stable-deployment-7cc45c64fb-6x8m4 1/1 Running 0 31m
stable-deployment-7cc45c64fb-cdwk4 1/1 Running 0 31m
stable-deployment-7cc45c64fb-vtmt6 1/1 Running 0 31m

2 - get services

image
image769×62 16.8 KB

3 - describe service

Name: stable-service
Namespace: default
Labels: app=aspnetcore
Annotations:
Selector: app=aspnetcore
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.108.92.58
IPs: 10.108.92.58
LoadBalancer Ingress: localhost
Port: 80/TCP
TargetPort: 8080/TCP
NodePort: 31951/TCP
Endpoints: 10.1.0.70:8080,10.1.0.71:8080,10.1.0.72:8080 + 1 more…
Session Affinity: None
External Traffic Policy: Cluster
Events:

When I browse the application at http:localhost:8080 it does not respond. The question is : How can I troubleshoot/debug why the web server is not responding? Knowing that when I run the same image used in the deployment using docker run, website responds correctly on port 8080
Thanks for your help

Finally, if your issue has not been addressed elsewhere, running:

  • :whale: :arrow_right: Settings :arrow_right: Diagnose & Feedback :arrow_right: Open Issues.

This forum is not an official product support or issue reporting channel.

2

Localhost is not the localhost fof your host but the virtual machine of Docker Desktop.

You probably forward ports using docker run and the Docker client on the host will make sure the host port is forwarded to the virtual machine and the Docker daemon can forward it from the VM to the container. If you want to do a similar port forwarding that you have with Docker, you need NodePort for Type and not LoadBalancer. I don’t think LoadBalancer makes sense when using Docker Desktop as it doesn’t have many IP addresses to choose from and localhost is not the same in the VM and on the host.

Update:

I guess I was wrong: Access a Kubernetes Service running locally in Docker For Desktop? - Stack Overflow

I have to reconsider my answer :slight_smile: I don’t use Kubernetes with Docker Desktop and in fact NodePort would require accessing the virtual machine’s IP address which is not available from the host as far as I know.

3

I answered too quickly in my previous post so here is what I think now. The NodePort in your service description and service list is 31951 and not 8080, so can you access localhost:31951?

4

Hi, with the other port, same thing no access

5

But how come it is not the 8080, the service.ytaml is as follows:L

kind: Service
apiVersion: v1
metadata:
  name: stable-service
  labels:
    app: aspnetcore
spec:
  type: LoadBalancer
  selector:
    app: aspnetcore
  ports:
    - port: 80
      targetPort: 8080

Does this mean that Kubernetes cointroller does not respect what is indicated in the service spec?

6

I tried it on macOS and here is the result.

My previous answers based on using Kubernetes in Rancher Desktop where I could choose a specific version, and I wasn’t sure if it works the same in Docker Desktop, but I suspected. This is why my first thought was that you need to use NodePort, but I didn’t remember why, because the NodePort is the port that is forwarded from the Kubernetes node (Docker Desktop’s virtual machine) to the container and the IP of the virtual machine is not available from the outside. When I tried it in Rancher Desktop I didn’t even think about why NodePort should not work by default, since it worked and it was enough for me.

So I went to try it in Docker Desktop and realized that you have multiple options. If it were a usual Kubernetes cluster you would be able to access the node IP, but Rancher Desktop and Docker Desktop can detect which type of service you are using and forward ports based on that.

Pod:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
  labels:
    app: demo
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - name: web
          containerPort: 80
          protocol: TCP

If you are using type: ClusterIP which is the default, you can access the port defined in “port” on the cluster ip.

apiVersion: v1
kind: Service
metadata:
  name: nginx-clusterip
  namespace: default
spec:
  ports:
    - protocol: TCP
      port: 8081
      targetPort: web
  selector:
    app: demo
  type: ClusterIP

Note that I used “web” as target port which is defined in the pod. It is not the port that you can access from the host. It is the port on which the process is listening in the container. Since it is a cluster ip service, nothing is forwarded, you can only access this port from another pod in the virtual machine.

You can use NodePort like this:

apiVersion: v1
kind: Service
metadata:
  name: nginx-nodeport
  namespace: default
spec:
  ports:
    - protocol: TCP
      port: 8081
      targetPort: web
      nodePort: 30000
  selector:
    app: demo
  type: NodePort

Again, port 8081 is available on the cluster ip and it is forwarded to the container’s “web” port, but you also have a nodePort which will be detected by Docker Desktop and it will be forwarded from your host to the container through your virtual machine and service to the container’s port (targetPort)

NodePort can’t be lower than 30000 by default.

The last option is using LoadBalancer.

apiVersion: v1
kind: Service
metadata:
  name: nginx-lb
  namespace: default
spec:
  ports:
    - protocol: TCP
      port: 8081
      targetPort: web
  selector:
    app: demo
  type: LoadBalancer

Now “port” will become a load balancer port and this will be forwarded from the host to the container. It will create a nodePort as well, but that is available only from the virtual machine since the type is not “NodePort” in this case.

7

Thnaks for your efforts and help, before starting sing your instructions one note : You use
kind: Pod but I deploy my Pods through deployment file as follows (is it the same thing?)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: stable-deployment
spec:
  replicas: 4
  selector:
    matchLabels:
      app: aspnetcore
      track: stable
  template:
    metadata:
      labels:
        app: aspnetcore
        track: stable
    spec:
      containers:
      - name: stable-app
        image: mcr.microsoft.com/dotnet/samples:aspnetapp
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
        resources: {}
8

I used a Pod to have the simplest example focusing on the service types which was related to Docker Desktop. You can still use any kind like Deployment or StatefuSet and use any selector, but yes, at first sight your deployment looks right.

9

After sharing the manifest of the deployment, the problem is clear.
The targetPort of the service manifest must match the containerPort of the deployment manifest.
The port of the service manifest is the “external” port of the service - it sends all incoming traffic to the targetPort (of the pod).

Your service will be reachable on http://localhost:8080, if your service manifest looks like this:

kind: Service
apiVersion: v1
metadata:
  name: stable-service
  labels:
    app: aspnetcore
spec:
  type: LoadBalancer
  selector:
    app: aspnetcore
  ports:
    - port: 8080
      targetPort: 80
10

I didn’t catch that :slight_smile:

11

Great meyay, you saved me me, so many thanks it works like a charm now

@rimelek, I tried to apply your single pod manifest but I get an eror as follows

Error: must specify one of -f and -k

error: unknown command “–f singlepod.yml”
See ‘kubectl create -h’ for help and examples

so it seems syntax is not correct

12

Finally, you got it going :slight_smile:

Single quotes, double quotes and dashes can be prone to encoding problems, that looks like the same character in the terminal, but actually is not. Just replace

The error message indicates that – (I copied this character from your error message) was used instead of - (this one is typed). It will work if your remove the false dash and replace it with a typed dash character.

13

Thanks again meyay, exact, I typed the - and it worked. used the example of the service using type NodePort as follows

apiVersion: v1
kind: Service
metadata:
  name: nginx-clusterip
  namespace: default
spec:
  ports:
    - protocol: TCP
      port: 8081
      targetPort: web
  selector:
    app: demo
  type: NodePort

get services gives

image
image771×65 18.7 KB

still cant access the webpage on port 8081 but I was able to browse http://localhost:32292/

14

Did you use the manifests that @rimelek provided?

You mixed the clusterip and nodeport example, which should be no problem as long as you made sure that the selector in the service manifest matches the label of the pod manifest, and that the targetPort name actually is defined in the port definition of the pod manifest.

Update: please ignore my post, I missed that it works with the node port 32292. I blurred the original content.

15

Since your service type is NodePort, that is normal and expected.

But I can see that my description about this example was misunderstandable. When I stated that port 8081 would be available on the cluster ip and forwarded to the container’s port, I meant only that if you could access the cluster IP inside the virtual machine, you could use the cluster IP, but only NodePort is forwarded from the Windows host to the container.

I didn’t want to confuse you yesterday with this example, but now it looks relevant, so I share how you could access the cluster IP in the VM.

service_ip="10.101.28.89"
service_port="8081"

docker run --rm -it --privileged --pid host ubuntu:20.04 \
  nsenter --all -t 1 \
    -- ctr -n services.linuxkit task exec -t --exec-id test docker \
         bash -c "exec 5<>/dev/tcp/$service_ip/$service_port && echo -e \"GET / HTTP/1.0\n\" >&5 && cat <&5"

Of course you need to make sure that the cluster ip and the port is correct ath the beginning of the code.

This is only for testing. I had to use a speciel way to get the output from the web server because there is no curl or wget in the container in which the Docker daemon is running, but this is where you can access the cluster IP and not directly from the virtual machine outside that container.

I runs a privileged Docker container, breaks out of it using nsenter and runs that speciel bash command inside the container of the Docker daemon.

I guess you can see now how different a usual Kubernetes host and Docker Desktop are. to give you (or try) the same experience on each platform :slight_smile: