Minikube fails on Ubuntu when also having Docker Desktop installed

General
1

Thank you Rimelek,

I discovered that, in my system, Ubuntu 22,04 AMD64, I cannot have both Minikube and Docker-Desktop. I am unable to use Minikune with Docker if Doker-Desktop is installed. Uninstalling Docker-Desktop resolves the issue while installing it back, brings the same issues again.

I have uninstalled Docker-Desktop on my machine because I saw some else reporting the same issue.

Should I file a bug report on this issue with more details? I don’t know whether the problem is Docker-Desktop or Minikube, but they are not working well together on my Ubuntu machine. I have another machine with macOS and I don’t experience this issue.

2
  • Are you sure you need to uninstall Docker desktop and stopping it is not enough?
  • What driver do you use with minikube? It supports for example KVM, Qemu or even Docker. I could imagine that when Docker Desktop is running (or being installed may be enough), minikube detects it and tries to use it instead of KVM which I think the default, but I don’t remember. I can’t tell why it would be a problem, but the preferred drivers for Minikube are different on macOS, so it could explain why it works on mac for you.

We can try to figure it out first. If you can make it work for example with specifiying the Minikube driver and if that is acceptable for you, then you don’t need to file a bug. At least not to Docker Desktop.

I try to find time later to install Docker Desktop for Linux again and run Minikube too.

3

I use Minikube 1.26.0 on both machine. I use minikube start --driver=docker on both machines to ensure the Docker driver is used. Everything works fine if Docker-Desktop is not installed on my Ubuntu machine.

4

I tried minikube on Linux with the Docker driver. coredns pods, kube-proxy and storage-provisione could not start.

minikube kubectl -- get node

did not work. So I ran a shell in the minikube container.

docker exec -it minikube bash

there was no kubectl so I downloaded it.

curl -LO https://dl.k8s.io/release/v1.20.2/bin/linux/amd64/kubectl
chmod +x kubectl

and set the config

export KUBECONFIG=/etc/kubernetes/admin.conf

and get the nodes and pods

./kubectl get node
./kubectl get pod --all-namespaces

Then I checked the logs of kube-proxy:

root@minikube:/# ./kubectl logs -n kube-system pod/kube-proxy-7s8mk

I0710 00:23:50.102716       1 node.go:172] Successfully retrieved node IP: 192.168.49.2
I0710 00:23:50.102782       1 server_others.go:142] kube-proxy node IP is an IPv4 address (192.168.49.2), assume IPv4 operation
W0710 00:23:50.117588       1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0710 00:23:50.117681       1 server_others.go:185] Using iptables Proxier.
I0710 00:23:50.118362       1 server.go:650] Version: v1.20.2
I0710 00:23:50.119316       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
F0710 00:23:50.119343       1 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied

Even if you try to change nf_conntrack_max will not work:

sysctl -w net.netfilter.nf_conntrack_max=131072

I also inspected the minikube container and I saw it was a privileged container, but it was not enough, so I entered the virtual machine of Docker Desktop:

docker run -it --rm --privileged --pid=host justincormack/nsenter1

and changed nf_conntrack_max here:

sysctl -w net.netfilter.nf_conntrack_max=131072

Then I exited end went back to minikube

docker exec -it minikube bash

and deleted the kube-proxy

export KUBECONFIG=/etc/kubernetes/admin.conf
./kubectl delete -n kube-system pod/kube-proxy-7s8mk

Then the daemonset started it again successfully. I waited some minutes and the coredns pods were running too, but the storage provisioner pod stopped again. I waited again and everything was running.

I tried

minikube kubectl -- get node

but it still didn’t work. So I exported the config of kubernetes using the kubectl subcommand of minikube:

minikube kubectl -- config view --flatten > minikube.conf

and tried with this config from the host where my kubectl was a little newer. 1.21 but that is still compatible with 1.20 so I didn’t download an other.

export KUBECONFIG=$PWD/minikube.conf
kubectl get node

There was no answer, but then I ealized I had multiple clusters in the exported config since the kubectl was a local kubectl so I just copied the one inside the minikube container to the host:

docker cp minikube:/etc/kubernetes/admin.conf minikube-admin.conf

and tried with that config

export KUBECONFIG=$PWD/minikube-admin.conf
kubectl get node

It worked so I stopped investigating. IThe issue with nf_conntrack_max looks like something that could be fixed in Docker Desktop, but the access issue from the host could be just wrong configuration as it was for me.

minikube kubectl still doesn’t work, because it wants to access Docker Desktop’s private IP address (“192.168.49.2” in my case) which is not available from the host and I am pretty sure it would be the case on Mac too. I was a little surprised however, because I though minikube kubectl would run kubectl inside minikube, but it looks like it doesn’t, because I can access that IP addres when I am inside minikube.

curl https://192.168.49.2:8443

If you try what I described here and you also see that kube-proxy doesn’t work and you can fix it as I did, you can report this issue here

The issue with minikube kubectl might be something that could be fixed in minikube for example by recognizing that the Docker driver is connected to Docker Desktop and not the Docker Engine on Linux, but if it works for you on Mac, then it is possible that this is a Docker Desktop bug too. I just don’t know how it could be fixed.

And of course you need to make sure that your configurations are correct. At least until any of these bugs are fixed, you can probably use my workaround and fix the VM manuall, then use a local kubectl to connect to Kubernetes.

Or you can just switch to the KVM driver which is the default on Linux and works perfectly without any timeout. Note that Docker Desktop for Linux is still Beta.

2 Likes