What I learned from my struggle with microk8s? — Home Lab Part 3

So I started off on this “joyful” road after looking at https://microk8s.io/. The website looks so clean and the clear set of instructions that they give you makes it look extremely promising and then the add-ons are a cherry on top — Well not really.

After dumping docker-swarm for my home-lab and deciding to use Kubernetes the first thing I wanted to try out was microk8s. What I did not realise was that it will turn out to be an awful experience which will take me days to figure out things and eventually dump it.

DISCLAIMER: I was trying out 1.21/stable channel of microk8s. From what I have heard from people — 1.18, 1.19 are much more stable and should be preferred.

When does microk8s work well?

1. It will work well if you understand the internals of kubernetes setup and can quickly juggle to find out small issues here and there and fix them but as a newbie — don’t even try that.
2. Microk8s by default runs a highly available cluster which can tolerate multiple failures — which is amazing but increases the complexity way too much. Consider using it above other easier and lighter alternatives only if you are looking for highly available cluster.
3. You have fresh nodes which have nothing pre-installed, no complicated networking rules , no VPN — go ahead, microk8s might just work fine.

Problems I faced with microk8s

1. It’s too heavy: It promises to be feather light — but it isn’t. While running it on a t3.micro instance with scheduling disabled, there were multiple occasions when the entire box froze and I had to force restart the node to get things back to normal — obviously due to memory related issues. I understand who in their sane mind would run k8s node on a 1GB memory instance but well I did because they advertised it is “featherlight”.
2. Adding new nodes to a cluster is not so smooth: In documentation you can find the commands look very straightforward. If all goes well they’ll work but otherwise you can face unexpected failures like — 1) the node never gets added and you don’t get any error message, 2) node gets added but scheduling pods on it fails, 3) node gets added but never gets into ready state.
3. Default ha-cluster has multiple CNI related bugs: Pods don’t startup at times. Especially when the cluster is newly initialised or a new node is added to cluster. If all goes well for you and you manage to get all the nodes connected in a cluster — try running some pods, for me they failed with weird “cni” related errors. Something similar to this: https://github.com/ubuntu/microk8s/issues/2042. Then I had to spend a lot of time finding the root cause which defeats the whole purpose.
4. Node fails to join cluster after leaving it: Cluster was running happily for 3 days until one node went down due to power issues. When it came back up it did not connect automatically to the cluster. It was throwing up some weird errors in logs — again related to CNI. A restart and reinstallation microk8s on that node fixed the issue.

Before I started setting up microk8s cluster I made sure reading through the documentation to avoid any unexpected surprises. I meticulously went through the logs when I faced problems, did a lot of searches, found solutions, contributed to answers — and after all of it I gave up. I don’t want to work on something where every other day I have fix one thing or another.

Verdict:

1. Microk8s is NOT featherlight.
2. If you are trying to look for lighter replacement of docker-swarm then microk8s is not for you.
3. If you want to learn about setting up kubernetes from ground up without spending a lot of time — microk8s might help you speed up that process. You can dig into the code and see things working instead of doing it ground up.

So what do I do now? I heard a lot of noise around about k3s -> https://k3s.io/ and I am currently trying it out. Will write an article on how things go. This article is a part of my home lab series and if you liked this then you might like other articles in this series: ihttps://vik-y.medium.com/building-my-home-lab-with-docker-swarm-part-1-13cceb3c4f1f