How to setup a kubernetes cluster in under 15 mins using k3s? — Home Lab Part 4

I have been working lately on building my home-lab using old hardware on which I can run containers. Initially I had setup a cluster using microk8s which didn’t work out so well and I have shared my insights for the same here: https://vik-y.medium.com/what-i-learned-with-my-struggle-with-microk8s-home-lab-part-3-1d820f5e9420.

“Once Bitten, twice shy” — I took extreme caution while choosing next cluster setup tool. The options in front of me were k3s and kubeadm. Kubeadm is too complex to deal with, k3s is as simple as microk8s and very lightweight, only downside being that it can create single master cluster only which as of now isn’t a problem for me but might be in future. I will write a detailed review of k3s later in which I’ll cover more nuances.

What does my setup look like?

1. 2 laptops and 1 desktop at my home
2. Each node running ubuntu
3. EC2 instance on AWS with a public IP
4. VPN Server running on EC2 instance and all nodes at home connected to it.

More details of my overall infra available here: https://vik-y.medium.com/how-to-connect-your-home-systems-to-a-vpn-on-aws-in-under-30-mins-home-lab-part-2-6ef64802ccba

What will my cluster look like?

1. Cluster will contain all the 3 nodes at home with master running on one and the other 2 nodes as workers.
2. Http and https traffic coming to public IP of EC2 instance will be redirected to kubernetes cluster running at home — I will use iptables for that. This EC2 instance will be be part of kubernetes cluster (This will be covered in a separate blog)

Let’s build the cluster

Here are the nodes I have at my home with their internal IP. I am going to run the control-plane/master on “vikas-laptop-1” and the remaining nodes will be workers. Master nodes will also be able to run workloads just like other worker nodes.

vikas-laptop-1 -> 192.168.1.42
vikas-laptop-2 -> 192.168.1.41
home-pc -> 192.168.1.33

Setup control-plane/master/server

On vikas-laptop-1 run the command below. I have to explicitly pass --node-ip because I do not want k3s to bind my server to any other interface — especially the VPN interface to avoid unnecessary latency.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --node-ip=192.168.1.42" sh -

I didn’t require more configuration during installation, if you need more you can refer to documentation https://rancher.com/docs/k3s/latest/en/installation/install-options/

The above command will bootstrap everything required to bring up your master node. And that’s about it, your control-plane is ready.

vikas@vikas-laptop-1:~$ sudo k3s kubectl get nodes
NAME             STATUS   ROLES                  AGE   VERSION
vikas-laptop-1   Ready    control-plane,master   1m    v1.21.1+k3s1

Explore control-plane more

1. The installation script creates a systemd service k3s You can manage it using your usual systemctl commands.
2. Kubeconfig is located at /etc/rancher/k3s/k3s.yaml. You can do export KUBECONFIG=/etc/rancher/k3s/k3s.yaml and then start using kubectl
3. There’s a token stored at /var/lib/rancher/k3s/server/node-token which will be used later on to add worker nodes.
4. If you do not have kubectl installed you can still use it using command: sudo k3s kubectl.
5. Even with no services running yet I could see 20% CPU utilisation on one core and 8% memory utilisation (my RAM size on master node is 8GB). These numbers look as per the expected profiling numbers https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/resource-profiling/

Add worker node /agent

Now that we are done setting up the master and understanding the internals a bit, let’s quickly go ahead and attach a worker node to master. First of all get the token required for initiating the connection. You can get the token from file/var/lib/rancher/k3s/server/node-token in master node, in my case “vikas-laptop-1”.

MASTER_NODE_IP=192.168.1.42
curl -sfL https://get.k3s.io | K3S_URL=https://$MASTER_NODE_IP:6443 K3S_TOKEN=token_obtained_above sh -

Now, wait for the script to run completely — it will take a maximum of 1 minute.

sudo k3s kubectl get nodes -o wide
NAME             STATUS   ROLES                  AGE   VERSION        INTERNAL-IP
vikas-laptop-1   Ready    control-plane,master   10m    v1.21.1+k3s1   192.168.1.42   
home-pc          Ready    <none>                 1m    v1.21.1+k3s1   192.168.1.33   

That’s it your cluster is ready with 1 master and 2 worker nodes.

Verdict

1. I have been using this for over 1 week now. The cluster has performed flawlessly.
2. I didn’t see any networking related errors while creating any pods.
3. When the master goes down the pods on other nodes still keep running without any issues just that you can’t schedule new workloads. Things becoming fully functional once the master node boots up and automatically connects to the cluster without any manual intervention.
4. Just like microk8s, k3s also comes up with a lot of things like ingress, storage driver, etc bundled which make it extremely simple to use — I’ll probably have to write another article ont his one :D

Upcoming Blogs in this series

The cluster is up though we need to still do a lot of things to make this cluster more useful for us:

1. Setup ingress to route http/https traffic inside our cluster
2. Some iptables magic to bring in public internet traffic into our cluster.
3. Setup cluster monitoring using prometheus + grafana

I have set these up already on my cluster and will write separate blog about these. You can also read more about my home-lab plans here https://vik-y.medium.com/building-my-home-lab-with-docker-swarm-part-1-13cceb3c4f1f.