How to setup a kubernetes cluster in under 15 mins using k3s? — Home Lab Part 4

I have been working lately on building my home-lab using old hardware on which I can run containers. Initially I had setup a cluster using microk8s which didn’t work out so well and I have shared my insights for the same here: https://vik-y.medium.com/what-i-learned-with-my-struggle-with-microk8s-home-lab-part-3-1d820f5e9420.

“Once Bitten, twice shy” — I took extreme caution while choosing next cluster setup tool. The options in front of me were k3s and kubeadm. Kubeadm is too complex to deal with, k3s is as simple as microk8s and very lightweight, only downside being that it can create single master cluster only which as of now isn’t a problem for me but might be in future. I will write a detailed review of k3s later in which I’ll cover more nuances.

What does my setup look like?

  1. Each node running ubuntu
  2. EC2 instance on AWS with a public IP
  3. VPN Server running on EC2 instance and all nodes at home connected to it.

More details of my overall infra available here: https://vik-y.medium.com/how-to-connect-your-home-systems-to-a-vpn-on-aws-in-under-30-mins-home-lab-part-2-6ef64802ccba

What will my cluster look like?

  1. Http and https traffic coming to public IP of EC2 instance will be redirected to kubernetes cluster running at home — I will use iptables for that. This EC2 instance will be be part of kubernetes cluster (This will be covered in a separate blog)

Let’s build the cluster

vikas-laptop-1 -> 192.168.1.42
vikas-laptop-2 -> 192.168.1.41
home-pc -> 192.168.1.33

Setup control-plane/master/server

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --node-ip=192.168.1.42" sh -

I didn’t require more configuration during installation, if you need more you can refer to documentation https://rancher.com/docs/k3s/latest/en/installation/install-options/

The above command will bootstrap everything required to bring up your master node. And that’s about it, your control-plane is ready.

vikas@vikas-laptop-1:~$ sudo k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
vikas-laptop-1 Ready control-plane,master 1m v1.21.1+k3s1

Explore control-plane more

  1. Kubeconfig is located at /etc/rancher/k3s/k3s.yaml. You can do export KUBECONFIG=/etc/rancher/k3s/k3s.yaml and then start using kubectl
  2. There’s a token stored at /var/lib/rancher/k3s/server/node-token which will be used later on to add worker nodes.
  3. If you do not have kubectl installed you can still use it using command: sudo k3s kubectl.
  4. Even with no services running yet I could see 20% CPU utilisation on one core and 8% memory utilisation (my RAM size on master node is 8GB). These numbers look as per the expected profiling numbers https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/resource-profiling/

Add worker node /agent

MASTER_NODE_IP=192.168.1.42curl -sfL https://get.k3s.io | K3S_URL=https://$MASTER_NODE_IP:6443 K3S_TOKEN=token_obtained_above sh -

Now, wait for the script to run completely — it will take a maximum of 1 minute.

sudo k3s kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP
vikas-laptop-1 Ready control-plane,master 10m v1.21.1+k3s1 192.168.1.42
home-pc Ready <none> 1m v1.21.1+k3s1 192.168.1.33

That’s it your cluster is ready with 1 master and 2 worker nodes.

Verdict

  1. I didn’t see any networking related errors while creating any pods.
  2. When the master goes down the pods on other nodes still keep running without any issues just that you can’t schedule new workloads. Things becoming fully functional once the master node boots up and automatically connects to the cluster without any manual intervention.
  3. Just like microk8s, k3s also comes up with a lot of things like ingress, storage driver, etc bundled which make it extremely simple to use — I’ll probably have to write another article ont his one :D

Upcoming Blogs in this series

  1. Setup ingress to route http/https traffic inside our cluster
  2. Some iptables magic to bring in public internet traffic into our cluster.
  3. Setup cluster monitoring using prometheus + grafana

I have set these up already on my cluster and will write separate blog about these. You can also read more about my home-lab plans here https://vik-y.medium.com/building-my-home-lab-with-docker-swarm-part-1-13cceb3c4f1f.

SRE at Linkedin