How to setup a kubernetes cluster in under 15 mins using k3s? — Home Lab Part 4
I have been working lately on building my home-lab using old hardware on which I can run containers. Initially I had setup a cluster using microk8s which didn’t work out so well and I have shared my insights for the same here: https://vik-y.medium.com/what-i-learned-with-my-struggle-with-microk8s-home-lab-part-3-1d820f5e9420.
“Once Bitten, twice shy” — I took extreme caution while choosing next cluster setup tool. The options in front of me were k3s and kubeadm. Kubeadm is too complex to deal with, k3s is as simple as microk8s and very lightweight, only downside being that it can create single master cluster only which as of now isn’t a problem for me but might be in future. I will write a detailed review of k3s later in which I’ll cover more nuances.
What does my setup look like?
- 2 laptops and 1 desktop at my home
- Each node running ubuntu
- EC2 instance on AWS with a public IP
- VPN Server running on EC2 instance and all nodes at home connected to it.
More details of my overall infra available here: https://vik-y.medium.com/how-to-connect-your-home-systems-to-a-vpn-on-aws-in-under-30-mins-home-lab-part-2-6ef64802ccba
What will my cluster look like?
- Cluster will contain all the 3 nodes at home with master running on one and the other 2 nodes as workers.
- Http and https traffic coming to public IP of EC2 instance will be redirected to kubernetes cluster running at home — I will use iptables for that. This EC2 instance will be be part of kubernetes cluster (This will be covered in a separate blog)
Let’s build the cluster
Here are the nodes I have at my home with their internal IP. I am going to run the control-plane/master on “vikas-laptop-1” and the remaining nodes will be workers. Master nodes will also be able to run workloads just like other worker nodes.
vikas-laptop-1 -> 192.168.1.42
vikas-laptop-2 -> 192.168.1.41
home-pc -> 192.168.1.33
On vikas-laptop-1 run the command below. I have to explicitly pass
--node-ip because I do not want k3s to bind my server to any other interface — especially the VPN interface to avoid unnecessary latency.
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --node-ip=192.168.1.42" sh -
I didn’t require more configuration during installation, if you need more you can refer to documentation https://rancher.com/docs/k3s/latest/en/installation/install-options/
The above command will bootstrap everything required to bring up your master node. And that’s about it, your control-plane is ready.
vikas@vikas-laptop-1:~$ sudo k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
vikas-laptop-1 Ready control-plane,master 1m v1.21.1+k3s1
Explore control-plane more
- The installation script creates a systemd service
k3sYou can manage it using your usual systemctl commands.
- Kubeconfig is located at
/etc/rancher/k3s/k3s.yaml. You can do
export KUBECONFIG=/etc/rancher/k3s/k3s.yamland then start using
- There’s a token stored at
/var/lib/rancher/k3s/server/node-tokenwhich will be used later on to add worker nodes.
- If you do not have
kubectlinstalled you can still use it using command:
sudo k3s kubectl.
- Even with no services running yet I could see 20% CPU utilisation on one core and 8% memory utilisation (my RAM size on master node is 8GB). These numbers look as per the expected profiling numbers https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/resource-profiling/
Add worker node /agent
Now that we are done setting up the master and understanding the internals a bit, let’s quickly go ahead and attach a worker node to master. First of all get the
token required for initiating the connection. You can get the token from file
/var/lib/rancher/k3s/server/node-token in master node, in my case “vikas-laptop-1”.
MASTER_NODE_IP=192.168.1.42curl -sfL https://get.k3s.io | K3S_URL=https://$MASTER_NODE_IP:6443 K3S_TOKEN=token_obtained_above sh -
Now, wait for the script to run completely — it will take a maximum of 1 minute.
sudo k3s kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP
vikas-laptop-1 Ready control-plane,master 10m v1.21.1+k3s1 192.168.1.42
home-pc Ready <none> 1m v1.21.1+k3s1 192.168.1.33
That’s it your cluster is ready with 1 master and 2 worker nodes.
- I have been using this for over 1 week now. The cluster has performed flawlessly.
- I didn’t see any networking related errors while creating any pods.
- When the master goes down the pods on other nodes still keep running without any issues just that you can’t schedule new workloads. Things becoming fully functional once the master node boots up and automatically connects to the cluster without any manual intervention.
- Just like microk8s, k3s also comes up with a lot of things like ingress, storage driver, etc bundled which make it extremely simple to use — I’ll probably have to write another article ont his one :D
Upcoming Blogs in this series
The cluster is up though we need to still do a lot of things to make this cluster more useful for us:
- Setup ingress to route http/https traffic inside our cluster
- Some iptables magic to bring in public internet traffic into our cluster.
- Setup cluster monitoring using prometheus + grafana
I have set these up already on my cluster and will write separate blog about these. You can also read more about my home-lab plans here https://vik-y.medium.com/building-my-home-lab-with-docker-swarm-part-1-13cceb3c4f1f.