I have been curious for a while if I could build a powerful Kubernetes cluster that is compact, cheap and easy to use. Then one day I came across BitScope and their blade racks, and I knew I had to get one!
Yepp, that is a 20 node Kubernetes cluster of Raspberry PI 4 model B 4GB.
The Build
I got the 20 unit blade rack which includes two Duo Pi Blade Packs and 10 Duo Pis. It is delivered as a kit that you need to assemble yourself.
Time for assembly!
You also need a couple of Raspberry PIs!
That’s a lot of computers
To handle networking I use a 24 port Netgear GS324T gigabit switch with 20 0.3m flat U/UTP Cat6 RJ45 ethernet cables. I use 32GB SanDisk Ultra micro-SDHC (Class 10 / UHS-I) cards as storage for each PI, which is enough to handle the DDR50 transfer mode supported by the PI. Since the Raspberry PI does not have any cooling and is installed in a quite cramped area, I bought some passive heatsinks and jerry-rigged two silent 200mm case fans from Noctua in order to prevent throttling during load. The blade rack is delivered with four 30mm fans that are attached to the back, but these are very noisy. To power it all I use a 12V20A power supply.
The cost? Around €2000 excl. tax. Not that bad, considering this a full-blown cluster with a total of 80 cpu cores @ 1.5GHz and 80GB DDR4-3200 SDRAM.
There she blows!
Configuration
I’m not going to go into details how to setup the PI’s and installing Kubernetes on them, it is already well explained here, but I will give an overview of the setup. The PIs are running Raspbian Lite, the official supported operating system from the Raspberry PI Foundation. The setup is quite easy with flashing the image and setting up some basic configuration. As Kubernetes cluster, I use k3s from Rancher Labs, a lightweight version of Kubernetes supporting the ARMv7 processor architecture supported by the PI’s Cortex-CPU. It does not come with Docker bundled but the smaller containerd, which is a graduated project under the Cloud Native Computing Foundation, CNCF.
Fun fact by the way; k8s is a numeronym for Kubernetes, but what k3s means is more of a mystery. I’m gonna go with Kubes!
Lessions Learned
During the installation of Kubes, I stumbled upon a couple of issues that I want to share.
Explict Version
Running curl -sfL https://get.k3s.io | sh - will download a boostrap script of k3s which targets the latest version of the k3s image. If the installation for the whole cluster takes a couple of days, you might like me end up with a new latest version released causing incompatibility. I ended up getting the error “tls: failed to find any PEM data in key input” when trying to join one newly installed agent because of a change in how Go decodes key files which was introduced in a later version.
Always target an explicit version using INSTALL_K3S_VERSION=<version>, ie curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=<version> sh -.
iptables
k3s is using iptables-legacy but Rasbian/Buster is using iptables 1.8.2 nft which sometimes seems to cause problems when adding the agent to the cluster. To solve it, set iptables to legacy by executing sudo update-alternatives --set iptables /usr/sbin/iptables-legacy.
VPN
Trying to resolve local hostnames on the local network in Ubuntu (WSL) when connected to an external VPN service can be a hassle!
Missing Files During Installation
Sometimes the k3s installation process fails storing all files on the SD card in the PI causing the k3s agent to fail with errors like:
FATA[0000] exec: "k3s-agent": executable file not found in $PATH
The resolution is to remove /var/lib/rancher/k3s/data and re-launch the configuration process.
Shutting Down the Cluster
Just pulling the plug from a Raspberry PI is never a good idea, but shutting down a whole cluster of them one by one is tedious. I use the following script to shut them all down.
hostnames=(<server>)
for ((i=1;i<=<number-of-clients>;i++)); do
hostnames+=("<client>-$i")
done
for hostname in ${hostnames[@]}
do
ssh pi@$hostname 'sudo poweroff'
done
Exchange <server> with the server’s hostname, <client> with the starting name of the clients hostname and <number-of-clients> with the actual number of clients.
Conclusions
Running Kubernetes using k3s on a couple of Raspberry PI works quite good, without having battle tested the setup yet. Heck, I don’t even know what I will use it for! But it’s pretty, right? 🙂
Leave a Comment