2019 – coding like a boss

Raspberry PI <3 Kubernetes

Published by Fredrik on December 12, 2019

I have been curious for a while if I could build a powerful Kubernetes cluster that is compact, cheap and easy to use. Then one day I came across BitScope and their blade racks, and I knew I had to get one!

Yepp, that is a 20 node Kubernetes cluster of Raspberry PI 4 model B 4GB.

The Build

I got the 20 unit blade rack which includes two Duo Pi Blade Packs and 10 Duo Pis. It is delivered as a kit that you need to assemble yourself.

Time for assembly!

You also need a couple of Raspberry PIs!

That’s a lot of computers

To handle networking I use a 24 port Netgear GS324T gigabit switch with 20 0.3m flat U/UTP Cat6 RJ45 ethernet cables. I use 32GB SanDisk Ultra micro-SDHC (Class 10 / UHS-I) cards as storage for each PI, which is enough to handle the DDR50 transfer mode supported by the PI. Since the Raspberry PI does not have any cooling and is installed in a quite cramped area, I bought some passive heatsinks and jerry-rigged two silent 200mm case fans from Noctua in order to prevent throttling during load. The blade rack is delivered with four 30mm fans that are attached to the back, but these are very noisy. To power it all I use a 12V20A power supply.

The cost? Around €2000 excl. tax. Not that bad, considering this a full-blown cluster with a total of 80 cpu cores @ 1.5GHz and 80GB DDR4-3200 SDRAM.

There she blows!

Configuration

I’m not going to go into details how to setup the PI’s and installing Kubernetes on them, it is already well explained here, but I will give an overview of the setup. The PIs are running Raspbian Lite, the official supported operating system from the Raspberry PI Foundation. The setup is quite easy with flashing the image and setting up some basic configuration. As Kubernetes cluster, I use k3s from Rancher Labs, a lightweight version of Kubernetes supporting the ARMv7 processor architecture supported by the PI’s Cortex-CPU. It does not come with Docker bundled but the smaller containerd, which is a graduated project under the Cloud Native Computing Foundation, CNCF.

Fun fact by the way; k8s is a numeronym for Kubernetes, but what k3s means is more of a mystery. I’m gonna go with Kubes!

Lessions Learned

During the installation of Kubes, I stumbled upon a couple of issues that I want to share.

Explict Version

Running curl -sfL https://get.k3s.io | sh - will download a boostrap script of k3s which targets the latest version of the k3s image. If the installation for the whole cluster takes a couple of days, you might like me end up with a new latest version released causing incompatibility. I ended up getting the error “tls: failed to find any PEM data in key input” when trying to join one newly installed agent because of a change in how Go decodes key files which was introduced in a later version.

Always target an explicit version using INSTALL_K3S_VERSION=<version>, ie curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=<version> sh -.

iptables

k3s is using iptables-legacy but Rasbian/Buster is using iptables 1.8.2 nft which sometimes seems to cause problems when adding the agent to the cluster. To solve it, set iptables to legacy by executing sudo update-alternatives --set iptables /usr/sbin/iptables-legacy.

VPN

Trying to resolve local hostnames on the local network in Ubuntu (WSL) when connected to an external VPN service can be a hassle!

Missing Files During Installation

Sometimes the k3s installation process fails storing all files on the SD card in the PI causing the k3s agent to fail with errors like:
FATA[0000] exec: "k3s-agent": executable file not found in $PATH

The resolution is to remove /var/lib/rancher/k3s/data and re-launch the configuration process.

Shutting Down the Cluster

Just pulling the plug from a Raspberry PI is never a good idea, but shutting down a whole cluster of them one by one is tedious. I use the following script to shut them all down.

hostnames=(<server>)
for ((i=1;i<=<number-of-clients>;i++)); do
    hostnames+=("<client>-$i")
done
for hostname in ${hostnames[@]}
do
    ssh pi@$hostname 'sudo poweroff'
done

Exchange <server> with the server’s hostname, <client> with the starting name of the clients hostname and <number-of-clients> with the actual number of clients.

Conclusions

Running Kubernetes using k3s on a couple of Raspberry PI works quite good, without having battle tested the setup yet. Heck, I don’t even know what I will use it for! But it’s pretty, right? 🙂

Scaling in process migrations with Kubernetes

Published by Fredrik on October 10, 2019

I was recently involved in migrating data from one database to another for an application deployed to Kubernetes. The idea was to use an in-process migration routine that kicks in during the startup process of the application. Resilience and scaling were to be handled entirely by Kubernetes. Any failure would cause the application to fail fast and let k8s restart it.

This turned out to be quite simple to implement!

We use a rolling update strategy in order to let the newly deployed service migrate the data side-by-side with the old one still running the actual application process. This is defined in the application’s deployment manifest:

strategy:
  type: RollingUpdate
  rollingUpdate:
     maxUnavailable: 50%
     maxSurge: 50%

With a replica count of 8, we will now end up with a minimum of 4 pods running the old version and up to 8 pods running the migration.

However, for the rolling update strategy to work, we also need a readiness probe for the service to tell the deployment when it is okay to swap out pods. Since the application is a message streaming service hosted in a generic .NET Core host, we could simply use an ExecAction probe that executes cat against a file which existence we can control during the life cycle of the application. Simple!

The application’s life cycle went from:

private static async Task Main(string[] args)
{
    using var host = new HostBuilder().Build();
    await host.RunAsync();
}

…to something like this:

internal class Program
{
    private static async Task Main(string[] args)
    {
        using var host = new HostBuilder().Build();
        await host.StartAsync();
        File.Create("/Ready");
        await host.WaitForShutdownAsync();
        File.Delete("/Ready");
        await host.StopAsync();
    }
}

During the startup phase, the migration takes place. At this time, no file called “Ready” exists in the application’s execution directory. When start finishes (the migration is done and the application is ready to serve), the Ready file is created. When sigterm is received, the Ready file is deleted and we start the shutdown process. At this time, the application is no longer ready to serve.

What about the probe configuration? Easy!

readinessProbe:
  exec:
    command: 
    - cat
    - /Ready
  periodSeconds: 2

Every other second, the container will be checked for a file called Ready. If it exists, the service is considered ready for service and the deployment will continue and deploy the next pod according to the update strategy.

Need more scaling? Just add more replicas!

Distributed Tracing with OpenTracing and Kafka

Published by Fredrik on January 15, 2019

OpenTracing has become a standardized way to write distributed tracing within the microservices architecture. It has been adopted by many libraries in all kinds of programming languages. I was missing a library in .NET for creating distributed traces between Kafka clients, so I decided to write one!

OpenTracing.Confluent.Kafka

OpenTracing.Confluent.Kafka is a library build in .NET Standard 1.3. With the help of some extension methods and decorators for the Confluent.Kafka consumer and producer it makes distributed tracing between the producer and the consumer quite simple.

Example

There is a simple console app that serves as an example on how to use the decorators to create a trace with two spans, one for the producing side and one for the consuming side. It’s build in .NET Core, so you can run it where ever you like. It integrates with Kafka and Jaeger, so in order to setup these systems I have provided a simple script and some Docker and Kubernetes yaml files for a one-click-setup. If you don’t run under Windows, you need to setup everything manually. The script should be able to act as an indication of what needs to be done.

The result when running the app should look something like the image below.

Happy tracing!

Year: 2019