Skip to content

coding like a boss Posts

Semantic Release Notes with Conventional Commits

Recently I wrote about how to use Conventional Commits to calculate the next release version based on semantic versioning. Today I’m going to show how to turn conventional commit messages into release notes using a simple Github Action called Semantic Release Notes Generator that I’ve written.

release-notes-generator

The action is basically just a wrapper for semantic-release/release-notes-generator which is a plugin for semantic-release, an opinionated version management tool. When trying out semantic-release I found it to be too much functionality entangled in a way I couldn’t get to work the way I wanted it to, but I really liked the release notes generator! Being a fan of the Unix philosophy, I decided to wrap it in a neat little action that I could pipe.

Usage

The generator fetches all commit messages between two git references, and feeds them into the release-notes-generator that formats the messages into a nice looking release log.

The semantic release notes generator actually uses a Github Action workflow to test itself (more about that in another post). It uses gitversion to determine next version and pipe that to the generator to generate release notes based on the commits logged since the last release.

Here’s how that could look like:

...
- name: Determine Version
  id: gitversion
  uses: gittools/actions/gitversion/execute@v0
  with:
    useConfigFile: true
    configFilePath: .github/version_config.yml
- name: Determine Release Info
  id: release
  run: |
    from_tag=$(git tag --points-at ${{ steps.gitversion.outputs.versionSourceSha }} | grep -m 1 ^v[0-9]*\.[0-9]*\.[0-9]* | head -1)
    tag=v${{ steps.gitversion.outputs.majorMinorPatch }}

    echo "::set-output name=tag::$tag"
    echo "::set-output name=from_ref_exclusive::$from_ref_exclusive"
- name: Create Tag
  uses: actions/github-script@v3
  with:
    script: |
      github.git.createRef({
        owner: context.repo.owner,
        repo: context.repo.repo,
        ref: "refs/tags/${{ steps.release.outputs.tag }}",
        sha: "${{ steps.gitversion.outputs.sha }}"
      });
- name: Generate Release Notes
  id: release_notes
  uses: fresa/release-notes-generator@v0
  with:
    version: ${{ steps.release.outputs.tag }}
    from_ref_exclusive: ${{ steps.release.outputs.from_ref_exclusive }}
    to_ref_inclusive: ${{ steps.release.outputs.tag }}
- name: Create Release
  uses: softprops/action-gh-release@v1
  with:
    body: ${{ steps.release_notes.outputs.release_notes }}
    tag_name: ${{ steps.release.outputs.tag }}
...

It calculates the next version and finds the last release’s git reference. Then it creates a tag to mark the next release, which also will be used as to_ref_inclusive when generating the release notes. Lastly it creates the Github release using the release notes and the tag created.

If there’s a need to change the release notes, it’s always possible to edit the release in Github either after the release has been published or by releasing it as a draft and manually publishing it after a review.

Summary

release-notes-generator is a great tool for generating release notes when using conventional commits. With Semantic Release Notes Generator you can in an unopiniated way choose how to use release notes within your automated continuous delivery processes!

Leave a Comment

Simple Release Versioning with Trunkbased Development, SemVer and Conventional Commits

I have scripted versioning strategies for Trunk Based Development many times thoughout my career, including using simple Github Actions like Github Tag Action, but the result was never spot on. So I finally gave in overcoming the complexity of tuning GitVersion!

Trunk Based Development

From trunkbaseddevelopment.com:

A source-control branching model, where developers collaborate on code in a single branch called ‘trunk’ *, resist any pressure to create other long-lived development branches by employing documented techniques. They therefore avoid merge hell, do not break the build, and live happily ever after.

A great fit for Continuous Delivery, and very easy to apply!

SemVer

Versioning does not need to be just about incrementing a number to distinguise releases. It can also be semantic; alas SemVer, or short for Semantic Versioning.

From the site:

Given a version number MAJOR.MINOR.PATCH, increment the:

  1. MAJOR version when you make incompatible API changes,
  2. MINOR version when you add functionality in a backwards compatible manner, and
  3. PATCH version when you make backwards compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

This is all nice and such, but how do you integrate that with the process of continuously delivering code?

Conventional Commits

Conventional Commits is a convention for how to turn human written commit messages into machine readable meaning. It also happen to nicely follow the convention for SemVer.

  • Major / Breaking Change
refactor!: drop support for Node 6

BREAKING CHANGE: refactor to use JavaScript features not available in Node 6.
  • Minor / Feature
feat(lang): add polish language
  • Patch
fix: correct minor typos in code

GitVersion

GitVersion looks simple and promising at first glance.

However, when digging into the documentation, a different picture immerges.

GitVersion comes with three different strategies; Mainline Development, Continuous Delivery and Continuous Deployment, all quite difficult to decipher and grasp. All of them seems at first to be a possible fit for Trunk Based Development, but I ended up only getting Continuous Delivery to follow the simple philipsopy of Trunk Based Development, yet being flexible enough of when to actually bump versions and not bump any of major/minor/patch more then once until merge to main happens.

Configuration

GitVersion uses a config file to configure strategies, and there are quite few configurable aspects of said strategies. Let’s get started!

Branches

Trunk Based Development is all about getting into trunk (main/master in git lingo) fast. We have two branch strategies; either we are on main (where we don’t do any development) or we are on a development branch. GitVersion however comes with a lot of them that needs to be disabled.

  feature:
    # Match nothing
    regex: ^\b$
  develop:
    # Match nothing
    regex: ^\b$
  release:
    # Match nothing
    regex: ^\b$
  pull-request:
    # Match nothing
    regex: ^\b$
  hotfix:
    # Match nothing
    regex: ^\b$
  support:
    # Match nothing
    regex: ^\b$y

We consider all branches except main as development branches. These should incremented with patch by default.

  development:
    increment: Patch
    # Everything except main and master
    regex: ^(?!(main|master)$)

Detecting Conventional Commits

The following regular expressions detect when to bump major/minor/patch, and are based on the Conventional Commit v1.0.0 specification.

  • Major
    major-version-bump-message: "(build|chore|ci|docs|feat|fix|perf|refactor|revert|style|test)(\\([a-z]+\\))?(!: .+|: (.+\\n\\n)+BREAKING CHANGE: .+)"
  • Minor
    minor-version-bump-message: "(feat)(\\([a-z]+\\))?: .+"
  • Patch
    patch-version-bump-message: "(build|chore|ci|docs|fix|perf|refactor|revert|style|test)(\\([a-z]+\\))?: .+"

Here’s the complete config:

mode: ContinuousDelivery
# Conventional Commits https://www.conventionalcommits.org/en/v1.0.0/
# https://regex101.com/r/Ms7Vx6/2
major-version-bump-message: "(build|chore|ci|docs|feat|fix|perf|refactor|revert|style|test)(\\([a-z]+\\))?(!: .+|: (.+\\n\\n)+BREAKING CHANGE: .+)"
# https://regex101.com/r/Oqhi2m/1
minor-version-bump-message: "(feat)(\\([a-z]+\\))?: .+"
# https://regex101.com/r/f5C4fP/1
patch-version-bump-message: "(build|chore|ci|docs|fix|perf|refactor|revert|style|test)(\\([a-z]+\\))?: .+"
# Match nothing
no-bump-message: ^\b$
continuous-delivery-fallback-tag: ''
branches:
  development:
    increment: Patch
    # Everything except main and master
    regex: ^(?!(main|master)$)
    track-merge-target: true
    source-branches: []
  feature:
    # Match nothing
    regex: ^\b$
  develop:
    # Match nothing
    regex: ^\b$
  main:
    source-branches: []
  release:
    # Match nothing
    regex: ^\b$
  pull-request:
    # Match nothing
    regex: ^\b$
  hotfix:
    # Match nothing
    regex: ^\b$
  support:
    # Match nothing
    regex: ^\b$

Tagging

A common practice when releasing in git is to tag a commit with the release number, i.e. v1.2.3. When practicing continuous delivery, this can become quite verbose, especially when producing pre-releases, as every commit becomes a potential release. GitVersion Continuous Delivery strategy traverses each commit from last version tag when calculating the current version, so pre-releases does not necessary need to be tagged. Instead, the commit sha can be used as the pre-release identifier for back-referencing any pre-release artifacts.

Ofcourse, when using Github or similar platforms, a release is a tag, so in that case you might want to use tagging for pre-releases anyway.

Caveats

Forgetting to merge a merge commit back to the development branch when continuing working on the same branch means the automatic patch increment that happens for commits after branching does not occur. This will skew any pre-release versions. According to the documentation, the parameter track-merge-target should solve that scenario, but it seems it has not been implemented!

Summary

GitVersion turned out to be a really nice versioning tool once overcoming the knowledge curve of understanding all the concepts, features and configurations. Check out the GitVersion Github Action and Azure Pipeline Task for simple integration!

Leave a Comment

Spdy – A .NET client

Protocols, more protocols! I love deep-diving into them, and have wanted to explore the secret mysteries of HTTP/2 and gRPC ever since they were released some years ago, but havn’t had any particular reasons to do so, until now.

Kubernetes and the Streaming APIs

You have probably used them many times, the streaming functions exposed by the Kubernetes API Server; exec, port-forward, attach, logs etc. They all stream data back and forth between your local host and containers running on a Kubernetes node. They more or less support one data transport layer; Spdy (…and partly WebSocket) (…and sooooon HTTP/2).

Spdy

Spdy is a deprecated Chromium project that has been superseeded by HTTP/2. The Spdy protocol specification layed the foundation for the HTTP/2 specification and therefor they naturally share a lot of concepts.

Spdy was born due to the increasing usage of network bandwidth and need for low latency communication by modern web applications where HTTP/1.1 (and WebSocket) become bottle necks.

Web applications today depend on a lot of resources, but HTTP/1.1 only supports one request at a time per connection. This has somewhat been mitigated by browsers creating more connections towards the web servers, which is not very effecient considering the growth rate of consumer ip traffic. At best, this is a work around which at the end leads to socket starvation problems for the web servers.

Other limitations are one-way-communication initiation (client to server), redundant and uncompressed metadata (headers), downstream throttling and head-of-line blocking.

Spdy <3 .NET

Up until now, there hasn’t been a Spdy implementation available for .NET (as far as I can tell), and since I needed a reliable, duplex communication transfer protocol to integrate my upcoming project, Port, with Kubernetes’ port-forward API, I decided to implement Spdy, a high level client / server implementation of the 3.1 protocol specification!

As with the Kafka Test Framework, Spdy also utilizes System.IO.Pipelines to minimize buffer copying resulting in lower latency and CPU usage. It also comes with an ASP.NET Core middleware (soon) enabling upgrades of HTTP/1.1 requests to Spdy similar to how requests can be upgraded to WebSocket.

For a sneak peek of the Kubernetes port-forward integration using Spdy, head over to Port’s repository and the Spdy port forwarder.

Leave a Comment

Pulsar – The New Kafka?

I’ve been following Apache Pulsar for a while now, reading what ever articles I come by. I thought I should write a short summary of my thoughts about this “new”, exiting, distributed message streaming platform.

Kafka

I have gained experience working with Apache Kafka for some time, and it does have a lot in common with Pulsar. It has been around for a couple of more years, and I think it is a good message streaming platform. I wrote a protocol definition of the Kafka protocol earlier this year together with an in-memory test framework including an example on how you can test applications that for example uses Confluent’s .NET Kafka client. You can read all about it here.

Pulsar vs Kafka

Last week I came accross a couple of articles by Confluent (managed cloud-native platform for Kafka) and StreamNative (managed cloud-native platform for Pulsar) comparing the two platforms performance abillities that I think are really interesting (though maybe a bit biased).

Spoiler alert! Both are best! 😉

You can study them here and here.

Performance

What I found really interesting is the high throughput and the almost constant low latency Pulsar seems to achieve no matter the amount of partitions used while still being able to guarantee really good durability (consistency and availability). If you know about the CAP theorem, you would almost think this is too good to be true!

Different Storage Strategies

Kafka stores messages in partitions, while Pulsar has another storage level called segments that make up a partition. This might partly explain why Pulsar can perform better at some scenarios.

Credit: Splunk

In Kafka as a partition grows, more data needs to be copied when brokers leave or join the cluster. In Pulsar, segments can be formed in smaller, static sizes. An important observation here is that the partions can have an inifite amount of segments spread out in the Apache BookKeeper storage nodes, also known as Bookies. Therefor partitions in Pulsar can have an infinite amount of messages, while in Kafka, partitions will be bound by the hardware it is stored on. Replicating a partition will become slower and slower in Kafka, but in Pulsar, it could theoratically stay almost constant because of the possibility to scale storage infinitly.

However, brokers in Pulsar need to potentially talk to many storage nodes to serve a partition, while a Kafka broker has the whole partition stored directly on it’s own disk. This could cause the brokers to end up having to handle a lot of connections to the storage nodes.

Consumer Group / Subscription Rebalancing

Kafka has made some improvements in their partion rebalancing strategy lately, like Static Membership and Incremental Cooperative Rebalancing which has sped up the consumer group rebalancing act.

In Pulsar, consumers use hash ranges in order to share the load when consuming from a shared subscription. The broker handles the rebalancing and as soon as a consumer leaves or joins, no messages will be delivered to any consumers until the hash ranges have been redistributed, either by the broker when using auto hash distribution, or by the consumers when using sticky hash ranges. This might cause downtime and latency due to the stop-the-world-strategy when redistributing the hashes. This was the case in Kafka as well before it was mitigated by Incremental Cooperative Rebalancing.

Broker Rebalancing

Kafka does not have any automated rebalancing built in. Instead users are left depending on tools like LinkedIn’s Cruise Control. Since Kafka stores the topic partition data and replicas directly on the brokers, this needs to be copied and rebalanced when adding a new broker.

Pulsar’s architecture on the other side, which separates computation (broker) and storage (partition), enables almost instant rebalancing since brokers can just switch from what storage they read from or write to. Apache ZooKeeper is in charge of monitoring broker health and initiates recovery when a broker is deemed lost. This will cause other brokers to take over the ownership of the lost broker’s owned topics. Any split-brain scenarios between the brokers are handled by BookKeeper through a fencing mechanism causing only one broker at a time to be allowed to write to the topic’s ledgers.


Credits: Jack Vanlightly

Metadata Storage

Both Pulsar and Kafka uses Apache Zookeeper as it’s metadata management storage. The Kafka team announced a while back that they are dropping Zookeeper in favor to bring metadata into Kafka for less complexity, replication, better scalability and bootstrap effeciency. It would be interesting to know if there has been a similar discussion around ZooKeeper within the Pulsar project, and what potential performance gains scrapping ZooKeper might give Kafka.

As a side note, ZooKeeper is a central part of BookKeeper as well, so dropping it all together would probably prove very difficult. If ZooKeeper goes down, Pulsar goes down.

Conclusion

Comparing two seamingly similar distributed platforms can be complex. While using similar test setups but with a few knobs tweeked, the result can differ a lot. I think this quote from one of StreamNative’s articles sums it up pretty good:

– “Ultimately, no benchmark can replace testing done on your own hardware with your own workloads. We encourage you to evaluate additional variables and scenarios and to test using your own setups and environments.”

Understanding the impact infrastructure and architecture has on systems and applications, observability, operations and to have the knowledge about all those things have become more important than ever.

Leave a Comment

Integration Testing with Kafka

Does the heading sound familiar? 2 years ago I released an integration testing framework for the AMQP protocol.

Now it’s time for something similar. Say hello to Kafka.TestFramework!

Kafka

Kafka was originally developed by LinkedIn and open-sourced in 2011. It is a distributed streaming platform that uses pub/sub to distribute messages to topics in form of a partitioned log. It is built to scale while maintaining performance no matter how much data you throw at it.

The Protocol

Kafkas protocol is build based on a request/response model. The client sends a request, and the broker sends back a response. Versioning is built into the messages, where each and every property is defined with a version range. This makes versioning quite easy to handle and it is possible to build one platform on top of all versions. Property types are defined by a mix of simple data types and variable-length zig-zag encoded types using Google Protocol Buffers.

All message definitions can be found on github, and the protocol is explained in detail in the Kafka protocol guide. Both of these resources make up the core of the Kafka.Protocol project.

Kafka.Protocol

The testing framework is built upon protocol definitions. Kafka.Protocol auto-generates all protocol messages, primitives and serializers into typed C# classes. The library makes it possible to write client and server implementations of the kafka protocol using only .NET. This makes it easier to debug and understand the protocol in a C# point of view and removes any dependencies to libraries like librdkafka and it’s interop performance implications.

Kafka.TestFramework

The Kafka.TestFramework aims to make it possible to test integrations with Kafka clients. It acts as a server and exposes an api that makes it possible to subscribe on requests and send responses to clients. It can run in isolation in-memory or hook up to a TCP socket and is based on the protocol definitions found in Kafka.Protocol.

Both Kafka.TestFramework and Kafka.Protocol are built on top of System.IO.Pipelines, a high-performant streaming library that greatly simplifies handling of data buffers and memory management. I’m not going to go into details on how that works, there’s already a lot of good articles about this library. Btw, did I say it’s FAST? 🙂

To see some examples on how the framework can be utilized, check out the tests where it communicates with Confluent’s .NET client for Kafka!

The Future

The Kafka protocol is quite complex, specifically when combined with the distribution nature of Kafka. Many of it’s processes can be hard to understand and might be interpretated as too low-level. These processes can probably be packaged in a simple way making it possible to focus on the high-level messages of the business domain.

Yet, at the same time, it is powerful being able to simulate different cluster behaviours that might be hard to replicate on high-level clients. Being able to get both would be nice.

Test it out, let me know what you think!

Leave a Comment

Raspberry PI <3 Kubernetes

I have been curious for a while if I could build a powerful Kubernetes cluster that is compact, cheap and easy to use. Then one day I came across BitScope and their blade racks, and I knew I had to get one!

Yepp, that is a 20 node Kubernetes cluster of Raspberry PI 4 model B 4GB.

The Build

I got the 20 unit blade rack which includes two Duo Pi Blade Packs and 10 Duo Pis. It is delivered as a kit that you need to assemble yourself.

Time for assembly!

You also need a couple of Raspberry PIs!

That’s a lot of computers

To handle networking I use a 24 port Netgear GS324T gigabit switch with 20 0.3m flat U/UTP Cat6 RJ45 ethernet cables. I use 32GB SanDisk Ultra micro-SDHC (Class 10 / UHS-I) cards as storage for each PI, which is enough to handle the DDR50 transfer mode supported by the PI. Since the Raspberry PI does not have any cooling and is installed in a quite cramped area, I bought some passive heatsinks and jerry-rigged two silent 200mm case fans from Noctua in order to prevent throttling during load. The blade rack is delivered with four 30mm fans that are attached to the back, but these are very noisy. To power it all I use a 12V20A power supply.

The cost? Around €2000 excl. tax. Not that bad, considering this a full-blown cluster with a total of 80 cpu cores @ 1.5GHz and 80GB DDR4-3200 SDRAM.

There she blows!

Configuration

I’m not going to go into details how to setup the PI’s and installing Kubernetes on them, it is already well explained here, but I will give an overview of the setup. The PIs are running Raspbian Lite, the official supported operating system from the Raspberry PI Foundation. The setup is quite easy with flashing the image and setting up some basic configuration. As Kubernetes cluster, I use k3s from Rancher Labs, a lightweight version of Kubernetes supporting the ARMv7 processor architecture supported by the PI’s Cortex-CPU. It does not come with Docker bundled but the smaller containerd, which is a graduated project under the Cloud Native Computing Foundation, CNCF.

Fun fact by the way; k8s is a numeronym for Kubernetes, but what k3s means is more of a mystery. I’m gonna go with Kubes!

Lessions Learned

During the installation of Kubes, I stumbled upon a couple of issues that I want to share.

Explict Version

Running curl -sfL https://get.k3s.io | sh - will download a boostrap script of k3s which targets the latest version of the k3s image. If the installation for the whole cluster takes a couple of days, you might like me end up with a new latest version released causing incompatibility. I ended up getting the error “tls: failed to find any PEM data in key input” when trying to join one newly installed agent because of a change in how Go decodes key files which was introduced in a later version.

Always target an explicit version using INSTALL_K3S_VERSION=<version>, ie curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=<version> sh -.

iptables

k3s is using iptables-legacy but Rasbian/Buster is using iptables 1.8.2 nft which sometimes seems to cause problems when adding the agent to the cluster. To solve it, set iptables to legacy by executing sudo update-alternatives --set iptables /usr/sbin/iptables-legacy.

VPN

Trying to resolve local hostnames on the local network in Ubuntu (WSL) when connected to an external VPN service can be a hassle!

Missing Files During Installation

Sometimes the k3s installation process fails storing all files on the SD card in the PI causing the k3s agent to fail with errors like:
FATA[0000] exec: "k3s-agent": executable file not found in $PATH

The resolution is to remove /var/lib/rancher/k3s/data and re-launch the configuration process.

Shutting Down the Cluster

Just pulling the plug from a Raspberry PI is never a good idea, but shutting down a whole cluster of them one by one is tedious. I use the following script to shut them all down.

hostnames=(<server>)
for ((i=1;i<=<number-of-clients>;i++)); do
    hostnames+=("<client>-$i")
done
for hostname in ${hostnames[@]}
do
    ssh pi@$hostname 'sudo poweroff'
done

Exchange <server> with the server’s hostname, <client> with the starting name of the clients hostname and <number-of-clients> with the actual number of clients.

Conclusions

Running Kubernetes using k3s on a couple of Raspberry PI works quite good, without having battle tested the setup yet. Heck, I don’t even know what I will use it for! But it’s pretty, right? 🙂

Leave a Comment

Scaling in process migrations with Kubernetes

I was recently involved in migrating data from one database to another for an application deployed to Kubernetes. The idea was to use an in-process migration routine that kicks in during the startup process of the application. Resilience and scaling were to be handled entirely by Kubernetes. Any failure would cause the application to fail fast and let k8s restart it.

This turned out to be quite simple to implement!

We use a rolling update strategy in order to let the newly deployed service migrate the data side-by-side with the old one still running the actual application process. This is defined in the application’s deployment manifest:

strategy:
  type: RollingUpdate
  rollingUpdate:
     maxUnavailable: 50%
     maxSurge: 50%

With a replica count of 8, we will now end up with a minimum of 4 pods running the old version and up to 8 pods running the migration.

However, for the rolling update strategy to work, we also need a readiness probe for the service to tell the deployment when it is okay to swap out pods. Since the application is a message streaming service hosted in a generic .NET Core host, we could simply use an ExecAction probe that executes cat against a file which existence we can control during the life cycle of the application. Simple!

The application’s life cycle went from:

private static async Task Main(string[] args)
{
    using var host = new HostBuilder().Build();
    await host.RunAsync();
}

…to something like this:

internal class Program
{
    private static async Task Main(string[] args)
    {
        using var host = new HostBuilder().Build();
        await host.StartAsync();
        File.Create("/Ready");
        await host.WaitForShutdownAsync();
        File.Delete("/Ready");
        await host.StopAsync();
    }
}

During the startup phase, the migration takes place. At this time, no file called “Ready” exists in the application’s execution directory. When start finishes (the migration is done and the application is ready to serve), the Ready file is created. When sigterm is received, the Ready file is deleted and we start the shutdown process. At this time, the application is no longer ready to serve.

What about the probe configuration? Easy!

readinessProbe:
  exec:
    command: 
    - cat
    - /Ready
  periodSeconds: 2

Every other second, the container will be checked for a file called Ready. If it exists, the service is considered ready for service and the deployment will continue and deploy the next pod according to the update strategy.

Need more scaling? Just add more replicas!

Leave a Comment

Distributed Tracing with OpenTracing and Kafka

OpenTracing has become a standardized way to write distributed tracing within the microservices architecture. It has been adopted by many libraries in all kinds of programming languages. I was missing a library in .NET for creating distributed traces between Kafka clients, so I decided to write one!

OpenTracing.Confluent.Kafka

OpenTracing.Confluent.Kafka is a library build in .NET Standard 1.3. With the help of some extension methods and decorators for the Confluent.Kafka consumer and producer it makes distributed tracing between the producer and the consumer quite simple.

Example

There is a simple console app that serves as an example on how to use the decorators to create a trace with two spans, one for the producing side and one for the consuming side. It’s build in .NET Core, so you can run it where ever you like. It integrates with Kafka and Jaeger, so in order to setup these systems I have provided a simple script and some Docker and Kubernetes yaml files for a one-click-setup. If you don’t run under Windows, you need to setup everything manually. The script should be able to act as an indication of what needs to be done.

The result when running the app should look something like the image below.

Happy tracing!

Leave a Comment

4 Ways to Include Symbols and Source Files when Shipping C# Libraries

The need for debugging NuGet packaged assemblies have always been around, but did you know there are a couple of ways to achieve the same thing? Let’s take a closer look.

One Package to Rule Them All

Back in the days, I’ve mostly done symbol packaging by including the pdb symbol file and the source code in the nuget package along with the assembly. Everything in one single NuGet package.

pros

  • No extra communication to download symbols and source
  • No need to know where symbols and source files are located

 cons

  • Larger NuGet package

 

Symbol Package

To mitigate bloated packages and unnecessary data due to lack of debugging needs, there is the possibility to pack the symbol file and source code in a symbol package. This is done by creating a .nuget package containing the runtime and a .symbol.nuget package containing runtime, symbols and source code. The package containing the runtime is uploaded to a NuGet package server and the symbol package is uploaded to a symbol source server, like NuGet smbsrc.

pros

  • Small NuGet package 

cons

  • Need to know where the symbols are stored
  • Extra communication to download symbols and source files

Note: SymbolSource does not support the portable PDBs that the .NET Core CLI tool generates.

SourceLink

Now part of the .Net Foundation, SourceLink support has been integrated into Visual Studio 2017 15.3 and the brand new smoking hot .NET Core 2.1 SDK. SourceLink include source file information into the symbol files, making symbol packages obsolete. The source files can then be downloaded on demand from the source code repository by the debugger using this piece of information.

pros

  • Medium NuGet package
  • No symbol source package needed
  • Only download the source files needed for debugging

cons

  • Might call the repository for source code many times

JetBrains dotPeek

Did you know you could utilize dotPeek as a local symbol server? Well now you do. dotPeek will automatically construct symbol files from the assemblies loaded, which will be associated with the decompiled source files.

pros

  • Small NuGet package
  • No symbols or source files needed!

cons

  • Quite slow
  • Need to disable Just My Code when debugging, making symbol loading even slower
  • Not the real source code; poorer code quality

Conclusion

There you have it! Which ever approach you prefer, there will as usual always be trade offs. However, SourceLink seems to gain traction and might have a shot to become the de facto standard of symbol source packaging.

Leave a Comment

Integration Testing with AMQP

So, last week I finally released the first version of my new shiny integration testing framework for AMQP, Test.It.With.AMQP. It comes with an implementation of the AMQP 0.9.1 protocol and integration with the popular .NET AMQP client of RabbitMQ.

Oh yeah, it’s all compatible with .NET Core 🙂

AMQP – Advanced Message Queuing Protocol

Wikipedia:

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security.

Example

A common test scenario is that you have an application that consumes messages from a queue and you want to assert that the application retrieves the messages correctly.

var testServer = new AmqpTestFramework(Amqp091.Protocol.Amqp091.ProtocolResolver);
testServer.On<Basic.Consume>((connectionId, message) => AssertSomething(message));
myApplicationsIOCContainer.RegisterSingleton(() => testServer.ConnectionFactory.ToRabbitMqConnectionFactory()));

This is simplified though. In reality there are alot of setup negotiation that needs to be done before you can consume any messages, like creating a connection and a channel. A real working test with a made up application and the test framework Test.It.While.Hosting.Your.Windows.Service can be found here.

Why?

The purpose of this test framework is to mock an AMQP communication based service in order to test the AMQP integration points and behaviour within an application without the need of a shared and installed instance of the actual AMQP service. It’s kind of what OWIN Test Server does for HTTP in Katana.

Fast

The test framework runs in memory, that means no time consuming network traffic or interop calls. 

Isolated

All instances are setup by the test scenario and has no shared resources. This means there is no risk that two or more tests affect each other.

Testable

The framework makes it possible to subscribe and send all AMQP methods defined in the protocols, or you can even extend the protocol with your own methods!

Easy Setup and Tear Down

Create an instance when setting up your test, verify your result, and dispose it when your done. No hassle with communication pools and locked resources.

 

Integration testing made easy.

 

 

Leave a Comment