Category: testing

In-Memory Service Integration Tests

Published by Fredrik on July 29, 2021

One concept I always use when writing applications or libraries is to test them using fast, reliable and automated integration tests.

The Test Trophy

You have probably heard about the test pyramid before, but if you haven’t heard about the test trophy, you should read this article.

Coming from the .NET world I have come to rely on real time code analyzers such as Roslyn and ReSharper, they help me write cleaner code faster and with little effort. It’s great having continuous feedback on code being written more or less instantly.

For similar reasons I use NCrunch for continuous test execution. I want instant feedback on the functionality I write. This means I need tests that are isolated and stable, run fast and continuously and test business functionality as realistically as possible.

Integration Tests

While unit tests are valuable when writing small isolated pieces of functionality, they are often too isolated to verify the bigger picture, hence the need for integration tests.

In order for integration tests to give near real time feedback they need to work similar to unit tests yet test real business scenarios end-to-end using stable integration points that do not break when refactoring. This is where in-memory, service integration tests come in.

In-memory – because it’s fast and can run anywhere.
Service – because it exposes stable APIs that don’t break compatibility.
Integrations – because they also use stable API’s and are pluggable.

These tests encapsulate an in-memory environment where the service is deployed to a host that acts similar to a real host through which the test can invoke the service. Third party in-memory representations of integrating services are used to mimic dependencies and observe the service’s behavior. The test shortcuts the service’s I/O integrations preferably as close to the network level as possible and redirects traffic back to itself for instrumentation and assertions.

An example of an integration test can be found here. It tests a web server that provides port forwarding functionality to Kubernetes, similar to kubectl port-forward.

Examples of in-memory versions of different components are Kafka.TestFramework, Test.It.With.AMQP, Entity Framework Effort as well as the AspNetCore TestServer.

So how fast do these integration tests run? Let’s churn.

Fast enough to run continuously!

JIT Lag

It is worth mentioning that most test runners do not run each test in a completely isolated application domain because of the overhead caused by the JIT compiler when loading assemblies. Therefor it is important to design services to be instantiable and not have mutating static properties as these would be shared when running tests in parallel. Reusing the same application domain for test executing when running multiple tests simultaneously is a trade-off that increases performance considerably.

Summary

Running integration tests continuously while writing code enables very fast feedback loops. These tests are however not a replacement to post-deployment end-to-end tests which test application functionality on production similar infrastructure.

Its main purpose is to give fast feedback continuously on business functionality during development by mimicking a production deployed instance of the service with minimal mocking while using well known, loosely coupled integration points for refactor stability.

Leave a Comment

Integration Testing with Kafka

Published by Fredrik on May 14, 2020

Does the heading sound familiar? 2 years ago I released an integration testing framework for the AMQP protocol.

Now it’s time for something similar. Say hello to Kafka.TestFramework!

Kafka

Kafka was originally developed by LinkedIn and open-sourced in 2011. It is a distributed streaming platform that uses pub/sub to distribute messages to topics in form of a partitioned log. It is built to scale while maintaining performance no matter how much data you throw at it.

The Protocol

Kafkas protocol is build based on a request/response model. The client sends a request, and the broker sends back a response. Versioning is built into the messages, where each and every property is defined with a version range. This makes versioning quite easy to handle and it is possible to build one platform on top of all versions. Property types are defined by a mix of simple data types and variable-length zig-zag encoded types using Google Protocol Buffers.

All message definitions can be found on github, and the protocol is explained in detail in the Kafka protocol guide. Both of these resources make up the core of the Kafka.Protocol project.

Kafka.Protocol

The testing framework is built upon protocol definitions. Kafka.Protocol auto-generates all protocol messages, primitives and serializers into typed C# classes. The library makes it possible to write client and server implementations of the kafka protocol using only .NET. This makes it easier to debug and understand the protocol in a C# point of view and removes any dependencies to libraries like librdkafka and it’s interop performance implications.

Kafka.TestFramework

The Kafka.TestFramework aims to make it possible to test integrations with Kafka clients. It acts as a server and exposes an api that makes it possible to subscribe on requests and send responses to clients. It can run in isolation in-memory or hook up to a TCP socket and is based on the protocol definitions found in Kafka.Protocol.

Both Kafka.TestFramework and Kafka.Protocol are built on top of System.IO.Pipelines, a high-performant streaming library that greatly simplifies handling of data buffers and memory management. I’m not going to go into details on how that works, there’s already a lot of good articles about this library. Btw, did I say it’s FAST? 🙂

To see some examples on how the framework can be utilized, check out the tests where it communicates with Confluent’s .NET client for Kafka!

The Future

The Kafka protocol is quite complex, specifically when combined with the distribution nature of Kafka. Many of it’s processes can be hard to understand and might be interpretated as too low-level. These processes can probably be packaged in a simple way making it possible to focus on the high-level messages of the business domain.

Yet, at the same time, it is powerful being able to simulate different cluster behaviours that might be hard to replicate on high-level clients. Being able to get both would be nice.

Test it out, let me know what you think!

Leave a Comment

Integration Testing with AMQP

Published by Fredrik on June 25, 2018

So, last week I finally released the first version of my new shiny integration testing framework for AMQP, Test.It.With.AMQP. It comes with an implementation of the AMQP 0.9.1 protocol and integration with the popular .NET AMQP client of RabbitMQ.

Oh yeah, it’s all compatible with .NET Core 🙂

AMQP – Advanced Message Queuing Protocol

Wikipedia:

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security.

Example

A common test scenario is that you have an application that consumes messages from a queue and you want to assert that the application retrieves the messages correctly.

var testServer = new AmqpTestFramework(Amqp091.Protocol.Amqp091.ProtocolResolver);
testServer.On<Basic.Consume>((connectionId, message) => AssertSomething(message));
myApplicationsIOCContainer.RegisterSingleton(() => testServer.ConnectionFactory.ToRabbitMqConnectionFactory()));

This is simplified though. In reality there are alot of setup negotiation that needs to be done before you can consume any messages, like creating a connection and a channel. A real working test with a made up application and the test framework Test.It.While.Hosting.Your.Windows.Service can be found here.

Why?

The purpose of this test framework is to mock an AMQP communication based service in order to test the AMQP integration points and behaviour within an application without the need of a shared and installed instance of the actual AMQP service. It’s kind of what OWIN Test Server does for HTTP in Katana.

Fast

The test framework runs in memory, that means no time consuming network traffic or interop calls.

Isolated

All instances are setup by the test scenario and has no shared resources. This means there is no risk that two or more tests affect each other.

Testable

The framework makes it possible to subscribe and send all AMQP methods defined in the protocols, or you can even extend the protocol with your own methods!

Easy Setup and Tear Down

Create an instance when setting up your test, verify your result, and dispose it when your done. No hassle with communication pools and locked resources.

Integration testing made easy.

Leave a Comment

Integration Testing

Published by Fredrik on July 14, 2017

Sexy title, isn’t it? 🙂 Well, maybe not, but still, it’s an important aspect of system development.

What is Integration Testing?

Good question, Fredrik. Well thank you, Fredrik.

Yeah yeah, okay, enough with this, let’s get serious.

When I talk about testing I almost always get misunderstood. Why? Because we all have different views and words to use when we talk about testing, and specially automated tests. Have you seen the explanation of Integration Testing on Wikipedia? Well, it’s not explicit, that you can say 🙂 https://en.wikipedia.org/wiki/Integration_testing

When I talk about integration tests, I usually mean firing up my application in memory and probe the integration points. I like this approach because it let’s me get rid of third party application dependency. It means I can run my tests anywhere without the need of installing and handling third party applications during testing, for example a database.

It fits nicely into the continues integration process.

Test It While Hosting Your Windows Service

So, I’d like to introduce you to some of the home brewed testing frameworks I like to use when building applications. Let’s start with Test.It.While.Hosting.Your.Windows.Service (C#, .NET Framework 4.6.2), which is a testing framework that helps you simplify how you write your integration tests for Windows Service applications.

I’m not going to go into details, you will find that if you follow the link. BUT, I think integration testing is something all developers should have at some degree when developing applications. They cover much more than a unit test, but at the same time they are autonomous and non-dependent which means you can run them anywhere where the targeted framework is installed.

Just by being able to start your application in a unit test like manner you have come far in testing your application’s functionality. How many times have you not experienced problems with the IOC registrations of your application?

Okay, I’ll cut the selling point. I think you get the point anyway. Testing is important. Multiple levels of testing is important. Finding issues fast is important, at least in an agile world, and you are agile, aren’t you? 🙂

Hi!

My name is Fredrik. I’m a system developer, newly gone freelance, and (obviously) I like automated testing. I would not call my self a ‘tester’, I’m a system developer who likes simplicity and when things just work.

This is my first blog post for my new company, FKAN Consulting. I’ll try to continuously post new (hopefully interesting) posts here about my experiences as a developer and consultant.

Leave a Comment