protocol – coding like a boss

Spdy – A .NET client

Published by Fredrik on February 4, 2021

Protocols, more protocols! I love deep-diving into them, and have wanted to explore the secret mysteries of HTTP/2 and gRPC ever since they were released some years ago, but havn’t had any particular reasons to do so, until now.

Kubernetes and the Streaming APIs

You have probably used them many times, the streaming functions exposed by the Kubernetes API Server; exec, port-forward, attach, logs etc. They all stream data back and forth between your local host and containers running on a Kubernetes node. They more or less support one data transport layer; Spdy (…and partly WebSocket) (…and sooooon HTTP/2).

Spdy

Spdy is a deprecated Chromium project that has been superseeded by HTTP/2. The Spdy protocol specification layed the foundation for the HTTP/2 specification and therefor they naturally share a lot of concepts.

Spdy was born due to the increasing usage of network bandwidth and need for low latency communication by modern web applications where HTTP/1.1 (and WebSocket) become bottle necks.

Web applications today depend on a lot of resources, but HTTP/1.1 only supports one request at a time per connection. This has somewhat been mitigated by browsers creating more connections towards the web servers, which is not very effecient considering the growth rate of consumer ip traffic. At best, this is a work around which at the end leads to socket starvation problems for the web servers.

Other limitations are one-way-communication initiation (client to server), redundant and uncompressed metadata (headers), downstream throttling and head-of-line blocking.

Spdy <3 .NET

Up until now, there hasn’t been a Spdy implementation available for .NET (as far as I can tell), and since I needed a reliable, duplex communication transfer protocol to integrate my upcoming project, Port, with Kubernetes’ port-forward API, I decided to implement Spdy, a high level client / server implementation of the 3.1 protocol specification!

As with the Kafka Test Framework, Spdy also utilizes System.IO.Pipelines to minimize buffer copying resulting in lower latency and CPU usage. It also comes with an ASP.NET Core middleware (soon) enabling upgrades of HTTP/1.1 requests to Spdy similar to how requests can be upgraded to WebSocket.

For a sneak peek of the Kubernetes port-forward integration using Spdy, head over to Port’s repository and the Spdy port forwarder.

Integration Testing with Kafka

Published by Fredrik on May 14, 2020

Does the heading sound familiar? 2 years ago I released an integration testing framework for the AMQP protocol.

Now it’s time for something similar. Say hello to Kafka.TestFramework!

Kafka

Kafka was originally developed by LinkedIn and open-sourced in 2011. It is a distributed streaming platform that uses pub/sub to distribute messages to topics in form of a partitioned log. It is built to scale while maintaining performance no matter how much data you throw at it.

The Protocol

Kafkas protocol is build based on a request/response model. The client sends a request, and the broker sends back a response. Versioning is built into the messages, where each and every property is defined with a version range. This makes versioning quite easy to handle and it is possible to build one platform on top of all versions. Property types are defined by a mix of simple data types and variable-length zig-zag encoded types using Google Protocol Buffers.

All message definitions can be found on github, and the protocol is explained in detail in the Kafka protocol guide. Both of these resources make up the core of the Kafka.Protocol project.

Kafka.Protocol

The testing framework is built upon protocol definitions. Kafka.Protocol auto-generates all protocol messages, primitives and serializers into typed C# classes. The library makes it possible to write client and server implementations of the kafka protocol using only .NET. This makes it easier to debug and understand the protocol in a C# point of view and removes any dependencies to libraries like librdkafka and it’s interop performance implications.

Kafka.TestFramework

The Kafka.TestFramework aims to make it possible to test integrations with Kafka clients. It acts as a server and exposes an api that makes it possible to subscribe on requests and send responses to clients. It can run in isolation in-memory or hook up to a TCP socket and is based on the protocol definitions found in Kafka.Protocol.

Both Kafka.TestFramework and Kafka.Protocol are built on top of System.IO.Pipelines, a high-performant streaming library that greatly simplifies handling of data buffers and memory management. I’m not going to go into details on how that works, there’s already a lot of good articles about this library. Btw, did I say it’s FAST? 🙂

To see some examples on how the framework can be utilized, check out the tests where it communicates with Confluent’s .NET client for Kafka!

The Future

The Kafka protocol is quite complex, specifically when combined with the distribution nature of Kafka. Many of it’s processes can be hard to understand and might be interpretated as too low-level. These processes can probably be packaged in a simple way making it possible to focus on the high-level messages of the business domain.

Yet, at the same time, it is powerful being able to simulate different cluster behaviours that might be hard to replicate on high-level clients. Being able to get both would be nice.

Test it out, let me know what you think!

Integration Testing with AMQP

Published by Fredrik on June 25, 2018

So, last week I finally released the first version of my new shiny integration testing framework for AMQP, Test.It.With.AMQP. It comes with an implementation of the AMQP 0.9.1 protocol and integration with the popular .NET AMQP client of RabbitMQ.

Oh yeah, it’s all compatible with .NET Core 🙂

AMQP – Advanced Message Queuing Protocol

Wikipedia:

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security.

Example

A common test scenario is that you have an application that consumes messages from a queue and you want to assert that the application retrieves the messages correctly.

var testServer = new AmqpTestFramework(Amqp091.Protocol.Amqp091.ProtocolResolver);
testServer.On<Basic.Consume>((connectionId, message) => AssertSomething(message));
myApplicationsIOCContainer.RegisterSingleton(() => testServer.ConnectionFactory.ToRabbitMqConnectionFactory()));

This is simplified though. In reality there are alot of setup negotiation that needs to be done before you can consume any messages, like creating a connection and a channel. A real working test with a made up application and the test framework Test.It.While.Hosting.Your.Windows.Service can be found here.

Why?

The purpose of this test framework is to mock an AMQP communication based service in order to test the AMQP integration points and behaviour within an application without the need of a shared and installed instance of the actual AMQP service. It’s kind of what OWIN Test Server does for HTTP in Katana.

Fast

The test framework runs in memory, that means no time consuming network traffic or interop calls.

Isolated

All instances are setup by the test scenario and has no shared resources. This means there is no risk that two or more tests affect each other.

Testable

The framework makes it possible to subscribe and send all AMQP methods defined in the protocols, or you can even extend the protocol with your own methods!

Easy Setup and Tear Down

Create an instance when setting up your test, verify your result, and dispose it when your done. No hassle with communication pools and locked resources.

Integration testing made easy.

Category: protocol