A while back I published a source code generator for C# that scaffolds web APIs from OpenAPI specifications. Since then I’ve realized that the available OpenAPI source generated clients, like Microsoft’s Kiota, seems to suffer similar lack in functionality and type safety as it’s web api scaffolding counterparts. So I decided to open source my own, strongly typed, fully featured, generator for web clients; OpenApi.WebClientGenerator.
The generator is exclusively generating clients in C#, and is, similar to the Web API generator, also an incremental source generator, meaning it generates the clients into an existing assembly.
using var client = new Example.Foo(Servers.Production);
The generated clients are strongly typed wrappers around HttpClient, so it’s still possible to customize behavior that might not be defined by an OpenAPI specification.
It supports all OpenAPI specifications currently published.
Keeping Web API implementations of an OpenAPI specification in-sync is difficult without automation, if not impossible. Several initiatives exists, they all come with various limitations and of course opiniated solutions, none have really solved all the various directives of OpenAPI, especially not schema validation. So I decided to build my own generator, OpenAPI.WebApiGenerator!
The generator is exclusive for .NET API scaffolding. It’s implemented as a source generator, so no external tooling required, it generates straight into your assembly!
Requests and responses are automatically validated and serialized according to the OpenAPI specification using the excellent JSON Schema model Corvus.JsonSchema. JSON schemas are difficult to express in C# structures, or any object oriented language for that matter. The traditional DTO approach which System.Text.Json takes are very limited in expressiveness, and validation is mostly left for the implementer. Corvus.Json delivers all-in one. JSON semantic structure, validation, serialization and marshalling to C# primitives, all according to the schema it was generated from. No scaffolding required!
ParameterStyleParsers.OpenAPI
Request parameters in HTTP is not defined as JSON, as content can be. Yet they are defined using JSON schemas in OpenAPI. ParameterStyleParsers.OpenAPI parses parameter styled instances to and from JSON according to the parameter specification. Read this post for more details.
The Future
It’s still early days, I’m planning on implementing many more advanced features, like streamable content and AuthN/AuthZ. Until then, give it a swirl!
Source generators that has package dependencies require some extra directives for Roslyn to load them properly, which differs depending if the generator is consumed as a package reference or a project reference.
Consider a source generator, MySourceGenerator, with a dependency to a fictive .NET Standard 2.0 package Foo.Bar. The package reference directive would look something like:
PrivateAssets="all" means that the consuming project doesn’t take a direct dependency, it’s not the consuming project that needs the dependency per se, it’s the compiler when running the source generator.
GeneratePathProperty="true" enables MSBuild targets to get the path to the corresponding assembly, which usually is within a global folder. The path will be used in both scenarios to reference the dependency.
Project Reference
A project consuming a source generator as a project reference would have a directive like:
We use the path (generated by GeneratePathProperty) to produce target path references to the dependency’s assembly to let MSBuild know about the dependency:
There is no MSBuild consuming the source generator project here, instead dependencies need to be packed into the source generator’s NuGet package. We can use the path in a similar fashion to reference the dependency to include it in the package using this directive:
analyzers/dotnet/cs is a special path in a NuGet package where analyzers and source generators are located. Their dependencies can be included at the same path.
Don’t forget transient dependencies, as described above.
Generated Code Dependencies
We’ve covered how to handle dependencies the source generator it self depends on, but how about dependencies that the code it generates depend on? There is currently no way to define such package reference, unless it targets the same framework as the source generator, meaning .NET Standard 2.0. PackageReference directives must be compatible with the project where they are declared. This means that required dependencies must be referenced manually by the consumer.
It can be a good idea to check that the expected dependency is installed in the target assembly (context.Compilation.ReferencedAssemblyNames) from the source generator and produce a diagnostics message if it is missing.
Conclusion
Handling dependencies in source generators is clonky and requires quite a lot of manually added directives. Hopefully we’ll see a more streamlined solution in the future where Roslyn deals with this automatically.
xUnit’s TestContainers library includes a fixture for simplifying writing integration tests, however it does not work well with reusable containers since TestContainers doesn’t support creating containers idempotently across parallel processes.
Why Reuse?
There are a couple of common strategies when orchestrating integration dependencies when writing integration tests:
In-memory orchestration
Real dependencies
Third-party emulators
TestContainers enable the two latter while simplifying orchestration, but the downside of these strategies compared to the first is start and stop latency, which may become a serious problem as the number of tests increase. Enter Reuse.
Lack of Idempotency
TestContainers uses the create container endpoint from the Docker API to create a new container. If Reuse is enabled, it first checks if their is a container matching a hash representing the labels of the current test container, if there is, it reuses it, otherwise it creates a new container using a random pseudo name which is not deterministic and hence makes this process non-idempotent. This works fine when running tests in a single, sequential process, but will eventually fail if using parallel processes or a process that run tests in parallel without synchronization.
An example is NCrunch, which by default parallelize test execution to multiple processes.
TestContainers will fail because it will eventually create more than one container with the same hash which will cause the framework to fail on the next execution, as it doesn’t support multiple containers with the same hash, nor does it align with the purpose of reusability.
TestContainers.Xunit.Reusable
TestContainers.Xunit.Reusable is a drop-in replacement for Testcontainers.XunitV3 that supports idempotency by simply using the reuse hash as the name of the reusable container. It implements optimistic concurrency, meaning if a process fails creating a reusable container due to conflict, it will reuse it as another parallel process managed to create the container first.
Simply replace <PackageReference Include="Testcontainers.XunitV3" Version="x.y.z" /> with <PackageReference Include="Reusable.XunitV3.TestContainers" Version="a.b.c" />. Enable reusability for a ContainerFixture by overriding the Reuse property or by setting the TESTCONTAINERS_REUSE_ENABLE environment variable to ‘true’.
Did you know that terraform apply might apply important state changes even if the plan states that no changes is detected?
The output should be familiar. When running terraform plan Terraform compares the current resource directives defined in code, with the current live state of the real resources it represents via the module’s remote state. If both the resource directives and the state matches the actual provisioned resources, no changes are needed.
Except, it might.
State Upgrades
Terraform modules keep track of the expected state of managed resources via what’s called the State. Besides keeping track of the state of the expected resources, it also contains metadata about the resource definitions currently used by the configured providers. In particular, it might contain information about resource schema migrations, or state upgrade directives. These are not well documented among user documentation, and is supposedly an internal mechanism for provider developers, however it might affect users as well.
Potential state upgrades are applied during apply, whether or not actual resource changes are needed or not. Not even the SDK documentation specifies this explicitly. There is also no record of such upgrades recorded in the plan.
Always Apply
It might be tempting to skip applying a plan that states that no changes are needed, but this could lead to incompatible and difficult upgrade problems later on due to missing state migrations and other metadata changes caused by upgrading a provider, or Terraform it self. For example subtle changes to an identifier of a resource, like case sensitivity, that may cause planning to fail for future versions of the provider. Add accidental breaking changes to the underlaying APIs that might be mitigated by future releases of the provider and you might end up in a catch-22 moment. Can’t apply state changes due to downstream API changes, can’t upgrade due to missing state upgrades.
I recently ported Kafka.Protocol‘s source code generation functionality from Text Templating (T4) to a Source Generator, and I thought I would share my experience with how they differ and what to expect.
Text Templating
Text Templating has been around since 2005 and is available on .NET Framework using C# 6. Mono.TextTemplating has been around for a couple of years which supports .NET and C# 10, and recently Visual Studio 2022 started to ship with a revamped CLI tool for text templating.
This is a design time template, it runs when the text template file is saved and produces the output in a separate file. There are also run time templates which can generate code from other code during runtime, which can simplify splitting generated code into multiple files, but requires some other code to run in order to do so. Design time templates produces a single file per text template. There are tooling that can get around this limitation, but it has it’s own limitations.
Source Generator
Source generators were first introduced in .NET 5, and runs during compile time. It has to target .NET Standard 2.0 but can use any C# version. It can generate source code from input like a data model specification or based on objects being compiled. Generated code is added to the compilation, meaning both ordinary written code and source generated code gets compiled together into the same assembly. This means that code generated isn’t written to any files, like for T4 templates, it’s written directly to the output assembly.
Writing Generated Code to Files
Emitting generated code to files can be enabled with some simple project directives:
There are no limitations on how many files a generator can produce or where it should be outputted. Storing generated source code in files are great if you want to track how changes in the source generator affects the generated code in source control, specifically if you are generating code from a specification and not content from the compilation.
Using a Generator
To use a source generator, add a reference to it from a project:
Note the OutputItemType="Analyzer" directive in the project reference directive. It tells the compiler that the project is to be treated as an analyzer instead of being a runtime reference. Output from an analyzer can be found under Dependencies in Visual Studio, and that’s where we find the generated types.
Note that they appear as files, even though they are not, it’s just an identifier. To find them in Visual Studio you would need to search for the type they contain or navigate to them in the Solution Explorer.
It would be possible to include the emitted files in a project and exclude them from compilation, that would make them searchable as any other file, but since they aren’t part of compilation they will lack some analysis disabling some functionality like symbol navigation etc. I recommend keeping emitted files solely for source control purposes.
Limitations
Source Generators, as analyzers, have limited exception handling. All exceptions thrown by a source generator is wrapped by a standard error message and contains very little information of what the problem is.
CSC : warning CS8785: Generator 'SourceGenerator' failed to generate source. It will not contribute to the output and compilation err
ors may occur as a result. Exception was of type 'NullReferenceException' with message 'Object reference not set to an instance of an object.'.
It’s possible to export the full exception including the stack trace by using the ErrorLog directive and output it as a SARIF formatted file. This isn’t great to work with as you’d like proper diagnostic feedback from the compiler directly. A workaround can be to construct a diagnostic error manually and include the stack trace, but neither multiline messages nor the description property is outputted so everything needs to be packed into a single-line message. Locations from stack traces are also problematic with incremental source generators where if a generator reruns with no code changes the stack frame location is gone.
Diagnostic reporting only work with error and warnings, other severities are ignored. A proposal on how informational diagnostic output should work can be found here.
Generated Code and Analyzers
The assemblies containing generated source code have had issues not being properly analyzed due to analyzers not being reloaded when the generated code in an assembly changes, which required deleting the .vs cache directory and restarting Visual Studio to force-reload them. This was resolved in the Visual Studio 2022 17.12 release.
Conclusion
Even though Source Generators still have a few quirks, they are much easier to work with than T4 Text Templates. It enables unit testing, file splitting and does not require running under Windows. They are also easier to distribute as they can be packed in NuGet-packages, and I’ve barely mentioned content based generator, which opens up a whole other world of opportunities! Check out the Source Generator Cook Book to get started.
Previously the suppression file, ApiCompatBaseline.txt, contained text directives describing suppressed compatibility issues.
Compat issues with assembly Kafka.Protocol:
TypesMustExist : Type 'Kafka.Protocol.ConsumerGroupHeartbeatRequest.Assignor' does not exist in the implementation but it does exist in the contract.
This format has changed to an XML based format, written by default to a file called CompatibilitySuppressions.xml.
This format is more verbose than the old format, a bit more difficult to read from a human perspective if you ask me. The description of the various DiagnosticIds can be found in this list.
Path Separator Mismatches
Looking at the suppression example, you might notice that the suppressions contain references to the compared assembly and the baseline contract. It’s not a coincidence that the path separators differs between the references to the contract assembly and the assembly being compared. The Left reference is a templated copy of the ApiCompatContractAssembly directive using OS agnostic forward slashes, but the Right directive is generated by ApiCompat and it is not OS agnostic, hence the backslash path separators generated when executing under Windows. If ApiCompat is executed under Linux it would generate front slash path separators.
You might also notice that the reference to the assembly being compared contains the build configuration name. This might not match the build configuration name used during a build pipeline for example (Debug vs Release).
Both these differences in path reference will make ApiCompat ignore the suppressions when not matched. There is no documentation on how to consolidate these, but fortunately there are a couple of somewhat hidden transformation directives which can help control how these paths are formatted.
<PropertyGroup>
<_ApiCompatCaptureGroupPattern>
.+%5C$([System.IO.Path]::DirectorySeparatorChar)(.+)%5C$([System.IO.Path]::DirectorySeparatorChar)(.+)
</_ApiCompatCaptureGroupPattern>
</PropertyGroup>
<ItemGroup>
<!-- Make sure the Right suppression directive is OS-agnostic and disregards configuration -->
<ApiCompatRightAssembliesTransformationPattern Include="$(_ApiCompatCaptureGroupPattern)" ReplacementString="obj/$1/$2" />
</ItemGroup>
The _ApiCompatCaptureGroupPattern regex directive captures path segment groups which can be used in the ApiCompatRightAssembliesTransformationPattern directive to rewrite the assembly reference path to something that is compatible to both Linux and Windows, and removes the build configuration segment.
Using this will cause the Right directive to change accordingly.
Parameters in the OpenAPI 3.1 specification can be defined in two ways using a JSON schema; either by using a media type object, which is useful when the parameter is complex to describe, or by using styles, which is more common in simple scenarios, which often is the case with HTTP parameters.
Let’s have a look at the styles defined and where they fit into a HTTP request.
Path Parameters
Path parameters are parameters defined in a path, for example id in the path /user/{id}. Path parameters can be described using label, matrix or simple styles, which are all defined by RFC6570, URI Template.
Here are some examples using the JSON primitive value 1 for a parameter named id :
Simple: /user/1
Matrix: /user/;id=1
Label: /user/.1
It’s also possible to describe arrays. Using the same parameter as above with two JSON primitive values, 1 and 2, it get’s serialized as:
Simple: /users/1,2
Matrix: /user/;id=1,2
Label: /user/.1.2
Given a JSON object for a parameter named user with the value, { "id": 1, "name": "foo" }, it becomes:
Simple: /user/id,1,name,foo
Matrix: /user/;user=id,1,name,foo
Label: /user/.id.1.name.foo
The explode modifier can be used to enforce composite values (name/value pairs). For primitive values this has no effect, neither for label and simple arrays. With the above examples and styles where explode has effect, here’s the equivalent:
Arrays
Matrix: /user/;id=1;id=2
Objects
Simple: /user/id=1,name=foo
Matrix: /user/;id=1;name=foo
Label: /user/.id=1.name=foo
Query Parameters
Query parameters can be described with form, space delimited, pipe delimited or deep object styles. The form style is defined by RFC6570, the rest are defined by OpenAPI.
Primitives
Using the example from path parameters, a serialized user, ?user={user}, or user id, ?id={id}, defined as a query parameter value would look like:
Form: id=1
Note that the examples doesn’t describe any primitive values for pipe and space delimited styles, even though they are quite similar to the simple style.
Arrays
Form: id=1,2
SpaceDelimited: id=1%202
PipeDelimited: id=1|2
Objects
Form: user=id,1,name,foo
SpaceDelimited: user=id%201%20name%20foo
PipeDelimited: user=id|1|name|foo
Note that the examples lack the parameter name for array and objects, this has been corrected in 3.1.1.
Not defining explode for deepObject style is not applicable, and like path styles, explode doesn’t have effect on primitive values.
Exploded pipe and space delimited parameters are not described in the example, even though they are similar to form. Do note though that neither of them would be possible to parse, as the parameter name cannot be inferred.
With all this in mind here are the respective explode examples:
Arrays
Form: id=1&id=2
SpaceDelimited: id=1%202
PipeDelimited: id=1|2
Objects
Form: id=1&name=foo
SpaceDelimited: id=1%20name=foo
PipeDelimited: id=1|name=foo
DeepObject: user[id]=1&user[name]=foo
Header Parameters
Header parameter can only be described using the simple style.
Given the header user: {user} and id: {id}, a respective header parameter value with simple style would look like:
Primitives
Simple: 1
Arrays
Simple: 1,2
Objects
Simple: id,1,name,foo
Similar to the other parameters described, explode with primitive values have no effect, neither for arrays. For objects it would look like:
Simple: id=1,name=foo
Cookie Parameters
A cookie parameter can only be described with form style, and is represented in a similar way as query parameters. Using the example Cookie: id={id} and Cookie: user={user} a cookie parameter value would look like:
Primitives
Form: id=1
Arrays
Form: id=1,2
Objects
Form: user=id,1,name,foo
Similar to the other parameters described, explode with primitive values have no effect. For arrays and objects it looks like:
Arrays
Form: id=1&id=2
Objects
Form: id=1&name=foo
Note that exploded objects for cookie parameters have the same problem as query parameters; the parameter name cannot be inferred.
Object Complexity
Theoretically an object can have an endless deep property structure, where each property are objects that also have properties that are objects and so on. RFC6570 nor OpenAPI 3.1 defines how deep a structure can be, but it would be difficult to define array items and object properties as objects in most styles.
As OpenAPI provides media type objects as a complement for complex parameters, it’s advisable to use those instead in such scenarios.
OpenAPI.ParameterStyleParsers
To support parsing and serialization of style defined parameters, I’ve created a .NET library, OpenAPI.ParameterStyleParsers. It parses style serialized parameters into the corresponding JSON instance and vice versa. It supports all examples defined in the OpenAPI 3.1 specification, corrected according to the inconsistencies described earlier. It only supports arrays with primitive item values and objects with primitive property values, for more complex scenarios use media type objects. The JSON instance type the parameter get’s parsed according to is determined by the schema type keyword. If no type information is defined it falls back on a best effort guess based on the parameter style.
I often get into the discussion about should you disable continuing on the captured context when awaiting a task or not, so I’m going to write down some of my reasoning around this rather complex functionality. Let’s start with some fundamentals first.
async/await
Tasks wrap operations and schedules them using a task scheduler. The default scheduler in .NET schedules tasks on the ThreadPool. async and await are syntactic sugar telling the compiler to generate a state machine that can keep track of the state of the task. It iterates forward by keeping track of the current awaiter’s completion state and the current location of execution. If the awaiter is completed it continues forward to the next awaiter, otherwise it asks the current awaiter to schedule the continuation.
Different components have different models on how scheduling of operations need to be synchronized, this is where the synchronization context comes into play. The default SynchronizationContext synchronizes operations on the thread pool, while others might use other means.
The most common awaiters, like those implemented for Task and ValueTask, considers the current SynchronizationContext for scheduling an operation’s continuation. When the state machine moves forward executing an operation it goes via the current awaiter which when not completed might schedule the state machines current continuation via SynchronizationContext.Post.
ConfigureAwait
The only thing this method actually does is wrap the current awaiter with it’s single argument, continueOnCapturedContext . This argument tells the awaiter to use any configured custom synchronization context or task scheduler when scheduling the continuation. Turned off, i.e. ConfigureAwait(false), it simply bypasses them and schedules on the default scheduler, i.e. the thread pool. If there are no custom synchronization context or scheduler ConfigureAwait becomes a no-op. Same thing apply if the awaiter doesn’t need queueing when the state machines reaches the awaitable, i.e. the task has already completed.
Continue on Captured Context or not?
If you know there is a context that any continuation must run on, for example a UI thread, then yes, the continuation must be configured to capture the current context. As this is the default behavior it’s not technically required to declare this, but by explicitly configure the continuation it sends a signal to the next developer that here a continuation is important. If configured implicitly there won’t be anything hinting that it wasn’t just a mistake to leave it out, or that the continuation was ever considered or understood.
In the most common scenario though the current context is not relevant. We can declare that by explicitly state the continuation doesn’t need to run on any captured context, i.e. ConfigureAwait(false).
Enforcing ConfigureAwait
Since configuring the continuation is not required, it’s easy to miss configuring it. Fortunately there is a Roslyn analyzer that can be enabled to enforce that all awaiters have been configured.
Summary
Always declare ConfigureAwait to show intent that the continuation behavior has explicitly been considered. Only continue on a captured context if there is a good reason for doing so, otherwise reap the benefits of executing on the thread pool.
Json Schema has a validation vocabulary which can be used to set constraints on json structures. OpenAPI uses Json Schemas to describe parameters and content, so wouldn’t it be nice to be able to evaluate HTTP request and response messages?
OpenAPI.Evaluation is a .NET library which can evaluate HTTP request and response messages according to an OpenAPI specification. Json schemas are evaluated using JsonSchema.NET together with the JsonSchema.Net.OpenApi vocabulary. It supports the standard HttpRequestMessage and HttpResponseMessage and comes with a DelegatingHandler, OpenApiEvaluationHandler, which can be used by HttpClient to intercept and evaluate requests going out and responses coming in according to an OpenAPI 3.1 specification. It’s also possible to manually evaluate requests and responses by traversing the parsed OpenAPI specification and feed it’s evaluators with the corresponding extracted content.
ASP.NET
OpenAPI.Evaluation.AspNet integrates OpenAPI.Evaluation with the ASP.NET request pipeline to enable server side evaluation. It comes with extension methods to evaluate HttpRequest and HttpResponse abstractions. It also supports integration via the HttpContext enabling easy access for ASP.NET request and response pipelines and controllers. A middleware is provided for simple integration and can be enabled via the application builder and service collection.
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenApiEvaluation(OpenAPI.Evaluation.Specification.OpenAPI.Parse(JsonNode.Parse(File.OpenRead("openapi.json"));
var app = builder.Build();
// Registers the middleware into the request pipeline
app.UseOpenApiEvaluation();
To evaluate directly from a request pipeline, the OpenAPI specification first needs to be loaded and registered as described above. Extension methods for HttpContext can then be used for request and response evaluation:
var requestEvaluationResult = context.EvaluateRequest();
...
var responseEvaluationResult = context.EvaluateResponse(200, responseHeaders, responseContent);
Evaluation Result
JsonSchema.NET implements the Json Schema output format which OpenAPI.Evaluation is influenced by. The OpenAPI specification doesn’t define annotations as Json Schema does, so I decided to adopt something similar. The evaluation result contains information describing each specification object traversed and what path through the specification the evaluation process took. An example produced by the default evaluation result json converter is shown belong, it uses a hierarchical output format.
Headers, path values, query strings and cookies can be described in an OpenAPI specification using a combination of instructive metadata, like styles, and schemas. It’s designed to cater for simple data structures and is complemented by content media types for more complex scenarios.
OpenAPI.Evaluation supports all the styles described in the specification, but it’s not explicitly defined how complex scenarios the specification should support, that is left to implementors to decide. In order to cater for more complex scenarios, it’s possible to define custom parsers per parameter by implementing the IParameterValueParser and register it when parsing the OpenAPI specification.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
