The Easiest Way to Do OpenTelemetry in .NET: OTLP + Collector

Decouple your observability configuration from your application code with OTLP and collectors

19 minutes to read

We know OpenTelemetry deeply at Petabridge. We’ve built Phobos, an OpenTelemetry instrumentation plugin for Akka.NET, so we understand the low-level bits. Beyond that, we’ve been using OpenTelemetry in production for years on Sdkbin and we’ve helped over 100 customers implement OpenTelemetry configurations very similar to our own. Through all this experience, one thing has become crystal clear: the easiest, most production-ready approach to OpenTelemetry in .NET is using OTLP (OpenTelemetry Line Protocol) with a collector.

In this post, I’ll walk you through why this approach beats vendor-specific exporters every time, show you exactly how to configure it, and demonstrate the real-world benefits we’ve experienced at Petabridge. This is the companion piece to my recent YouTube video on the topic.

The Problem with Vendor-Specific Exporters

When you’re getting started with OpenTelemetry for the first time in one of your projects, you know your team uses DataDog, or New Relic, or Application Insights. So naturally, the first thing you’ll try is figuring out how to connect your application directly to that specific tool.

You end up with something that looks like this:

builder.Services .AddOpenTelemetry() .WithTracing(builder => { builder .AddHttpClientInstrumentation() .AddAspNetCoreInstrumentation() // Coupling our app to vendor-specific implementations .AddDatadogTracing() // Application code now depends on DataDog SDK .AddNewRelicTracing() // And New Relic SDK .AddAzureMonitorTracing(); // And Azure Monitor SDK }); 

And you’re going to get frustrated doing this because of:

  1. Vendor Coupling: Your application code is now directly coupled to vendor-specific SDKs...

Why Akka.Streams.Kafka is the Best Kafka Client for .NET

Stop writing hundreds of lines of error handling code - there's a better way.

18 minutes to read

If you’re using Kafka in .NET, you’re probably writing hundreds of lines of code just to handle “what happens when my consumer crashes?” or “how do I retry failed messages?” or “what happens when I’m consuming messages too fast?”

What if I told you there was a way to handle all of that in just 5-10 lines of code?

That’s exactly what Akka.Streams.Kafka brings to the table - and it’s one of the most underrated parts of the entire Akka.NET ecosystem.

Self-Healing Akka.NET Clusters

How Split Brain Resolvers ensure consistency even during network turbulence.

13 minutes to read

We just recently wrote about using Akka.Management to help automate Akka.Cluster formation in most runtime environments - this post deals with a similar theme: how Akka.Cluster’s built-in infrastructure deals with network failures that will inevitably occur inside any distributed application over a sufficiently long period of time.

Split Brains and Akka.Cluster

Specifically, we’re going to address how Akka.Cluster deals with split brains - a type of network failure that breaks a once-functioning cluster apart into smaller, isolated islands that can no longer communicate with each other.

Let’s dive in.

Model Context Protocol, Without the Hype

MCP is very useful, but it's not curing cancer. Here's why you should use it.

10 minutes to read

We haven’t talked that much about AI and LLM-driven development here at Petabridge, aside from a webinar we ran a year ago, but we’ve been using it heavily on our day jobs:

One of the tools that’s allowed us to apply LLM-assisted coding successfully to massive code bases like Akka.NET and Sdkbin is the Model Context Protocol (MCP) - and in this post + accompanying YouTube video, we’re going explain what it is without dipping into the hyperbole you usually find on platforms like LinkedIn and X.

Form Akka.NET Clusters Dynamically with Akka.Management and Akka.Discovery

A safer, superior choice to using seed nodes with Akka.Cluster

15 minutes to read

Akka.Cluster is a very powerful piece of software aimed at helping .NET developers build highly available, low-latency, distributed software. At its core, Akka.Cluster is about building peer-to-peer networks—that’s what a “cluster” actually is: a peer-to-peer network that runs in a server-side environment controlled by a single operator.

What Clusters Need

This is a subject for another blog post, but what makes peer-to-peer networks a superior choice over client-server networks for high availability are the following:

  1. Horizontally scalable, because the “source of truth” is decentralized and distributed to the endpoints of the network (these are your actors running in each process) rather than centralized in a single location;
  2. Fault tolerant and resilient - having the source of truth decentralized and distributed also means that no single node in the network is so crucial that its disappearance is going to render the system unavailable; and
  3. Still supports inter-dependent services - you can still have multiple services with completely different purposes and code cooperating together inside a peer-to-peer network. This is what Akka.Cluster roles are for.

In order to build a peer-to-peer network, you need two primary ingredients:

  1. Topology-awareness - database-driven CRUD applications never need to do this. The load-balancer is aware of where the web servers are and the web servers are aware of where the database is, but that’s pretty much it. In a real peer-to-peer network, all applications need to know about each other and need to communicate with each other directly. These are what Akka.Remote (communication) and Akka.Cluster (topology) provide.
  2. Initial formation - there must be a strategy for processes to form a new peer-to-peer network or to join an existing one.

In this blog post, we’ll be discussing item number 2—how to make the formation and joining of Akka.Cluster networks more reliable,...

Introducing Incrementalist, an Incremental .NET Build Tool for Large Solutions

Reduced Akka.NET's average build time from ~1.25hrs to 15 minutes.

12 minutes to read

We blog a ton about Akka.NET here, but Petabridge really is a .NET open source company. Throughout our work on Akka.NET we create many other OSS tools in support of much more general purpose .NET use cases, such as the .NET Runtime Dashboards we touted on our YouTube channel a couple of weeks ago or, back in the day, tools like NBench1.

Today though we’re writing about a brand new tool we’ve been working on for the past several months: Incrementalist v1.0, a command-line tool that leverages git and Roslyn solution analysis to drastically reduce build times for large .NET solutions and monorepos.

Incrementalist - a Git-based incremental build and testing platform for .NET and .NET Core.

We’ve been using older versions of Incrementalist in production inside the Akka.NET build pipeline since 2019 - it cuts our average pull request build time down from about 1 hour and 15 minutes to ~15 minutes. Those older versions of Incrementalist just spat out the smallest possible build graphs as a .csv file - it was up to you to parse it and use the data accordingly.

Incrementalist v1.0 is a totally different animal: it runs the dotnet commands for you.

For example, from Akka.NET’s live build system:

dotnet incrementalist run --config .incrementalist/testsOnly.json -- test -c Release --no-build --framework net8.0 --logger:trx --results-directory TestResults 

This call:

  1. Invokes the run verb - which means we’re going to execute a dotnet command against all of the matching projects (C# or F#);
  2. Uses the

Retiring Akka.Persistence.SqlServer, Postgres, Sqlite

Akka.Persistence.Sql is the new flavor moving forward.

8 minutes to read

It was just about 10 years ago when we shipped Akka.NET v1.0.2, the release where we first introduced betas of some of our most popular Akka.Persistence plugins: Akka.Persistence.Postgres, Akka.Persistence.SqlServer, and Akka.Persistence.Sqlite.

All of these plugins were based off of a shared ADO.NET Akka.Persistence architecture called Akka.Persistence.Sql.Common and this architecture has served both us and our users / customers well over the past 10 years, somewhere to the tune of 1.6 million installations!

But, as of Akka.NET v1.5.40 these plugins are all deprecated in favor of Akka.Persistence.Sql - the successor plugin for all SQL RDBMS we introduced along with the release of Akka.NET v1.5.

Akka.Persistence.SqlServer, Postgres, and Sqlite have all been retired in favor of Akka.Persistence.Sql going forward

In the next sections we’ll explain our decision along with showing you our migration guide for moving off any of the affected Akka.Persistence.Sql.Common plugins and onto Akka.Persistence.Sql.

.NET Heisenbug Mystery Theater: How Did an Exception Escape its Catch Block?

A painful lesson on atomicity and the assignment of structs.

21 minutes to read

Over the past several months the Akka.NET team has had reports of the following Exception popping up unexpectedly throughout many of our plugins and end-user applications that use the Akka.Streams1 SelectAsync stage - such as Akka.Streams.Kafka and Akka.Persistence.Sql:

ArgumentException stemming from the cancellation of an Akka.NET stream without a cause

That error message seems simple enough - it comes from here inside GraphStage.cs:

[InternalApi] public void InternalOnDownstreamFinish(Exception cause) { try { if (cause == null) throw new ArgumentException("Cancellation cause must not be null", nameof(cause)); 

In Akka.Streams parlance, a stream gets cancelled when an unhandled Exception is thrown and that error should be propagated all the way down to this GraphStage.InternalOnDownstreamFinish method so we can log why the stream is being cancelled / terminated.

Here’s the mystery - this is the code that “threw” the Exception inside Akka.Persistence.Sql for instance:

.SelectAsync( JournalConfig.DaoConfig.Parallelism, async promisesAndRows => { try { await WriteJournalRows(promisesAndRows.Rows); foreach (var taskCompletionSource in promisesAndRows.Tcs) taskCompletionSource.TrySetResult(NotUsed.Instance); } catch (Exception e) { foreach (var taskCompletionSource in promisesAndRows.Tcs) taskCompletionSource.TrySetException(e); } return NotUsed.Instance; }) 
...

Introducing Akka.NET Bootcamp 2.0

Modern Akka.NET Best Practices in Free, Self-Paced Lessons

2 minutes to read

Akka.NET Bootcamp has a special place in my heart because it’s really “the thing” that launched Petabridge as a business 10 years ago.

However, the original Akka.NET Bootcamp should have really been replaced years ago as both Akka.NET and the .NET ecosystems had changed tremendously since it was originally written.

So on that long-overdue note, it’s my pleasure to announce that the first two units of Akka.NET Bootcamp 2.0 are now live and available for immediate consumption!

Akka.NET Bootcamp Logo

Here’s what’s new in Bootcamp 2.0:

10 Important Engineering Lessons Learned Over 10 Years of Petabridge

Petabridge turns 10 in January 2025!

1 minute to read

When we first started Petabridge in January 2015, I never imagined that we’d be at this for 2-3 years, let alone 10! But here we are. It’s been an amazing journey and our privilege to work with so many amazing customers and software developers on building some of the world’s most important software on top of Akka.NET.

I already published a small post on my personal website about the business side of running Petabridge over this span of time, but for our subscribers and Akka.NET users here at Petabridge, I thought it’d be more appropriate to share some of the engineering lessons we’ve learned over the course of 10 years from all of our experiences here.

Please enjoy “10 Most Important Engineering Lessons Learned from 10 Years of Petabridge”.

The Lessons