Phobos 2.10 is here, and it’s a game-changer for anyone running Akka.NET applications in production. This release doesn’t just incrementally improve observability - it fundamentally transforms how you understand and troubleshoot actor performance in your clusters.

The headline features: accurate backpressure measurement across all actors, a bird’s-eye view of your entire Akka.NET cluster activity, detailed actor performance analysis dashboards, and the ability to easily filter /system and /user actors from each other. But here’s what makes this release special - it’s not just about the new metrics (though those are substantial). It’s about the beautiful, production-ready dashboards that make all this data instantly actionable.

We know OpenTelemetry deeply at Petabridge. We’ve built Phobos, an OpenTelemetry instrumentation plugin for Akka.NET, so we understand the low-level bits. Beyond that, we’ve been using OpenTelemetry in production for years on Sdkbin and we’ve helped over 100 customers implement OpenTelemetry configurations very similar to our own. Through all this experience, one thing has become crystal clear: the easiest, most production-ready approach to OpenTelemetry in .NET is using OTLP (OpenTelemetry Line Protocol) with a collector.

In this post, I’ll walk you through why this approach beats vendor-specific exporters every time, show you exactly how to configure it, and demonstrate the real-world benefits we’ve experienced at Petabridge. This is the companion piece to my recent YouTube video on the topic.

The Problem with Vendor-Specific Exporters

When you’re getting started with OpenTelemetry for the first time in one of your projects, you know your team uses DataDog, or New Relic, or Application Insights. So naturally, the first thing you’ll try is figuring out how to connect your application directly to that specific tool.

You end up with something that looks like this:

builder.Services .AddOpenTelemetry() .WithTracing(builder => { builder .AddHttpClientInstrumentation() .AddAspNetCoreInstrumentation() // Coupling our app to vendor-specific implementations .AddDatadogTracing() // Application code now depends on DataDog SDK .AddNewRelicTracing() // And New Relic SDK .AddAzureMonitorTracing(); // And Azure Monitor SDK }); 

And you’re going to get frustrated doing this because of:

1. Vendor Coupling: Your application code is now directly coupled to vendor-specific SDKs...

If you’re using Kafka in .NET, you’re probably writing hundreds of lines of code just to handle “what happens when my consumer crashes?” or “how do I retry failed messages?” or “what happens when I’m consuming messages too fast?”

What if I told you there was a way to handle all of that in just 5-10 lines of code?

That’s exactly what Akka.Streams.Kafka brings to the table - and it’s one of the most underrated parts of the entire Akka.NET ecosystem.

We just recently wrote about using Akka.Management to help automate Akka.Cluster formation in most runtime environments - this post deals with a similar theme: how Akka.Cluster’s built-in infrastructure deals with network failures that will inevitably occur inside any distributed application over a sufficiently long period of time.

Split Brains and Akka.Cluster

Specifically, we’re going to address how Akka.Cluster deals with split brains - a type of network failure that breaks a once-functioning cluster apart into smaller, isolated islands that can no longer communicate with each other.

Let’s dive in.

We haven’t talked that much about AI and LLM-driven development here at Petabridge, aside from a webinar we ran a year ago, but we’ve been using it heavily on our day jobs:

• Creating the visuals for “Why Learn Akka.NET?”;
• Doing the CSS and Jekyll auto-layout for Akka.NET Bootcamp 2.0;
• Transforming Incrementalist into a significantly more powerful .NET build tool;
• Radically improving Akka.IO.Tcp’s performance and stability; and
• Just this week we deployed massive performance & architecture improvements to Sdkbin - and Claude / Cursor were absolutely essential in helping us design, test, and bug-fix those.

One of the tools that’s allowed us to apply LLM-assisted coding successfully to massive code bases like Akka.NET and Sdkbin is the Model Context Protocol (MCP) - and in this post + accompanying YouTube video, we’re going explain what it is without dipping into the hyperbole you usually find on platforms like LinkedIn and X.

Akka.Cluster is a very powerful piece of software aimed at helping .NET developers build highly available, low-latency, distributed software. At its core, Akka.Cluster is about building peer-to-peer networks—that’s what a “cluster” actually is: a peer-to-peer network that runs in a server-side environment controlled by a single operator.

What Clusters Need

This is a subject for another blog post, but what makes peer-to-peer networks a superior choice over client-server networks for high availability are the following:

1. Horizontally scalable, because the “source of truth” is decentralized and distributed to the endpoints of the network (these are your actors running in each process) rather than centralized in a single location;
2. Fault tolerant and resilient - having the source of truth decentralized and distributed also means that no single node in the network is so crucial that its disappearance is going to render the system unavailable; and
3. Still supports inter-dependent services - you can still have multiple services with completely different purposes and code cooperating together inside a peer-to-peer network. This is what Akka.Cluster roles are for.

In order to build a peer-to-peer network, you need two primary ingredients:

1. Topology-awareness - database-driven CRUD applications never need to do this. The load-balancer is aware of where the web servers are and the web servers are aware of where the database is, but that’s pretty much it. In a real peer-to-peer network, all applications need to know about each other and need to communicate with each other directly. These are what Akka.Remote (communication) and Akka.Cluster (topology) provide.
2. Initial formation - there must be a strategy for processes to form a new peer-to-peer network or to join an existing one.

In this blog post, we’ll be discussing item number 2—how to make the formation and joining of Akka.Cluster networks more reliable,...

We blog a ton about Akka.NET here, but Petabridge really is a .NET open source company. Throughout our work on Akka.NET we create many other OSS tools in support of much more general purpose .NET use cases, such as the .NET Runtime Dashboards we touted on our YouTube channel a couple of weeks ago or, back in the day, tools like NBench¹.

Today though we’re writing about a brand new tool we’ve been working on for the past several months: Incrementalist v1.0, a command-line tool that leverages git and Roslyn solution analysis to drastically reduce build times for large .NET solutions and monorepos.

We’ve been using older versions of Incrementalist in production inside the Akka.NET build pipeline since 2019 - it cuts our average pull request build time down from about 1 hour and 15 minutes to ~15 minutes. Those older versions of Incrementalist just spat out the smallest possible build graphs as a .csv file - it was up to you to parse it and use the data accordingly.

Incrementalist v1.0 is a totally different animal: it runs the dotnet commands for you.

For example, from Akka.NET’s live build system:

dotnet incrementalist run --config .incrementalist/testsOnly.json -- test -c Release --no-build --framework net8.0 --logger:trx --results-directory TestResults 

This call:

1. Invokes the run verb - which means we’re going to execute a dotnet command against all of the matching projects (C# or F#);
2. Uses the
   Read more