We’ve had a number of posts on Akka.Cluster.Sharding on this blog:

• “Introduction to Akka.Cluster.Sharding in Akka.NET”
• “Technical Overview of Akka.Cluster.Sharding in Akka.NET”

Both of those were written by Bartosz Sypytkowski, who originally contributed those features to Akka.NET. We’ve had a lot of demand for an updated introduction video to Akka.Cluster.Sharding that explores the concepts, behavior, code, and configuration of Akka.Cluster.Sharding in more detail than previously - and so I’ve produced a video doing just that: “Distributing State Reliably with Akka.Cluster.Sharding”

This is my companion blog post to go along with the video - I’m not going to cover everything in the video, but I’ll provide a high-level synopsis that explains:

• What Akka.Cluster.Sharding does;
• When you should use it; and
• How it works.

Watch “Distributing State Reliably with Akka.Cluster.Sharding” for the complete details, or read on for the high level overview.

We just finished shipping Phobos 2.5 and it’s a massive performance upgrade over previous versions of Phobos.

For those that are not aware: Phobos is our commerical OpenTelemetry add-on for Akka.NET.

This past summer we posted about Phobos 2.4’s performance being 62% faster than Phobos 2.3.1 Phobos 2.5 is 161% faster than Phobos 2.3.1 - and in this post we’re going to share the generalized .NET performance optimization technique we’ve been using to accomplish these improvements: deferred allocations.

Problem Context

Imagine you have a performance-critical hotpath in your application, such as an Akka.NET actor or an ASP.NET Controller - we ideally want to keep the latency in this critical path as low as possible in order to maximize responsiveness and per-process throughput.

But, we are also given a secondary requirement - maybe we have to add logging or OpenTelemetry tracing here for observability purposes, or maybe we have to push some data points into an internal-facing analytics / reporting system for internal stakeholders.

Implementing that secondary requirement along the critical path is going to increase our processing time at the expense of our mission-critical processing and ultimately, our end-users. Deferring allocations to outside the critical path is how we can avoid this problem.

It’s been a very long time since we’ve refreshed our Akka.NET training courses or Akka.NET support plans - so I’m pleased to announce that we are doing both effective immediately.

You can watch my video summary below or read on:

When troubleshooting performance problems in distributed systems or locally-run, high-throughput-required software I tell our users “your most severe performance problems are almost always going to be caused by flow control issues.”

My preferred batting order for troubleshooting performance issues is:

1. Improve or resolve flow control issues;
2. Eliminate wasteful I/O and round-trips; and
3. Technical improvements - improve how efficiently work is performed leveraging mechanical sympathy.

This list is ranked in the order of “most likely to have largest real-world performance impact.”

In this post we’re going to address how you can use Akka.NET actors and Akka.Streams to easily resolve some one of the most painful flow control issues: database contention and bottlenecking.

It seems like just yesterday that I wrote “Akka.NET - One Year Later”; as of November 21st, 2023, Akka.NET is now ten years old.

I don’t have my original prototype Akka code anymore; all I have is Helios - the socket library I originally created to power Akka.NET’s remoting and clustering systems, which it did from 2014 until 2017. My first commit on that project dates back to Nov 21 2013. That’s when I mark this phase of my career: the Akka.NET years.

In this post I wanted to share some lessons learned from developing and maintaining one of the most ambitious, professional grade, and independent open source projects in the .NET ecosystem for over 10 years. Some of these lessons are technical; some are more business-oriented; and some are just kind of funny. In any case, I hope you find them helpful.

Chinese version.

Last Thursday, September 7th we executed our “Akka.NET Application Management Best Practices” webinar and I’ve made the recording available on YouTube for everyone to watch for themselves.

The code sample I created for this webinar can also be found here: https://github.com/Aaronontheweb/akkadotnet-app-management-presentation.

However, I wanted to expand upon the webinar’s key points and emphasize some strategic / product management points that might not get noticed if you don’t pay close attention to the video.

If you’re interested in having any of these concepts taught to your company or team, please contact us about training options.

I’ve reviewed 100+ Akka.NET code bases at this point in my career, and I’ve reviewed stand-alone ASP.NET applications without any Akka.NET whatsoever. What many of these code bases all have in common is someone gets the bright idea to abstract over Akka.NET / ASP.NET / Entity Framework with an in-house framework that automates infrastructure decisions and enforces a one-size-fits-all design on how all domains are implemented.

This is a tremendously expensive design mistake that destroys optionality and creates more problems than it solves.

Bespoke Company Frameworks

I introduced the term “Bespoke Company Framework” (BCF) in a post on my personal blog earlier this year: “DRY Gone Bad: Bespoke Company Frameworks.”

As I mention in the presentation - BCFs attempt to standardize the way work is done inside a company’s specific domain. BCFs are actually quite helpful at solving infrastructure problems.

For instance, if you’re trying to automate manufacturing processes that have to be configured uniquely for multiple lines of products and all lines are built using the same universe of vendor drivers / components, writing a BCF that abstracts...

Phobos is our observability + monitoring library for Akka.NET and last year we launched Phobos 2.0 which moved our entire implementation onto OpenTelemetry.

One of the issues we’ve had with Phobos: many of our customers build low-latency, real-time applications and adding observability to a software system comes with a noticeable latency + throughput penalty. We were asked by a customer at the beginning of July “is there any way you can make this faster?”

Challenge accepted - and completed.

Earlier this week we shipped Phobos 2.4.0, which is a staggering 62% faster than all of our previous Phobos 2.x implementations - actually Phobos 2.4 is even faster than that for real-world applications and we’ll get to that in a moment.

This blog post is really about optimizing hot-paths for maximum OpenTelemetry tracing and metrics performance and the techniques we used in the course of developing Phobos 2.4.

Let’s dig in.

In our previous post we covered the Akka.NET v1.5 release and in particular, we focused on the changes made to the core Akka.NET modules.

In this blog post we’re going to cover the three new libraries we’ve added to Akka.NET as part of the v1.5 development effort:

As of today, Akka.NET v1.5 is now available as a stable release package on NuGet for both .NET Standard 2.0 and NET 6.0. This is a big release aimed at addressing pain points for current Akka.NET users.

We’ve published a detailed article on the Akka.NET website that describes what’s new in Akka.NET v1.5, but we wanted to capture some of the highlights here.

One of our major engineering milestones for Akka.NET v1.5 (ships on February 28th, 2023):

Make CQRS a priority in Akka.Persistence

This blog post is about an interesting engineering challenge we had to solve to accomplish this for Akka.NET v1.5: supporting hundreds of thousands or even millions of concurrent Akka.Persistence.Query projection queries all targeting a single database.

Fundamentally - there are many different facets of Akka.Persistence and Akka.Persistence.Query scalability that needed solving, but this post fixates on an acute problem users have in production with Akka.Persistence.Query today: large numbers of concurrent queries absolutely melting down production-grade database deployments.

Users reported APQ knocking their databases out with as few as 3-4 thousand streaming queries running at rates of once every 1-3 seconds - with several nodes all running similar workloads (so in aggregate, perhaps closer 10k+ queries per second.)

Beginning in Akka.NET v1.5, we have eliminated this issue and have explicitly tested it up to 100,000 queries per second targeting a dinky SQL Server 2019 database running inside a Docker container. We suspect our design can scale up to support 10s of millions of concurrent queries (although for reasons I get into later, end-users should use different approach.)

Here’s what we did.