In addition to the video, I wanted to expand on why behavior-switching is so powerful and how you can use it to transform inherently complex domain problems into something that is approachable, understandable, and expressable with a very small amount of purpose-built code.
A 70+ hour journey into fixing a ~10 year old Akka.NET bug.
The month of July, 2024 was pretty rough for me and the Petabridge crew from a workload standpoint. Our office got hit with a hurricane. My entire family and I were out sick with COVID in the middle of being hit by a hurricane. Fun times.
But the worst of it was a pair of truly nasty Akka.NET issues that both turned out to both be the same bug:
I ended up spending about 70+ hours myself investigating and eventually solving this bug starting from June 27th onwards.
If you’re an Akka.NET user and don’t want to read any further, just upgrade to Akka.NET v1.5.27 or later - the bugs are fixed in those versions of the software. Every other earlier version of the software, dating back to when Akka.Cluster.Tools was first introduced in 2015, has this bug. Please upgrade now.
The fundamentals of Akka.NET actor construction and message processing.
It’s been a little while since we covered Akka.NET 101 on our blog, so we decided it was time to visit one of the most fundamental and significant matters users need to understand about Akka.NET in order to leverage it properly: how actor message processing actually works.
Both of those were written by Bartosz Sypytkowski, who originally contributed those features to Akka.NET. We’ve had a lot of demand for an updated introduction video to Akka.Cluster.Sharding that explores the concepts, behavior, code, and configuration of Akka.Cluster.Sharding in more detail than previously - and so I’ve produced a video doing just that: “Distributing State Reliably with Akka.Cluster.Sharding”
This is my companion blog post to go along with the video - I’m not going to cover everything in the video, but I’ll provide a high-level synopsis that explains:
This past summer we posted about Phobos 2.4’s performance being 62% faster than Phobos 2.3.1 Phobos 2.5 is 161% faster than Phobos 2.3.1 - and in this post we’re going to share the generalized .NET performance optimization technique we’ve been using to accomplish these improvements: deferred allocations.
Problem Context
Imagine you have a performance-critical hotpath in your application, such as an Akka.NET actor or an ASP.NET Controller - we ideally want to keep the latency in this critical path as low as possible in order to maximize responsiveness and per-process throughput.
But, we are also given a secondary requirement - maybe we have to add logging or OpenTelemetry tracing here for observability purposes, or maybe we have to push some data points into an internal-facing analytics / reporting system for internal stakeholders.
Implementing that secondary requirement along the critical path is going to increase our processing time at the expense of our mission-critical processing and ultimately, our end-users. Deferring allocations to outside the critical path is how we can avoid this problem.
Alleviate strain on production systems with in-process Akka.NET streams.
When troubleshooting performance problems in distributed systems or locally-run, high-throughput-required software I tell our users “your most severe performance problems are almost always going to be caused by flow control issues.”
My preferred batting order for troubleshooting performance issues is:
Improve or resolve flow control issues;
Eliminate wasteful I/O and round-trips; and
Technical improvements - improve how efficiently work is performed leveraging mechanical sympathy.
This list is ranked in the order of “most likely to have largest real-world performance impact.”
In this post we’re going to address how you can use Akka.NET actors and Akka.Streams to easily resolve some one of the most painful flow control issues: database contention and bottlenecking.
Lessons learned from working on Akka.NET over the years.
It seems like just yesterday that I wrote “Akka.NET - One Year Later”; as of November 21st, 2023, Akka.NET is now ten years old.
I don’t have my original prototype Akka code anymore; all I have is Helios - the socket library I originally created to power Akka.NET’s remoting and clustering systems, which it did from 2014 until 2017. My first commit on that project dates back to Nov 21 2013. That’s when I mark this phase of my career: the Akka.NET years.
In this post I wanted to share some lessons learned from developing and maintaining one of the most ambitious, professional grade, and independent open source projects in the .NET ecosystem for over 10 years. Some of these lessons are technical; some are more business-oriented; and some are just kind of funny. In any case, I hope you find them helpful.
Last Thursday, September 7th we executed our “Akka.NET Application Management Best Practices” webinar and I’ve made the recording available on YouTube for everyone to watch for themselves.
However, I wanted to expand upon the webinar’s key points and emphasize some strategic / product management points that might not get noticed if you don’t pay close attention to the video.
I’ve reviewed 100+ Akka.NET code bases at this point in my career, and I’ve reviewed stand-alone ASP.NET applications without any Akka.NET whatsoever. What many of these code bases all have in common is someone gets the bright idea to abstract over Akka.NET / ASP.NET / Entity Framework with an in-house framework that automates infrastructure decisions and enforces a one-size-fits-all design on how all domains are implemented.
This is a tremendously expensive design mistake that destroys optionality and creates more problems than it solves.
As I mention in the presentation - BCFs attempt to standardize the way work is done inside a company’s specific domain. BCFs are actually quite helpful at solving infrastructure problems.
For instance, if you’re trying to automate manufacturing processes that have to be configured uniquely for multiple lines of products and all lines are built using the same universe of vendor drivers / components, writing a BCF that abstracts...
