Over the past several months the Akka.NET team has had reports of the following Exception popping up unexpectedly throughout many of our plugins and end-user applications that use the Akka.Streams¹ SelectAsync stage - such as Akka.Streams.Kafka and Akka.Persistence.Sql:

That error message seems simple enough - it comes from here inside GraphStage.cs:

[InternalApi] public void InternalOnDownstreamFinish(Exception cause) { try { if (cause == null) throw new ArgumentException("Cancellation cause must not be null", nameof(cause)); 

In Akka.Streams parlance, a stream gets cancelled when an unhandled Exception is thrown and that error should be propagated all the way down to this GraphStage.InternalOnDownstreamFinish method so we can log why the stream is being cancelled / terminated.

Here’s the mystery - this is the code that “threw” the Exception inside Akka.Persistence.Sql for instance:

.SelectAsync( JournalConfig.DaoConfig.Parallelism, async promisesAndRows => { try { await WriteJournalRows(promisesAndRows.Rows); foreach (var taskCompletionSource in promisesAndRows.Tcs) taskCompletionSource.TrySetResult(NotUsed.Instance); } catch (Exception e) { foreach (var taskCompletionSource in promisesAndRows.Tcs) taskCompletionSource.TrySetException(e); } return NotUsed.Instance; }) 

Akka.NET Bootcamp has a special place in my heart because it’s really “the thing” that launched Petabridge as a business 10 years ago.

However, the original Akka.NET Bootcamp should have really been replaced years ago as both Akka.NET and the .NET ecosystems had changed tremendously since it was originally written.

So on that long-overdue note, it’s my pleasure to announce that the first two units of Akka.NET Bootcamp 2.0 are now live and available for immediate consumption!

Here’s what’s new in Bootcamp 2.0:

When we first started Petabridge in January 2015, I never imagined that we’d be at this for 2-3 years, let alone 10! But here we are. It’s been an amazing journey and our privilege to work with so many amazing customers and software developers on building some of the world’s most important software on top of Akka.NET.

I already published a small post on my personal website about the business side of running Petabridge over this span of time, but for our subscribers and Akka.NET users here at Petabridge, I thought it’d be more appropriate to share some of the engineering lessons we’ve learned over the course of 10 years from all of our experiences here.

Please enjoy “10 Most Important Engineering Lessons Learned from 10 Years of Petabridge”.

The Lessons

We’re kicking off 2025 with a fresh look at Akka.NET and what we have planned for our next big release: Akka.NET v1.6!

Akka.NET v1.6: a Performance-Focused Release

As it says in our promo image here - Akka.NET v1.6 is a performance-focused release. To that end, we’re addressing the following three major changes:

One of the most frequent pieces of feedback we get from developers who are new to Akka.NET is that the “learning curve” is high. I want to explore that today and why I think this isn’t actually true, but what developers are saying is that working with actors is unfamiliar, not difficult.

I’d also argue that really, really bad pre-existing habits many software developers have, such as frameworkism, become a hundred-fold more destructive when you introduce unfamiliar paradigms, such as stateful programming with actors, to the mix. You could substitute “stateful programming with actors” with “using NoSQL” or “going cloud native” and that sentence would be equally true.

Let’s assume you’ve already decided that learning actors is worth your time and now you want to know: “how can I begin to learn how to work with actors for my own purposes?” That’s what this post is about.

A really, really common question Akka.NET beginners have within the first couple of hours of looking at Akka.NET: “how is the IActorRef.Tell method asynchronous if it’s a void method? Shouldn’t it return a Task I can await on?”

Today I’m going to clarify Akka.NET’s API design and explain, in terms of Akka.NET’s behavior and benefits to users, why we don’t do this.

Even if you’re a seasoned Akka.NET user, you might find some value in this post.

We put together a new YouTube video on our Petabridge channel (which you should subscribe to) yesterday, “Akka.NET Actors’ Hidden Super Power: Switchable Behaviors” - all about one of my favorite features in Akka.NET: the Become method for swapping actor message-processing behavior dynamically at runtime.

In addition to the video, I wanted to expand on why behavior-switching is so powerful and how you can use it to transform inherently complex domain problems into something that is approachable, understandable, and expressable with a very small amount of purpose-built code.

The month of July, 2024 was pretty rough for me and the Petabridge crew from a workload standpoint. Our office got hit with a hurricane. My entire family and I were out sick with COVID in the middle of being hit by a hurricane. Fun times.

But the worst of it was a pair of truly nasty Akka.NET issues that both turned out to both be the same bug:

• Akka.Cluster.Sharding: duplicate shards / entities
• Akka.Cluster.Tools.Singleton: singleton moves earlier than expected - as soon as new node joins

I ended up spending about 70+ hours myself investigating and eventually solving this bug starting from June 27th onwards.

If you’re an Akka.NET user and don’t want to read any further, just upgrade to Akka.NET v1.5.27 or later - the bugs are fixed in those versions of the software. Every other earlier version of the software, dating back to when Akka.Cluster.Tools was first introduced in 2015, has this bug. Please upgrade now.

Otherwise, read on.

It’s been a little while since we covered Akka.NET 101 on our blog, so we decided it was time to visit one of the most fundamental and significant matters users need to understand about Akka.NET in order to leverage it properly: how actor message processing actually works.

We first introduced Akka.Hosting a couple of years ago and released it to market one of the major pillars of Akka.NET v1.5. Without any exaggeration, it is the single best usability improvement we have ever made to Akka.NET and we are furiously rewriting all of the official Akka.NET documentation and training courses to prioritize it.

In that same vein, we’re going to introduce how you can use Akka.Hosting to easily integration Akka.NET with:

• Microsoft.Extensions.DependencyInjection - injecting things into actors and actors into things;
• ASP.NET Core / SignalR / gRPC / etc - anything that can be injected via MSFT.EXT.DI really; and
• Running Akka.NET in headless IHostedServices for things like stand-alone Windows Services.

Read on.