Author’s Note: Petabridge turned 8 years old this month and over the years I’ve only written a couple of articles about our internal processes for developing software and running an open source software business. In late 2021 we began using OKRs - “Objectives and Key Results” - as a general management system for setting quarterly goals and allocating accountabilities across the organization.

OKRs took some adjusting to, but have worked out really well for us overall - in combination with using Notion to record daily plans, document critical procedures, write technical specifications, and track progress against our key results each week.

Tracking these OKRs and our daily work plans made it quite easy for me to summarize everything our team accomplished in each area of our business in 2022 - and I wanted to take advantage of that and show everyone the impact their work had in the previous year.

So to kick off 2023 I showed a summary of all of our key results in each area of our business to our team members and we spent a day going through them - what follows below are the sections about Akka.NET, what we accomplished last year, and what’s next in store for Akka.NET.

Akka.NET Releases in 2022

So how did we do on Akka.NET releases in 2022?

In 2022 we shipped 86 releases of all projects combined, which means we shipped a new NuGet release every ~4 days on average during the year.

These releases included the entirety of the Akka.Hosting development effort, the introduction of major Akka.NET v1.5 features (i.e. the async TestKit, sharding rewrite), the professionalization of Akka.Management, and several important additions to Akka.NET itself (i.e. full IAsyncEnumerable support in Akka.Streams, GetEntityLocation querying, and more.)

We set a goal early on in the year of shipping more frequently - a new Akka.NET release once every 2-3 weeks in order to make sure that bug fixes, performance improvements, and new features were delivered to end-users more frequently.

Akka.NET v1.4

We made numerous fixes, improvements, and feature additions to Akka.NET v1.4 over the course of 2022.

Performance

Performance is a crucial feature of Akka.NET and we put a considerable amount of effort into improving memory allocation, network throughput, GC reduction, and addressing sources of latency in 2022.

All benchmark results are on .NET 6.0.11 with Dynamic PGO disabled.

Akka.NET v1.4.31 (Dec, 2021)

RemotePingPong Results

OSVersion:                         Microsoft Windows NT 10.0.19044.0       
ProcessorCount:                    16                                      
ClockSpeed:                        0 MHZ                                   
Actor Count:                       32                                      
Messages sent/received per client: 200000  (2e5)                           
Is Server GC:                      True                                    
Thread count:                      112                                     
                                                                           
Num clients, Total [msg], Msgs/sec, Total [ms], Start Threads, End Threads 
         1,  200000,    126985,    1575.42,           112,             141 
         5, 1000000,    291037,    3436.50,           150,             162 
        10, 2000000,    329327,    6073.48,           170,             170 
        15, 3000000,    319387,    9393.55,           178,             163 
        20, 4000000,    325840,   12276.48,           171,             154 
        25, 5000000,    324634,   15402.07,           162,             141 
        30, 6000000,    320565,   18717.30,           150,             139

PingPong Results

Warming up...
OSVersion:              Microsoft Windows NT 10.0.19044.0
ProcessorCount:         16
ClockSpeed:             0 MHZ
Actor Count:            32
Messages sent/received: 30000000  (3e7)
Is Server GC:           True
Thread count:           34

ActorBase    first start time:  9.97 ms
ReceiveActor first start time: 39.29 ms

            ActorBase                          ReceiveActor
Throughput, Msgs/sec, Start [ms], Total [ms],  Msgs/sec, Start [ms], Total [ms]
         1,  9146000,     439.28,    3719.59,   9430000,     476.97,    3658.37
         5, 20547000,     425.65,    1885.77,  21413000,     457.89,    1859.73
        10, 29182000,     301.16,    1329.89,  29041000,     347.09,    1380.11
        15, 34762000,     490.96,    1354.11,  32715000,     346.19,    1263.37
        20, 38961000,     430.07,    1200.13,  38610000,     507.01,    1284.94
        30, 41899000,     503.65,    1220.39,  44444000,     317.22,     992.76
        40, 40816000,     254.19,     989.65,  45941000,     474.79,    1128.06
        50, 49504000,     254.16,     860.66,  47543000,     127.20,     759.04
        60, 49342000,     474.90,    1083.44,  40760000,     315.08,    1051.45
        70, 47468000,     461.00,    1093.73,  49916000,     222.17,     824.15
        80, 50167000,     222.05,     820.15,  49099000,     409.91,    1021.09
        90, 46583000,     507.97,    1152.97,  49019000,     508.99,    1121.61
       100, 49423000,     476.98,    1084.76,  51194000,     460.96,    1047.87
       200, 49833000,     437.85,    1040.69,  50505000,     445.10,    1039.87
       300, 45180000,     462.04,    1126.46,  48076000,     477.93,    1102.62
       400, 49668000,     222.56,     826.70,  56818000,     508.04,    1036.44
       500, 57142000,     507.04,    1032.64,  52910000,     506.01,    1073.55
       600, 53285000,     492.67,    1056.42,  42372000,     492.06,    1200.60
       700, 41782000,     254.97,     973.57,  46439000,     507.98,    1154.34
       800, 45045000,     428.03,    1094.54,  49833000,     488.04,    1090.34
       900, 48309000,     506.99,    1128.15,  56497000,     505.89,    1037.65

Akka.NET v1.4.47 (Dec, 2022)

RemotePingPong Results

OSVersion:                         Microsoft Windows NT 10.0.19044.0      
ProcessorCount:                    16                                     
ClockSpeed:                        0 MHZ                                  
Actor Count:                       32                                     
Messages sent/received per client: 200000  (2e5)                          
Is Server GC:                      True                                   
Thread count:                      112                                    
                                                                          
Num clients, Total [msg], Msgs/sec, Total [ms], Start Threads, End Threads
         1,  200000,    164610,    1215.22,           112,             142
         5, 1000000,    315956,    3165.93,           150,             161
        10, 2000000,    328624,    6086.03,           169,             169
        15, 3000000,    332337,    9027.82,           177,             160
        20, 4000000,    329870,   12126.67,           169,             153
        25, 5000000,    334516,   14947.26,           161,             143
        30, 6000000,    327583,   18316.36,           151,             139

PingPong Results

OSVersion:                         Microsoft Windows NT 10.0.19044.0      
ProcessorCount:                    16                                     
ClockSpeed:                        0 MHZ                                  
Actor Count:                       32                                     
Messages sent/received per client: 200000  (2e5)                          
Is Server GC:                      True                                   
Thread count:                      112                                    
                                                                          
Num clients, Total [msg], Msgs/sec, Total [ms], Start Threads, End Threads
         1,  200000,    164610,    1215.22,           112,             142
         5, 1000000,    315956,    3165.93,           150,             161
        10, 2000000,    328624,    6086.03,           169,             169
        15, 3000000,    332337,    9027.82,           177,             160
        20, 4000000,    329870,   12126.67,           169,             153
        25, 5000000,    334516,   14947.26,           161,             143
        30, 6000000,    327583,   18316.36,           151,             139

Performance Summary

The overall performance of Akka.NET v1.4.47’s in-memory messaging is 8% faster for ActorBase, 3% faster for ReceiveActor. As for Akka.Remote - v1.4.47’s throughput is 3% faster on average than Akka.NET v1.4.31. While I don’t have quantifiable numbers for this, Akka.NET v1.4.47 also uses significantly fewer allocations than 1.4.31 and earlier.

So what’s the big takeaway from all of this hard performance optimization work and how it relates to our performance goals with Akka.NET?

The “low hanging fruit” for micro optimizations has largely been picked clean since the start of the v1.4 effort in 2020 (1.4.47 is ~50% faster than 1.4.0) - the return on investment of engineering labor simply isn’t there for the parts of Akka.NET that have received a lot of performance testing over the years (core, Akka.Remote;) there are definitely still some gains left on the table in areas that have been more relatively underserved, such as DData, Akka.Streams, and Akka.Persistence.

So what are we to do about improving Akka.NET’s performance in the future? As I’ve often said in my public talks on performance - flow control is almost always the dominant factor in the performance of complex software, not mechanical sympathy and micro-optimizations - the latter of which are effective only when major flow control problems are alleviated.

In layman’s terms, this means we need to take bigger bets on bigger structural changes within Akka.NET’s core constructs in order to improve performance going forward: rewriting dispatchers, targeting newer versions of .NET, improving the serialization system, and most importantly: the completion of Artery or some other similar multi-plexing network architecture that avoids the head-of-line blocking issues that Akka.Remote currently has today.

Major Features, Improvements, and Bug Fixes

In terms of Akka.NET v1.4, what were the biggest features we shipped to end-users inside the core Akka.NET package itself?

• Full IAsyncEnumerable support for Akka.Streams - this is a major improvement that facilitated a lot of the designs and creations we’ve used in a new product that has yet to be announced.
• Fixed downstream error propagation in Akka.Streams (so you can actually get access to downstream errors from an upstream stage.)
• Made single-threaded startup with the ChannelScheduler possible - I still believe there are some lingering issues with this but it was literally impossible to get this to work properly before.
• Fixed key memory leaks inside the EventStream and CoordinatedShutdown - these had been problems for several years.
• Fixed a major “deserialization Exception escalation” issue inside Akka.Remote that could cause disassociations to occur.
• Added built-in actor telemetry to core Akka.NET.
• Improved Akka.Cluster.Metrics collection and accuracy, so this library is now fully usable on .NET Standard 2.0.

Overall, our set of changes to Akka.NET’s releases over 2022 reflect a high degree of support responsiveness to issues reported by paying customers and OSS users - most of our major feature development has been concentrated on the Akka.NET v1.5.

Akka.NET Website, Build System, and Project Templates

Among the other large projects we undertook during the early part of 2022, prior to making Akka.NET v1.5 our priority, was redesigning https://getakka.net/ to use its own unique DocFx theme and hardening our build system for documentation so we’d never run into broken links again.

Before

Akka.NET website using mostly the default DocFx theme.

After

https://getakka.net/ after Akka.NET v1.4.35 shipped.

These changes included:

• Adding spell-checking and markdown linting to the website’s build process;
• Fixing every single build warning raised by DocFx (there were 500-600 of them);
• Adding a new build step to fail any PR that triggered a build warning - so we could catch link-breaking and / or removing a reference code sample when it happened; and
• Hiring a designer to create a custom DocFx theme, hiring a contractor to implement it, and deploying it to live to the website in March, 2022.

NUKE Build and dotnet new Project Templates

One of the other major projects we embarked on early this year was the restructuring all of our dotnet new templates to use a simpler, friendlier build system:

1. No dependency on Mono for Linux / OS X builds;
2. Prefer GitHub Actions over AzDo; and
3. Runs in C# instead of F#.

For this purpose we chose NUKE build and updated all of https://github.com/petabridge/petabridge-dotnet-new - and we even used these new templates to power repositories such as https://github.com/akkadotnet/Akka.Hosting. The new templates have been a success and have greatly simplified the tooling needed to run new OSS libraries and commercial projects.

Akka.NET v1.5

We had a very productive year for Akka.NET v1.5, although we have not completely delivered all of it yet:

• Created, designed, and RTM’d Akka.Hosting - a significant re-imaging of Akka.NET’s configuration, DI, hosting, and logging experience that ties us in closer to the greater .NET ecosystem. Akka.Hosting also addresses some of the biggest pain points associated with Akka.NET: HOCON and cluster lifecycle management.
• Rewrote the Akka.TestKit to become fully asynchronous - this will help speed up execution time of the test suite as well as decrease the number of race conditions caused by thread blocking.
• Restructured all of Akka.Cluster.Sharding in order to make the system more scalable and performant.
• Rebuilt Akka.Management from the ground up, including:
  • Adding Azure Service Discovery and Coordination;
  • Standardizing our ASP.NET hosting model; and
  • Making Akka.Management easy to use on top of Akka.Hosting.
• Rewrote Akka.HealthCheck to support Microsoft’s IHealthCheck standard.
• Added .NET 6 dual targeting to Akka.NET core and took advantage of new threading APIs to greatly accelerate in-memory processing rates.
• Added extensive backwards compatibility testing for Akka.Persistence.Linq2Db and are moving forward with plans to make it safe and easy for users to migrate from Akka.Persistence.Sql.Common implementations to it with full backwards compatibility.

Per our Akka.NET v1.5 GitHub Project - we’re nearing the end of the line and we should be able to ship in early 2023.

Code Samples and Tutorials

In 2022 we rewrote or created many major code samples in order to take advantage of Akka.Hosting and other best practices:

• https://github.com/petabridge/akkadotnet-code-samples
• https://github.com/petabridge/AkkaDotNet.LargeNetworkTests
• https://github.com/petabridge/kinesis-sample
• https://github.com/petabridge/azure-app-service-akkadotnet
• https://github.com/petabridge/azure-container-app-akkadotnet

We didn’t touch Akka.NET Bootcamp or our clustering tutorial much - those will need to be revisited and updated to incorporate new practices such as Akka.Hosting in 2023.

Akka.NET Community Standups

One of the major marketing undertakings for 2022 was the restoration of our Akka.NET Community Standups, which we launched around March of 2022.

In all, we successfully produced 10 Community Standups, 7 of which featured a third party Show and Tell presentations about the speakers’ production use of Akka.NET.

This was a huge get for the Akka.NET community - in the span of one year we produced more testimonials from users than we had done in all of the years prior. Sharing begets more sharing - the standups have really been a great opportunity for the sharing of knowledge between Akka.NET users.

Akka.NET in 2023

So what’s coming up in 2023 for Akka.NET?

1. Splitting Akka.NET v1.5 into two releases: 1.5 and 1.6.
   • 1.5 will be ready to ship before the end of February, 2023.
   • 1.6 will contain some of the key performance changes - Artery, new serialization system, and dispatcher upgrades. No timeframe for that release yet.
   • We want to get 1.5 out the door now with the key upgrades that have already been tested and validated, such as the Akka.Cluster.Sharding improvements and CQRS support for Akka.Persistence. Hence the split: don’t block ready features because of yet-to-be-developed ones.
2. Maker bigger bets on core architecture changes - as we saw with the engineering work on Akka.NET v1.4, clever engineering improvements can only get us so far. We have to make bigger, more aggressive bets on architectural changes to achieve more in 2023.
3. Sharpen the saw - revisit all of our previous courses, samples, and tutorials and update them to use modern Akka.NET best practices such as Akka.Hosting and Akka.Management. We’ve already updated Cluster.WebCrawler to do this, for instance. We should do the same to Akka.NET Bootcamp, our training courses, the official website documentation, and more.

If you want to hear more about our plans for Akka.NET v1.5 and v1.6, we covered this in detail during our January, 2023 Akka.NET Community Standup.

Observe and Monitor Your Akka.NET Applications with Phobos

Did you know that Phobos can automatically instrument your Akka.NET applications with OpenTelemetry?

Click here to learn more.