How Do You Fix 70% Data Loss Across 1 Million Concurrent Connections?

A case for why you should consider purchasing an Akka.NET Support Plan for your organization.

6 minutes to read

When your Akka.NET application starts dropping 70-80% of incoming connections in production, who do you call? That’s the situation one of our Production Support customers faced this year - and it’s exactly the kind of problem our Akka.NET Support Plans are designed to solve.

Opportunities to purchase developer expertise with a credit card are rare. That’s exactly what we offer - and I want to show you what that looks like in practice.

A Million Connections and 70% Data Loss

We had a large-scale IOT customer running a massive utility meter network. Their requirement: handle ~1M meters all phoning home at once - a burst of 1 million concurrent TCP connections hitting their Akka.NET listeners simultaneously.

They sent us fairly detailed WireShark data indicating that up to 70-80% of these connections were being lost and thus the meter data wasn’t getting saved.

I suspected, right off the bat, that the problem was likely a combination of poor Akka.NET performance on Akka.IO (it’s old code I didn’t write) and Linux system limits on TCP settings.

To resolve this issue, I started with:

  1. Performance-optimizing Akka.IO to quickly clear its TCP backlog;
  2. Instrumenting the user’s application with several OpenTelemetry meters / gauges to help measure system-level and application-level TCP connectivity;
  3. Creating a Grafana dashboard for visualizing that; and
  4. Setting up a CI/CD system to deploy their application instance into our test lab environment locally here at Petabridge, so we could rapidly test changes.

The most important thing we needed to do, in order to cut cycle time down with the customer’s team, was to be able to reproduce the data loss locally. That required us to run a ~1M connection load test right here in our office. With k6 distributed across 200 pods in our local K8s cluster (5 machines), we could do it. This pegged our test infrastructure harder than the actual application we were testing - but we reproduced the connection loss.

Eventually we fixed the problem through changing some Linux kernel settings; shipping the improved Akka.IO implementation; and spreading the load out over several machines using a modulus (round-robin) distribution.

This is the kind of support Production Support customers get. We didn’t just answer a question - we reproduced their million-connection environment, instrumented their application, built dashboards, and shipped fixes to Akka.NET itself. All included in the plan.

What Support Plan Customers Got This Year

One of the features of our Production Support plan is “patch priority” - we will patch bugs, add features, or write documentation for Akka.NET requested by these customers ahead of our normal open source development cadence.

Here’s what that looked like in 2025:

At the request of our customers, we’ve contributed well over 100 improvements and fixes to Akka.NET, its plugins, and documentation in 2025 alone.

Solutions Application-Specific Problems

Beyond framework patches, we solve hundreds of customer-specific issues each year through our Developer Support Portal - problems with how Akka.NET is being used or configured in end-user applications:

  • Memory leaks with application-specific actors
  • Akka.NET + Kubernetes DNS / addressing errors
  • Work-distribution system design for high-volume IOT systems
  • Fail-over and recovery strategies for multi-datacenter disaster recovery
  • Akka.Streams / actor workflow optimizations
  • Deep dives into Akka.Remote / DotNetty code to find memory-management constructs causing networking problems on specific platforms

My favorite war story from last year: “The Worst .NET Bug I’ve Ever Fixed.”

Training

All Akka.NET Support Plan customers get access to our live Akka.NET training. We’ve run 23 sessions this year with over 150 total attendees, and we’re about to release our 2026 schedule.

What Our Customers Say

“I can highly recommend the Petabridge support plans. In pre-production development, we applied Petabridge’s expertise to review our design decisions and approach, which taught me more about the framework, saved us a few production issues, and (most importantly) really gave us the confidence to proceed. In another instance, our request for new product functionality was given due consideration. Petabridge support has been responsive and thorough.” — Sean Killeen, SCT Software

How to Buy Akka.NET Support

We offer two plans:

  • Design-Time Support - $3,995 per year. Unlimited seats in our support portal and ability to attend our trainings. This plan is for teams building new Akka.NET applications who want guidance, training, and deployment questions answered in a timely fashion via Email and the Akka.NET Discord.

  • Production Support - $9,995 per year. For teams with Akka.NET applications in production. Same benefits as Design-Time Support, plus:

    • Video calls upon request
    • 1-business day response time (SLA)
    • Patch Priority - which is where all of these Akka.NET fixes come from
    • Log review - send us your 100MB log dump and we’ll figure out what went wrong

Get Akka.NET Support Contact Us With Questions

Planning Your Q1 2026 Budget?

If you want Akka.NET support in place for January, now’s the time to start the conversation - even if it’s just a 15-minute call to scope what you need.

The next time you’re stuck on a tricky distributed systems problem, do you want to be asking ChatGPT at 2 AM - or do you want the team that built the framework standing behind you?

If you liked this post, you can share it with your followers or follow us on Twitter!
Written by Aaron Stannard on November 21, 2025

 

 

Observe and Monitor Your Akka.NET Applications with Phobos

Did you know that Phobos can automatically instrument your Akka.NET applications with OpenTelemetry?

Click here to learn more.