• The CRUD Tax: Why Our Old Design Was Costing Us
• Inverting Control: Actors as the Source of Truth
• Actors as State Machines: Modeling Complex Business Logic
• Syncing Actor State with Read Models
• New Akka.Persistence APIs: Async Event Handlers
• Results: Simplified Testing and Development
• Try It Yourself

We’re about a year into the process of completely re-working Sdkbin to better support our needs here at Petabridge and, eventually, other third-party software vendors.

Sdkbin was originally built on an extremely flimsy CRUD architecture that violates most of my personal design preferences - you can read about the history behind that here. But to summarize, I tend to use the following heuristics when building software:

1. Prefer optionality-preserving designs - make sure your design decisions can be reversed or altered when things inevitably change.
2. Use as few moving parts as possible - most of Akka.NET is constructed this way.
3. No magic - if nothing magically works, then nothing magically breaks either.
4. Ensure that coupling happens only where it’s necessary - coupling usually needs to happen in your integration layer (i.e. your UI or HTTP API.) Your accounting system should probably not be coupled to your payments system.

Sdkbin’s original CRUD design violated all of these principles:

1. Used hard deletes, destroying data with no ability to recover or audit (impossible to reverse);
2. Relied heavily on AutoMapper-powered generic repositories (magic with lots of moving parts); and
3. Was highly coupled throughout - Stripe payment events served double-duty as invoices, for instance.

We’ve fixed a ton of these issues already, and one of the most important tools we’re using is Akka.Persistence. We’ve also made some significant improvements to Akka.Persistence in recent releases that made it much easier for us to accomplish our ambitious goals with Sdkbin.

Let me show you what we did.

The CRUD Tax: Why Our Old Design Was Costing Us

Before diving into the solution, let me illustrate the problem we were facing. Take our SubscriptionEntity - it tracks who’s paying for what, their billing plan, and what benefits they receive. It’s one of the most important entities in the entire system.

public class SubscriptionEntity
{
    public long Id { get; set; }
    public DateTime Created { get; set; }
    public DateTime? Cancelled { get; set; }

    // Legacy - migrating to SubscriptionBenefits
    [Obsolete] public string ProxyFeedReservedName { get; set; }
    [Obsolete] public string ProxyFeedReservedApiKey { get; set; }
    [Obsolete] public long? ProxyFeedId { get; set; }

    // Legacy - migrating to BillingPlan
    [Obsolete] public decimal BillingAmount { get; set; }
    [Obsolete] public BillingFrequencyEnum BillingFrequency { get; set; }

    // New - nullable until migration completes
    public SubscriptionTierId? SubscriptionTierId { get; set; }
    public BillingPlanId? BillingPlanId { get; set; }
    public SubscriptionState? State { get; set; }

    // ... 12 more properties
}

This is the CRUD tax in action. The real problem isn’t the [Obsolete] attributes - those are intentional, part of our migration strategy. The problem is that all of this crap was bolted onto a single entity in the first place, and there’s no way to audit or track how any of it changed over time.

Want to know why a customer’s subscription was suspended last month? Good luck - hope you enjoy grepping through application logs. Need to adjust a customer’s billing window? The original hack for that involved backdating the entity’s Created timestamp, which is insane but was the only way to make the billing calculations work without rewriting everything.

We’re now using extend-only design to gradually deprecate old properties and replace them with new, more sophisticated ones. This allows us to use a Strangler Fig approach to deploy changes incrementally without a big-bang rewrite.

But here’s the key insight: we’re not making our legacy entities more CRUD-able - we’re making them read-only CQRS projections instead.

Inverting Control: Actors as the Source of Truth

The fundamental shift we made was this: instead of treating our database entities as the authoritative source of truth that gets modified through CRUD operations, we made our Akka.Persistence actors the source of truth and turned the database entities into projections of actor state.

This might sound like a subtle distinction, but it fundamentally changes how we think about state management:

Old approach (CRUD):

• HTTP API receives request → Validates → Modifies database entity → Saves changes
• Business logic scattered across controllers, services, and entity validation
• State changes have no audit trail
• Complex business rules hard to test and reason about

New approach (CQRS with Akka.Persistence):

• HTTP API receives request → Sends command to actor → Actor validates and processes
• Actor persists events using Akka.Persistence
• Actor state gets projected to database entity for queries
• Full audit trail of every state change
• Business logic encapsulated in well-tested actor state machines

Actors as State Machines: Modeling Complex Business Logic

One area where Sdkbin’s CRUD design really broke down was recurring billing. It turns out recurring billing is deceptively complex once you start handling real-world scenarios:

1. Customer’s credit card gets declined - what do we do?
   • Suspend service immediately?
   • Offer a grace period and allow the customer to retry payment with a different card?
   • Retry the previous billing method up to n times in case the decline was temporary¹, eventually suspending service?
2. Customer wants to migrate from credit cards to NET30 purchase order billing
   • Change the billing due date to lead the subscription due date by -30?
   • What if they want to make this change less than 30 days before their bill is due?
3. Customer is on NET30 billing but their procurement department is extremely slow
   • Start the billing reminder system MUCH earlier than NET30 so their procurement department doesn’t get caught off-guard by automatic service suspension?
   • Extend the grace period without moving the billing due date?

It gets even more complicated than this, and Sdkbin’s current CRUD system couldn’t handle any of these scenarios particularly well. We’d end up with spaghetti logic spread across controllers, services, and background jobs - none of which had any clear ownership of the subscription’s lifecycle.

Fortunately, this is something that Akka.NET actors model extremely well using state machines. Here’s the state machine for our SubscriptionActor:

stateDiagram-v2
    [*] --> PendingPayment: CreateSubscription

    PendingPayment --> Provisioning: InvoicePaid
    PendingPayment --> Cancelled: CancelSubscription

    Provisioning --> Provisioning: ProvisioningFailed (retry)
    Provisioning --> Active: BenefitsProvisioned

    Active --> PendingPayment: BillingDue
    Active --> Deprovisioning: PaymentFailed (retries exhausted)
    Active --> Cancelled: CancelSubscription

    Deprovisioning --> Suspended: BenefitsDeprovisioned

    Suspended --> PendingPayment: PaymentMethodUpdated
    Suspended --> Cancelled: CancelSubscription

    Cancelled --> [*]: ServiceTerminationDate

Each state has specific commands it can handle and specific events it can emit. The actor’s Become() method switches behaviors dynamically - when you’re in Active, you can process billing; when you’re in Suspended, you can only update payment methods or cancel.

This state machine makes the business rules explicit and testable. Instead of trying to reason about what happens when you call subscription.Cancel() while there’s a pending payment retry, you can see exactly what transitions are valid from each state. If you try to execute an invalid transition, the actor rejects the command.

A combination of Akka.Cluster.Sharding and a new prototype Akka.Reminders library handle all of the scheduling logic in production - but the real challenge was keeping the actor state synchronized with our read model for queries.

Syncing Actor State with Read Models

Here’s where we ran into a practical problem: our existing application code expects to query SubscriptionEntity objects from the database. We didn’t want to rewrite every query in the system - we just wanted to migrate the write path to use actors.

The solution was to update the SubscriptionEntity whenever the SubscriptionActor persists events. Here’s how we do it:

/// <summary>
/// Persist events, sync read model, and execute async post-persist actions.
/// Ensures async operations complete before replying to sender.
/// </summary>
private void PersistEventsAndSyncReadModel(
    ISubscriptionEvent[] events,
    ISubscriptionCommandResult result,
    Func<Task> asyncPostPersistAction)
{
    if (events.Length == 0)
    {
        Sender.Tell(result);
        return;
    }

    var sender = Sender; // Capture sender

    // Persist all events
    PersistAll(events,
        e => _state = _state.Apply(e),
        async () =>
        {
            // Sync read model after all events persisted
            await SyncSubscriptionReadModel();

            // Execute async post-persistence action (e.g., schedule reminders)
            await asyncPostPersistAction();

            // Reply AFTER async operations complete
            sender.Tell(result);
        });
}

We call this method each time we persist a state change inside our SubscriptionActor. The key is the third parameter to PersistAll - that’s a new API we added in Akka.NET v1.5.57 that makes CQRS scenarios like this much simpler.

New Akka.Persistence APIs: Async Event Handlers

Rather than use Akka.Persistence.Query for CQRS, which is a perfectly valid approach for building read models asynchronously, we opted to update the Akka.Persistence Persist and PersistAll APIs to support async Task callbacks. This lets us sync our Entity Framework Core models immediately after the actor state is updated, ensuring our read model is always consistent with the actor’s state.

We also added a new overload for PersistAll that takes an optional onComplete callback - a “run this code once all individual event callbacks have been completed” hook:

// PersistAll with completion callback - know when all events are done
PersistAll(orderEvents, evt =>
{
    // Each event handler can be async
    _state = _state.Apply(evt);
}, onComplete: async () =>
{
    // All events persisted and individual handlers executed
    await SyncOrderReadModel();
    await SendConfirmationEmail();
    _logger.Info("Order batch completed");
    Sender.Tell(new BatchComplete());
});

The onComplete parameter on PersistAll and PersistAllAsync lets us perform our read-model sync, kick off email notifications, schedule reminders, or handle any other side effects after we’ve successfully persisted and processed all events in the batch.

This is a huge improvement over the old approach, where you’d have to manually track whether all events had been processed before executing your completion logic. Now it’s built right into the API.

Results: Simplified Testing and Development

We’ve successfully leveraged these new APIs to keep our existing SubscriptionEntity values in sync with our actors while migrating all of the complex business logic out of flimsy ASP.NET Core CRUD handlers and into well-tested Akka.NET actors instead.

The benefits have been substantial:

1. Clearer business logic: All subscription lifecycle rules are in one place - the SubscriptionActor - instead of scattered across controllers, services, and background jobs.

2. Better testability: Testing state machines is straightforward. You send a command, verify the events that were persisted, and assert that the actor transitioned to the expected state. No need to mock database access or HTTP calls.

3. Full audit trail: Every state change is captured as an event. If a customer disputes a charge or asks why their service was suspended, we can replay the event stream and show them exactly what happened and when.

4. Easier refactoring: When business rules change (and they always do), we modify the state machine in the actor. The rest of the system continues to work because it’s just querying the read model.

5. Incremental migration: We didn’t have to rewrite everything at once. We migrated one entity at a time, keeping the old CRUD code running while we built and tested the new actor-based approach alongside it.

Try It Yourself

If you’re dealing with similar technical debt - CRUD entities that have grown unwieldy, business logic scattered throughout your application, or complex state machines that are hard to test - I’d encourage you to give Akka.Persistence a look.

The new async event handler APIs in Akka.NET v1.5.57 make it easier than ever to integrate Akka.Persistence with existing applications. You don’t have to rewrite everything - you can take an incremental approach just like we did with Sdkbin.

Upgrade today, give it a try, and let us know how you find it in the comments below.

Want to learn more about Akka.Persistence and CQRS? Check out our Akka.NET Training Calendar, which includes in-depth coverage of event sourcing, CQRS patterns, and real-world architecture patterns using Akka.NET. These training course come included with an Akka.NET Support Plan.

Observe and Monitor Your Akka.NET Applications with Phobos

Did you know that Phobos can automatically instrument your Akka.NET applications with OpenTelemetry?

Click here to learn more.