Last Thursday, September 7th we executed our “Akka.NET Application Management Best Practices” webinar and I’ve made the recording available on YouTube for everyone to watch for themselves.

The code sample I created for this webinar can also be found here: https://github.com/Aaronontheweb/akkadotnet-app-management-presentation.

However, I wanted to expand upon the webinar’s key points and emphasize some strategic / product management points that might not get noticed if you don’t pay close attention to the video.

If you’re interested in having any of these concepts taught to your company or team, please contact us about training options.

I’ve reviewed 100+ Akka.NET code bases at this point in my career, and I’ve reviewed stand-alone ASP.NET applications without any Akka.NET whatsoever. What many of these code bases all have in common is someone gets the bright idea to abstract over Akka.NET / ASP.NET / Entity Framework with an in-house framework that automates infrastructure decisions and enforces a one-size-fits-all design on how all domains are implemented.

This is a tremendously expensive design mistake that destroys optionality and creates more problems than it solves.

Bespoke Company Frameworks

I introduced the term “Bespoke Company Framework” (BCF) in a post on my personal blog earlier this year: “DRY Gone Bad: Bespoke Company Frameworks.”

As I mention in the presentation - BCFs attempt to standardize the way work is done inside a company’s specific domain. BCFs are actually quite helpful at solving infrastructure problems.

For instance, if you’re trying to automate manufacturing processes that have to be configured uniquely for multiple lines of products and all lines are built using the same universe of vendor drivers / components, writing a BCF that abstracts over hardware-specific APIs in order to streamline configurations makes perfect sense.

However, infrastructure problems are rarely where BCFs are used.

Most often they’re used to standardized how software domains are expressed and implemented, which is absolutely the wrong place to use them for the following reasons:

1. No two domains are identical, therefore shared abstractions typically require a superfluous configuration layer in order to support each domain’s idiosyncracies and
2. Shared abstractions between domains lead to coupling between them - so touching one piece of shared infrastructure means touching everything at the same time. This leads to “high volatility” changes, which are inherently high-risk.

With Akka.NET specifically there’s additional incentives to try to create BCFs:

1. Akka.NET is not as widely known as technologies like ASP.NET - perhaps our in-house Akka.NET champions can “hide” Akka.NET through some domain-specific abstractions in order to make it easier for other developers on the team to be productive with Akka.NET without having to learn it;
2. IActorRefs are not strongly typed - maybe we can add some of our own type constraints to surface design errors at compile-time rather than run-time; and
3. Akka.NET and the actor model are open-ended - Akka.NET is unopinionated about most things and that means we can have a lot of diverse approaches to persistence, event processing, observability, testing, and state management if we’re not careful. Maybe we should standardize this to help make the software more predictable through our own generalized abstractions.

These are extremely common - in many Akka.NET code bases I review I run into BCFs.

If you’d like an example of what these look like, watch the “Webinar: Akka.NET Application Management Best Practices” video and view the Akka.BCF code sample.

Problems with BCFs

Take this piece of code, a standardized Akka.Persistence base class for providing strong generic typing support and abstracting away the business logic from the actor implementation itself:

public abstract class EventSourcedActorBase<TKey, TState, TSnapshot, TEvent, TCommand> : ReceivePersistentActor
	where TState : IDomainStateWithSnapshot<TKey, TSnapshot>
	where TSnapshot : new()
	where TEvent : IDomainEvent<TKey>
	where TCommand : IDomainCommand<TKey>
	where TKey : notnull
{
	protected EventSourcedActorBase(TKey key, 
		IStateProcessor<TKey, TState, TCommand, TEvent> stateProcessor,
		IStateBuilder<TKey, TState, TEvent> stateBuilder,
		IOptions<ActorConfig> actorOptions,
		ILogger<EventSourcedActorBase<TKey, TState, TSnapshot, TEvent, TCommand>> logger)
	{
		State = CreateInitialState(key);
		StateProcessor = stateProcessor;
		StateBuilder = stateBuilder;
		Config = actorOptions.Value;
		Logger = logger;

		DefaultRecovers();
		DefaultCommands();
	}

	protected ActorConfig Config { get; }

	protected virtual ILogger<EventSourcedActorBase<TKey, TState, TSnapshot, TEvent, TCommand>> Logger { get; }

	protected virtual IStateProcessor<TKey, TState, TCommand, TEvent> StateProcessor { get; }
	protected virtual IStateBuilder<TKey, TState, TEvent> StateBuilder { get; }

	/// <summary>
	/// Used to populate the default state of this actor.
	/// </summary>
	/// <param name="key">The unique id of this actor</param>
	protected abstract TState CreateInitialState(TKey key);

	protected TState State { get; set; }

	// rest of class
}

You can view the full source of the Akka.BCF.Abstractions.EventSourcedActorBase.cs here.

We have:

• A set of generic constraints that require the types of messages, state, state validation, state processing, and the actor to all fit together;
• A standardized State + DTO scheme to keep our state and our data transfer objects separated;
• External validation / state processing / state application components, which can be written without any Akka.NET infrastructure at all.

If we apply this design to our “Subscriptions” domain in the code sample, we get something like this:

public sealed class SubscriptionStateActor : EventSourcedActorBase<string, SubscriptionState, SubscriptionSnapshot, ISubscriptionEvent, ISubscriptionCommand>
{
    public SubscriptionStateActor(string key, IStateProcessor<string, SubscriptionState, ISubscriptionCommand, ISubscriptionEvent> stateProcessor, 
    	IStateBuilder<string, SubscriptionState, ISubscriptionEvent> stateBuilder, 
    	IOptions<ActorConfig> actorOptions, 
    	ILogger<EventSourcedActorBase<string, SubscriptionState, 
    		SubscriptionSnapshot, ISubscriptionEvent, ISubscriptionCommand>> logger) 
    		: base(key, stateProcessor, stateBuilder, actorOptions, logger)
    {
        PersistenceId = $"subscription-{key}";
    }

    public override string PersistenceId { get; }
    protected override SubscriptionState CreateInitialState(string key)
    {
        return new SubscriptionState(key);
    }

    protected override SubscriptionState FromSnapshot(SubscriptionSnapshot snapshot)
    {
        return new SubscriptionState(snapshot.SubscriptionId)
        {
            ProductId = snapshot.ProductId,
            UserId = snapshot.UserId,
            Status = snapshot.Status,
            UpcomingPaymentAmount = snapshot.UpcomingPaymentAmount,
            NextPaymentDate = snapshot.NextPaymentDate,
            Interval = snapshot.Interval
        };
    }
}

Pretty easy! What could be wrong with this?

Well, imagine that we have many domains that all use EventSourcedActorBase<TKey, TState, TSnapshot, TEvent, TCommand> - which uses the following non-overrideable method for processing TCommands:

CommandAsync<TCommand>(async cmd =>
{
    var (canProcess, events, message) = await StateProcessor.ProcessAsync(State, cmd);
    if (!canProcess)
    {
        Logger.LogWarning("Cannot process command {@Command} for {ActorType} {@State} - {Message}", cmd, GetType(), State, message);
        cmd.ReplyTo?.Tell(new CommandFailed<TKey>(cmd, message));
        return;
    }

    if (events.Length == 0)
    {
        // no-op
        Logger.LogDebug("No events produced in response to {@Command} for {ActorType} {@State} - {Message}.", cmd, GetType(), State, message);
        return;
    }

    var replied = false;

    PersistAll(events, e =>
    {
        StateBuilder.Apply(State, e);
        Logger.LogDebug("Persisted event {@Event} for {ActorType} {@State}", e, GetType(), State);

        if (!replied)
        {
            cmd.ReplyTo?.Tell(new CommandSucceeded<TKey>(cmd));
            replied = true;
        }

        if (LastSequenceNr % Config.MessagesPerSnapshot == 0)
            SaveSnapshot(State.ToSnapshot());
    });
});

Notice that this CommandAsync will await on StateProcessor.ProcessAsync(State, cmd); - this actor can’t process any other messages while the async state machine is still being executed (by design.)

As long as our IStateProcessor.ProcessAsync completes in under 1 second, that shouldn’t be a problem - but what happens if we need to add support for purchase orders in our Subscriptions domain? Purchase orders can take weeks to complete - not something you can cleanly model using a single Task<T> operation.

In order to support that in this application, we would need to do the following:

1. Add virtual methods to the base classes to allow each caller to customize the behavior (relax standards); or
2. Introduce additional ActorConfig options for this class that permit per-domain customization; or
3. More than likely, we’d have to introduce a new design to our framework or introduce new base classes to allow for modeling of long-running operations using state machines. This abstraction will need to be supported inside all of the dependencies supported by our new actor base class.

All of these solutions mean having to touch EVERY DOMAIN in order to solve problems that occur in one domain - so multiply the complexitiy of fixing this problem for a single domain times all of the work to introduce it safely for every domain AND the testing overhead to ensure that you didn’t introduce any regressions in the process.

If you’ve reached this stage, congratulations: you’re now in the “touching anything means touching everything” phase of the software project - deployments are no longer safe and every minor change is now a high-risk project.

It doesn’t have to be this way. Our Akka.NET training and Akka.NET developer support can help prevent your team from reaching this stage; our Akka.NET code & architecture review can help you get out of it.

Solution: Pattern-Oriented Design

There is a way to re-use code / standardize / not reinvent the wheel without inviting the BCF productivity vampire into your team: pattern-oriented design.

public sealed class SubscriptionStateActor : ReceivePersistentActor
{
    private readonly ILoggingAdapter _log = Context.GetLogger();
    private readonly IPaymentsService _paymentsService;
    
    public SubscriptionState State { get; set; }
    
    public override string PersistenceId { get; }

    public SubscriptionStateActor(string subscriptionId, IPaymentsService paymentsService)
    {
        _paymentsService = paymentsService;
        PersistenceId = $"subscription-{subscriptionId}";
        State = new SubscriptionState(subscriptionId);
        
        Recovers();
        Commands();
        Queries();
    }

    // rest of code

}

You can view the full source of the Akka.Pattern.Domains.Subscriptions.Actors.SubscriptionStateActor.cs here.

This actor has even less code than its BCF counterpart - here are its standard message processing routines:

CommandAsync<ISubscriptionCommand>(async cmd =>
{
    _log.Debug("Processing command {0} in State {1}", cmd, State);
    var (resp, events) = await State.ProcessCommandAsync(cmd, _paymentsService);

    if (events.Length == 0)
    {
        _log.Debug("No events produced in response to command {0} for State {1}", cmd, State);
        Sender.Tell(resp);
    }

    var replied = false;
    PersistAll(events, e =>
    {
        _log.Info("Persisted event {0} for command {1} in State {2}", e, cmd, State);
        State = State.Apply(e);
        if (!replied)
        {
            Sender.Tell(resp);
            replied = true;
        }
        
        if (LastSequenceNr % MessagesPerSnapshot == 0)
            SaveSnapshot(State);
    });
});

The State.ProcessCommandAsync and State.Apply do most of the work here - pure functions implemented as extension methods on this domain-specific type:

public static async Task<(ISubscriptionCommandResponse resp, ISubscriptionEvent[] @events)> ProcessCommandAsync(
    this SubscriptionState state, ISubscriptionCommand cmd, IPaymentsService paymentsService)
{
    switch (cmd)
    {
        case CreateSubscription create:
            return await ProcessCreateSubscription(state, create, paymentsService);
        case CheckSubscriptionStatus check:
            return await ProcessCheckSubscription(state, check, paymentsService);
        case CancelSubscription cancel:
            return await ProcessCancelSubscription(state, cancel, paymentsService);
        case ResumeSubscription resume:
            return await ProcessResumeSubscription(state, resume, paymentsService);
        default:
            throw new ArgumentOutOfRangeException(nameof(cmd));
    }
}

How is this pattern-oriented design? You should watch the “Webinar: Akka.NET Application Management Best Practices” video if you want the full details, but in sum:

• The message grammar is categorized as an “effects system” - commands / events / queries;
• All message types are composed of interfaces that signal the unique identity of a domain-specific entity (i.e. ISubscriptionCommand : IWithSubscriptionId);
• All state is defined as simple, immutable record types - no DTOs / mapping;
• All business rules are implemented as pure extension methods on the state types - this helps keep the actors thin and makes the actor code easily testable without the Akka.TestKit; and
• No code is really shared between domains - rather it’s the same patterns and systematic approach to design that is used over and over again.

What problems does this solve, exactly?

1. Eliminates the need to reinvent the wheel by establishing clear, easy-to-repeat patterns that can be applied to each domain - “how should I organize messages for the entities in this domain?” and the like are questions with ready-made answers;
2. Because nothing is shared, it’s very easy to customize entity actors / state / messages to each domain independently from others - no more “everything touches everything” problem; and
3. Atomizes the amount of code you have to write in the first place - no DTOs, no unnecessary configuration, no inheritance, no crazy generic parameters and constraints, and very little actor-specific code.

If you want to see the full Akka.Pattern solution, click here.

Mindset Shift

Once you start looking at the guts of the Akka.Pattern solution and comparing it to the Akka.BCF implementation, you’ll see a lot of similarities (i.e. message grammars, “thin” actors) - but the differences are stark: patterns don’t share code, don’t inherit, and are quite minimalist.

What drives the creation of BCFs is a defensive mindset: “I can eliminate errors, waste, and inefficiency through the type system and polymorphism.” That’s a noble goal, but it’s premised on a subtle classification error - the type system already does all of these things for the developer inherently; what the BCF creator is really trying to control is constrain how types are defined.

Essentially, BCF developers are trying to limit .NET’s type system to a smaller universe of permissible expressions. This is a tremendous mistake is it introduces coupling and becomes very expensive to refactor later if the BCF designer was too opinionated in their design (and BCFs, by their very nature, tend to be very opinionated.

When it comes to Akka.NET, the defensive mindset is around Akka.NET’s learning curve: it’s too high and therefore we need some way to hide it from the other developers. Akka.NET is actually pretty easy to learn, but that’s a story for a different day and the developers writing code for a system built on top of it are going to have to learn it anyway - it’s like trying to have developers build web apps without learning how HTTP works.

The mindset all comes down to worrying about what future developers are going to do, what they might do wrong, what they might do differently, and essentially trying to baby-proof your applications.

Stop it.

The pattern-oriented approach surrenders the need for control, recognizing that it’s both futile and self-defeating. Instead we propose the following:

1. Establish effective patterns that are readable, easy to understand, and easy to repeat;
2. Lead by example - help sell the patterns to your team members and be able to clearly explain why these are helpful and effective;
3. Recognize and anticipate the need for differences in different domains - one-size-fits-all-ism for domain design is an anti-pattern; and
4. Encourage minimalism - skinny actors, simple data types, minimal amount of inheritance, and so on. The less the code tries to do the less code there is. Simple does not mean “under-engineered” - it usually means “as few moving parts as needed.”

I’ll be expanding on this in some subsequent blog posts - leave your comments below!

Observe and Monitor Your Akka.NET Applications with Phobos

Did you know that Phobos can automatically instrument your Akka.NET applications with OpenTelemetry?

Click here to learn more.