The actor model is a radically new concept for the majority of Akka.NET users, and therein lies some challenges. In this post we’re going to outline some of the common mistakes we see from beginners all the time when we take Akka.NET questions in Gitter chat, StackOverflow, and in our emails.

7. Making message classes mutable

One of the fundamental principles of designing actor-based systems is to make 100% of all message classes immutable, meaning that once you allocate an instance of that object its state can never be modified again.

The reason why immutable messages are crucial is because you can send the same message to 1000 actors concurrently and if each one of those actors makes a modification to the state of the message, all of those state changes are local to each actor.

There are no side effects to other actors when one actor modifies a message, which is why you should never need to use a lock inside an actor!

For example, this is an immutable message class:

public class Foo{
	public Foo(string name, ReadOnlyList<int> points){
		Name = name;
		Points = points;
	}

	public string Name {get; private set;}
	public ReadOnlyList<int> Points {get; private set;}
}

This class is immutable because:

• string is an immutable class - any modifications to it produce an entirely new string. The original is never modified, so to all of the other actors processing this Foo instance Name will always have the same value.
• ReadOnlyList<T> is immutable because its contents can’t be modified - it’s, obviously, read-only.
• All of the setters are private, meaning that no one can modify Name and Points to refer to different instances of string and ReadOnlyList<T> respectively. Those properties will only refer to the objects whose references are passed into Foo’s constructor.

What we tend to see newbies write are a lot of basic POCO classes:

public class Foo{
	public string Name {get; set;}
	public List<int> Points {get; set;}
}

This class is not immutable and it’s unsafe for actors because:

• Any actor can change the object Name or Points refers to and create a side-effect for another actor and
• Any actor can modify the contents of Points via the List<int>.Add or List<int>.Remove method and create yet another side-effect.

When you use mutable messages, you have to start using shared memory synchronization mechanisms like lock, ReaderWriterLockSlim, and Interlocked inside your actors in order to make your application thread-safe. This defeats the entire point of using actors!

Always define your message classes to be immutable, but if you can’t do that, here’s a HOCON configuration setting you can turn on that will force each message to be serialized and deserialized to each actor who receives it, which guarantees that each actor receives their own unique copy of the message:

akka.actor.serialize-messages = on

This setting is intended for testing only, and if you are using it in production you will see a 40% drop in throughput (because serialization is expensive.)

6. Losing sleep over Disassociated messages

I get this question constantly in Gitter chat whenever someone starts playing with Akka.Remote and they see two nodes disassociate for the first time. If you’re not familiar with Akka.Remote, check out our video about Akka.NET Internals: How Akka.Remote Connections Work or the official Akka.NET Akka.Remote documentation.

Disassociation refers to the process by which two Akka.NET ActorSystem instances disconnect from each-other.

Sometimes it’s intermittent, such as when one system is at 100% CPU utilization for a few minutes and fails a heartbeat check; sometimes disassociation occurs as a result of hardware or software failure; and other times disassociation is part of a planned process when you are done working and wish to shutdown an ActorSystem.

Regardless of why a disassociation occured a Disassociated message will always be written to the error log. Many Akka.NET developers freak out and take this to mean that there’s something wrong with their application, even on planned shutdown.

The reason why we do this is to brute-force disassociation messages onto the error log regardless of your logging settings because they’re critical events. If we made planned disassociation events an Info level event and you had that setting turned off, it’d be a lot harder to diagnose problems related to accidental shutdown and others.

Disassociation messages only matter in the context of what the system is doing when they occur. And for that, you should be using Akka.NET’s logging system to help make that context available.

5. Sending MASSIVE messages over Akka.Remote

Andrew wrote a brilliant post about “Large Messages and Sockets in Akka.NET” where he explains this issue beautifully, but I’ll recap it here.

One of the ways you can break the location transparency of actors in Akka.NET is by ramming massive messages down the sockets that remote actors use to communicate with eachother - the reason being that the transports themselves can’t support arbitrarily large messages and larger messages have a higher latency cost when it comes to transmission and serialization.

Stuffing 100mb of data into a single message and hoping for the best is a recipe for disaster - instead, break up those messages into something reasonably small (kilobytes, not megabytes) and stream them to your destination actors. You’ll achieve faster response times, lower likelihood of failure, and higher degrees of parallelism if you do.

4. Executing long-running actions inside an actor’s Receive method

Actors process exactly one message at a time inside their Receive method, as shown below.

This makes it extremely simple to program actors, because you never have to worry about race conditions affecting the internal state of an actor when it can only process one message at a time.

Actors signal their mailbox that they’re ready to begin processing more messages once the Receive block exits - that’s what makes serial processing work.

public class FooActor : ReceiveActor{
	public FooActor(){
		Receive<Foo>(f => {
			// work...

			//done processing - signals mailbox.
		});
	}
}

Unfortunately, there’s a price you pay for this: if you stick a long-running operation inside your Receive method then your actors will be unable to process any messages, including system messages, until that operation finishes. And if it’s possible that the operation will never finish, it’s possible to deadlock your actor.

The solution to this is simple: you need to encapsulate any long-running I/O-bound or CPU-bound operations inside a Task and make it possible to cancel that task from within the actor.

Here’s an example of how you can use behavior switching, stashing, and control messages to do this.

public class FooActor : ReceiveActor,
						IWithUnboundedStash{

	private Task _runningTask;
	private CancellationTokenSource _cancel;

	public IStash Stash {get; set;}

	public FooActor(){
		_cancel = new CancellationTokenSource();
		Ready();
	}

	private void Ready(){
		Receive<Start>(s => {
			var self = Self; // closure
			_runningTask = Task.Run(() => {
				// ... work
			}, _cancel.Token).ContinueWith(x =>
			{
				if(x.IsCancelled || x.IsFaulted)
					return new Failed();
				return new Finished();
			}, TaskContinuationOptions.ExecuteSynchronously)
			.PipeTo(self);

			// switch behavior
			Become(Working);
		})
	}

	private void Working(){
		Receive<Cancel>(cancel => {
			_cancel.Cancel(); // cancel work
			BecomeReady();
		});
		Receive<Failed>(f => BecomeReady());
		Receive<Finished>(f => BecomeReady());
		ReceiveAny(o => Stash.Stash());
	}

	private void BecomeReady(){
		_cancel = new CancellationTokenSource();
		Stash.UnstashAll();
		Become(Ready);
	}
}

3. Using ActorSelection instead of IActorRef

All Akka.NET actors have their own unique address and that opens up all kinds of possibilities within the actor programming model, but many new Akka.NET developers stick to the habits they formed when developing and working with HTTP APIs, which is to explicitly address every resource on the network by its URI.

That’s the only way to do it in HTTP land, because that’s how HTTP works. With Akka.NET actors, you have access to a much more powerful tool: ActorRefs.

The IActorRef type in Akka.NET is what implements this concept, and it serves as a handle to an actor. That handle’s location is intended to be transparent - meaning that you can change at deploy-time whether or not the underlying actor exists locally in-memory or remotely over the network. This is an extremely powerful concept that makes it trivially easy to distribute your actors, refactor them, and scale out horizontally.

Unfortunately, a lot of new Akka.NET developers fail to utilize this concept because they’re still thinking in REST, so they write code that heavily uses ActorSelections and explicit ActorPaths instead. This is an anti-pattern. By default you should always be using IActorRefs throughout your actor code.

Read more about this in “When Should I Use ActorSelection?”

2. Over-dependence on Dependency Injection frameworks

This is an issue with .NET developers in general, but it rears it head more often in Akka.NET actor code because of the long lifespans of many actors.

Some actors live very short lives - they might only be used for a single request before they’re shutdown and discarded. Other actors of the same type can potentially live forever and will remain in memory until the application process is terminated. This is a problem for most DI containers as most of them expect to work with objects that have fairly consistent lifecycles - not a blend of both. On top of that, many disposable resources such as database connections get recycled in the background as a result of connection pooling - so your long-lived actors may suddenly stop working if you’re depending on a DI framework to manage the lifecycle of that dependency for you.

Thus it’s considered to be a good practice for actors to manage their own dependencies, rather than delegate that work to a DI framework.

If an actor needs a new database connection, it’s better for the actor to fetch that dependency itself than to trust that the DI framework will do the right thing. Because most DI frameworks are extremely sloppy about cleaning up resources and leak them all over the place, as we’ve verified through extensive testing and framework comparisons. The only DI framework that works correctly by default with Akka.NET actors is Autofac.

1. Confusing the procedural async / await model with actors

The most profound mistake we see developers make is a misconception where actors are still not viewed as “units of concurrency” by the developers. Actors are inherently asynchronous and concurrent - every time you send a message to an actor you’re dispatching work asynchronously to it.

But because the async and await keywords aren’t there, a commonly held view among some new Akka.NET users is that therefore the IActorRef.Tell operation must be “blocking.” This is incorrect - Tell is asynchronous; it puts a message into the back of the destination actor’s mailbox and moves on. The actor will eventually process that message once it makes it to the front of the mailbox.

Moreover, inside many Receive methods we see end users develop lots of nested async / await operations inside an individual message handler. There’s a cost overlooked by most users to doing this: the actor can’t process any other messages between each await operation because those awaits are still part of the “1 message at a time” guarantee for the original message!

There are some occassions where async / await makes life a lot easier, but most of the time it’s code smell inside Akka.NET actors. You’re much better off using the PipeTo operator and treating your Tasks like message sources - this will allow your actor to run mulitple asynchronous operations simultaneously while still observing all of the actor concurrency guarantees and continuing to process messages from the mailbox.

Observe and Monitor Your Akka.NET Applications with Phobos

Did you know that Phobos can automatically instrument your Akka.NET applications with OpenTelemetry?

Click here to learn more.