Today we’ll talk about one of the most common communication patterns, which is publish/subscribe, and how to perform it in a distributed environments using Akka.NET. In case if you want to save yourself some time implementing it, its already available as part of a bigger package called Akka.Cluster.Tools. To install it, you can simply get the latest prerelease version from NuGet:

install-package Akka.Cluster.Tools -pre

Below, we’ll cover the abilities and limitations that come with the Akka.Cluster.Tools.DistributedPubSub implementation.

When EventStream isn’t enough

An idea of distributed publish subscribe came from slightly different design choices than those of the default Akka pub/sub model (EventStream).

EventStream has been invented to work with actor systems from the ground up, even in systems that are not inherently distributed. It’s extremely simple, lightweight, and fast. Dead letters - a place where every undeliverable message lands by default - is implemented on top of it.

However, EventStream comes with some limitations. It works only locally in the scope of the current actor system. This means that you cannot publish/subscribe actors from other nodes and publish messages in a global address space without worrying on which node they live. These limitations were motivation for a separate, cluster-specific publish-subscribe feature.

For the sake of this article you can think of distributed pub/sub as a decentralized message broker. Just like any other cluster plugin,it must be initialized by every node it’s going to work on - and like any other akka plugin can be limited to a subset of cluster nodes setting a specific role for each node (see: akka.cluster.pub-sub.role).

All of the API capabilities are achieved through special messages send to an actor called mediator:

var mediator = DistributedPubSub.Get(system).Mediator;

Unlike the EventStream, which groups subscribers by a message type, the mediator uses topic-based subscriptions. Before we go further let’s quickly describe what they are all about.

How do topic-based subscriptions work in Akka.NET?

Topics are keys used to abstract the address space from actor paths. They work in a distributed manner - when using them, you don’t need to think about addressing specific nodes, because each topic represents a cluster-wide key space.

To subscribe to a topic simply send a Subscribe message to a local mediator. Since the subscription process is a distributed one, if you want to make sure that all nodes correctly acknowledged it, you should receive SubscribeAck message back, once the process completes on all nodes. Exactly same rules apply when unsubscribing from the topic (which is realized by Unsubscribe/UnsubscribeAck messages send to/from the mediator).

So how does it look like in the code?:

const string TopicName = "topic-name";

// Distributed pub/sub subscriber actor
class Subscriber : ReceiveActor
{
    public Subscriber()
    {
        var mediator = DistributedPubSub.Get(Context.System);
        mediator.Tell(new Subscribe(TopicName, Self));
        Receive<SubscribeAck>(ack => Console.WriteLine($"Subscribed to '{TopicName}'"));
        Receive<string>(s => Console.WriteLine($"Received a message: '{s}'"));
    }
}

// publish messages on distributed pub/sub topic
mediator.Tell(new Publish(TopicName, "hello world"));

We didn’t unsubscribe this actor from the topic explicitly here because it’s not necessary. Once this actor stops, it will be unsubscribed automatically from all of the topics.

Different publishing modes

By default Publish will always send a message to subscribers of the target topic. This can be modified by using groups. When subscribing to a topic, we may provide an additional groupId - this will build a different key space inside of topic itself. This means that subscribers, which specified group aside of the topic, will not get notified with messages using ordinary Publish message.

However, when sending a Publish message with flag sendOneMessageToEachGroup set to true, we send a payload only to a single actor in each group within provided topic. Subscribers that haven’t specified any groupId when subscribing will be ignored.

To avoid any potential confusion we can present it in a following way:

This affords Akka.NET users a variety of publishing modes. For example, if we want to send a message only to one subscriber in each topic we simply can define a group that has the same boundaries as its topic.

Deciding which subscriber in group should receive a message can be customized by using akka.cluster.pub-sub.routing-logic config and can be one of 3 values:

• random (default) will send messages without any order.
• round-robin will form a ring from subscribers and iterate over them when sending next message.
• broadcast will send message to everyone which makes this kind of publishing to work like sendOneMessageToEachGroup flag was set to false but still apply to subscribers with groupId.

Distributed pub/sub using actor paths

Akka.NET distributed pub/sub allows us also to send a message to a single actor inside a cluster registered under its relative actor path. In order to use it, first you need to register actor reference by sending Put message to a mediator, we were speaking about before. To publish a message to actors registered this way you send either a Send (to send to a single actor) or SendToAll (to publish message to all actor sharing the same relative path, but living on a different nodes) message to mediator.

To better visualize that, we’ll use a following example:

var actor = system.ActorOf(Props.Create(() => new MyActor()), "my-actor");
// register an actor - it must be an actor living on the current node
mediator.Tell(new Put(actorRef));

// send message to an single actor anywhere on any node in the cluster
mediator.Tell(new Send("/user/my-actor", message, localAffinity: false));

// send message to all actors registered under the same path on different nodes
mediator.Tell(new SendToAll("/user/my-actor", message, excludeSelf: false));

At this point you may want to ask yourself: why don’t just use clustered group routers? There are several traits which make DistributedPubSub approach better in some situations:

• Unlike a cluster router you don’t need an IActorRef handle in order to send a message. You can just use relative actor path without specifying on which node does it live.
• Ability to set a localAffinity may save you a time necessary to send a message over the network when there’s a candidate able to receive the message living on a current node. A great use case for that is a stateless worker pattern: actors that don’t have any internal state and are used only for data processing. This makes any of them perfectly capable to handle the message you want to send, no matter which node they live on.
• Ability to excludeSelf allows you to determine if message shouldn’t be published into current node which makes it a great choice for situations where actors want to gossip messages to each other but don’t want to unnecessarily message themselves.

Of course downside is that you cannot pre-configure actors used this way like cluster routers using HOCON settings. You cannot automatically create them either using remote deployment either.

What distributed pub/sub is not?

The goal of this library isn’t to replace existing distributed message queue or log implementations like RabbitMQ, MSMQ, Azure Service Bus or Kafka. It’s only about distributed publish subscribe and therefore there are some important issues not covered by the pub/sub mediators:

• Published messages are not persisted. If something happens, for example a node goes down before publishing, those messages are lost. You can use an Akka.Persistence or your own persistent backend to fill this gap if necessary.
• Published messages are delivered using at most once delivery semantics. It’s possible (but very rare) that messages will get lost while being transported over the wire. This is a nature of working with distributed systems. Distributed pub/sub doesn’t give you any form of acknowledgment in this case. You can implement it by yourself, by sending an ACK/NACK messages from subscribers back to a publisher if necessary.
• Since delivery reliability isn’t a concern of the plugin itself, delivery retries are also not covered by it.

About the Author

Bartosz Sypytkowski is one of the Akka.NET core team members and contributor since 2014. He’s one of the developers behind persistence, streams, cluster tools and sharding plugins, and a maintainer of Akka F# plugin. He’s a great fan of distributed functional programming. You can read more about his work on his blog: http://bartoszsypytkowski.com

Observe and Monitor Your Akka.NET Applications with Phobos

Did you know that Phobos can automatically instrument your Akka.NET applications with OpenTelemetry?

Click here to learn more.