Real World Akka.NET Clustering: State Machines

Using Akka.Cluster and State Machines to Build Reactive Systems

Petabridge recently launched its first cloud service, Sdkbin, and it’s been a great opportunity for our team to use the Akka.NET tools, patterns, and practices that we’ve been recommending to our customers for years.

In this blog post I’m going to walk through one of the most universal real-world Akka.NET actor patterns: the state machine.

Finite State Machines: a Primer

One of the major drawbacks of web programming is that due to its stateless and non-local affinity, many programming patterns that were commonplace in embedded, desktop, and client programming have lost their importance in everyday programming. These “lost patterns” tend to be stateful ones - and none more so than the “finite state machine” or “FSM” for shorthand.

Finite State Machine concept

The idea behind a finite state machine is simple but powerful:

  1. For a given type of entity modeled in your software system it can exist in one of many possible states - but it can only occupy a single state at a time;
  2. An entity usually has some “state data” objects that provide contextual information for the current unit of work in process - i.e. an Order entity might be in an Open state and have an OrderData object that contains a unique orderId, customerId, and so on;
  3. As the state machine processes events both its state and its state data can change - in particular, the state machine can transition from one state to another in response to 1 or more events being processed; and
  4. States can be re-entrant - an entity can return to its original state in the event of a timeout, error, or based on the type of event it processed.

When Do You Need a State Machine?

If...

Now Open for Business: Sdkbin - the Marketplace for .NET Developers

A Better Way to Buy and Sell Business-to-Developer Software and Services

Sdkbin Logo - The Marketplace for Software Developers I formally introduced Sdkbin back on my personal blog in June, 2020, but it was not yet ready for Petabridge, its customers, or the users of the open source projects we support. As of September 30th, Sdkbin is now live and open for business.

So, what is Sdkbin?

A Better Way to Buy and Sell Business-to-Developer Software and Services

Petabridge has been in business for nearly 6 years and nearly all of our business comes from supporting Akka.NET and its users.

Among the products and services we sell:

Some of these services, such as code review and onsite training, are inherently bespoke - we have to tailor what we deliver to the unique needs of each customer.

But other services, such as Phobos and our Akka.NET Support Plans, are valuable because they are so standardized - pick a license / service level, a quantity, buy it, and use it. As we’ve learned over the course of time: it is very painstaking to build and run sales, fulfillment, renewal, and support operations for these services manually.

The manual experience is also suboptimal for our customers:

  1. It’s difficult to keep track of license keys, remaining support hours, and which team members have access to each of these through all of the separate systems we use for delivering these products and services (i.e. Freshdesk, MyGet, and Github;)
  2. Business continuity is an issue - if a key decision maker leaves then service disruptions often occur; and
  3. Managing renewals often means an arduous procurement process if...

The Current State of OpenTelemetry in .NET

OpenTelemetry vs. OpenTracing and the Future of Observability in .NET

In the past we’ve discussed why distributed tracing is becoming commonplace and the OpenTracing standard for instrumenting libraries and applications. In this post I want to touch on the emerging OpenTelemetry standard, which will become a common component used to instrument ASP.NET Core applications in the not too distant future.

OpenTelemetry logo

What is OpenTelemetry?

OpenTelemetry is the convergence of two competing tracing standards:

  • OpenTracing - developed by a community of APM vendors and library authors and
  • OpenCensus - developed by Google.

The goal is to provide a unified set of APIs library authors can include inside their applications in order to:

  • Propagate distributed tracing context, including the new W3C HTTP tracing standard;
  • Aggregate metrics (counters, meters, etc); and
  • Export metrics and trace data to a variety of different Application Performance Monitoring (APM) backends, which can be configured entirely by the application developer.

OpenTelemetry vs. OpenTracing

OpenTelemetry vs. OpenTracing

So what are the material differences between OpenTelemetry and OpenTracing? Why do we need another new standard?

The major technical differences are:

  1. OpenTelemetry’s core library is the Tracer implementation - the traces are created and correlated using OpenTelemetry calls and then only during the export process do the traces hit any vendor-specific code. This makes the performance of OpenTelemetry very consistent regardless of what vendors end-users choose. In contrast, with OpenTracing all of the real calls are done by a vendor-specific implementation of the OpenTracing APIs - so as a library author you could have a great set of benchmarks using a Zipkin OpenTracing library but not-so-great ones using a Jaeger OpenTracing library. I prefer OpenTelemetry’s approach here.
  2. OpenTelemetry supports metrics instrumentation in addition to tracing - a library author...

Akka.NET v1.4 is Available on .NET Standard 2.0

Akka.Remote Performance Improvements, Akka.Cluster.Sharding out of Beta, Akka.Cluster.Metrics, and More

As of this week, Akka.NET v1.4 is now fully available for users to use. We’ve published a detailed article on the Akka.NET website that describes what’s new in Akka.NET v1.4, but we wanted to capture some of the highlights here.

Akka.Cluster.Sharding and Akka.DistributedData are out of Beta

Akka.Cluster.Sharding has been used by Akka.NET users in production for years, mostly relying on the underlying Akka.Persistence storage engine to save all of the sharding state used to distributed entity actors evenly across the cluster.

Akka.Cluster.Sharding actor distribution

However, the reason why the module remained in beta for the past couple of years is because Akka.Cluster.Sharding’s alternate storage mode, Akka.DistributedData, which uses eventually consistent in-memory replication to manage cluster sharding state throughout an Akka.NET cluster, lacked a stable wire format and hadn’t been certified for production yet.

Introduction to Akka.Streams

Building High-Level Streaming Worklows in Akka.NET

The goal of this blog post is to give you a glimpse of the idea and basics behind Akka.Streams. What they are and why you may find them useful addition in your day to day job.

In short: Akka.Streams is a library build on top of Akka.NET, which allows you to consume and process potentially infinite streams of data in type-safe and resource-safe way.

To make it easier to visualize, let’s take an example: we have a queue (i.e. RabbitMQ), that sends us a documents, which we have to parse, turn into structured data and save in the database.

How to Use Github Professionally

Best Practices for Working with Github in Team Settings

I originally started writing this post as internal documentation for our own team here at Petabridge, but I thought this would be useful for our readers and users as well.

Github has evolved over the years into a vast, rich ecosystem filled with lots of first and third party features that make developers more productive and effective.

Yet the vast majority of developers haven’t had much experience working effectively with Github in day-to-day work. Many developers don’t have a Github account; some have created some simple projects or filed some bug reports on popular projects; and few have forked a repository and made a pull request.

In this post you’re going to learn the best practices for working with teams of developers on Github who are working towards producing production-ready software. Everything in this post is equally applicable for developers working behind the firewall on proprietary software via Github Enterprise as it is for developers who want to submit a patch to popular open source projects like Akka.NET.

Two Unnecessary Costs of Software Development

Putting my “Chief Technology Officer” hat on for a second, there are lots of cost levers behind the total expense of software development and most of them are necessities. Yes, we should always allow plenty of time and money for testing and user feedback. Yes, we should try to pay down technical debt. We’re not talking about any of that.

What I’m talking about are unnecessary costs, waste costs, that can be avoided via using Github effectively as a communication platform among a development team. Those costs are:

  1. False starts - designing the wrong thing from the beginning;
  2. Blind alleys - designing the right thing using the wrong strategy.

In both of these cases the developers’ time and company money is wasted....

Best Practices for Designing Akka.NET Domain Events and Commands

How to Make Akka.NET Programming Easier by Designing Events Well

In this blog post we’re going to cover some best practices you can use when designing domain events and objects intended to work with Akka.NET. If you follow these best practices you’ll run into fewer errors, clearer log messages, and a better extensibility experience.

Use Marker / Identity Interfaces Generously

This first tip is designed to make it easier to extend your messaging systems without having to manually update the Receive<T> statements on a large number of actors.

Suppose I’m working the code from our new Akka.Cluster Workshop - in this application we have a large number of domain events for the purpose of trading stocks:

  1. Bid - offer to buy N units of stock at a specific price point;
  2. Ask - offer to sell N units of stock at a specific price point;
  3. Match - the stock trading system has matched an Ask order with a Bid; and
  4. Fill - some amount of a Bid or Ask order has been filled by a Match.

All of these events have several common identifiers and properties that can be really useful for routing, sharding, or distributing these messages:

  1. The stock ticker symbol (MSFT, TEAM, AMD, etc…);
  2. The id of the order; and
  3. They all represent live trading events happening as a result of trader activity - this is distinct from an event emitted by the exchange indicating what the newest “market price” for a specific ticker symbol is.

Well, in order to make my system more extensible and easier to debug I’m going to introduce some common marker interfaces - for instance, an IWithStockId interface:

/// <summary> 
    
  

New Akka.Remote and Akka.Cluster.Sharding Command Palettes for Petabridge.Cmd

Tools for Akka.Cluster.Sharding Management and Akka.Remote Management

We introduced Petabridge.Cmd in 2017 to make it easy for Akka.NET users to deploy, monitor, and manage their Akka.NET applications. We have now expanded the capabilities of Petabridge.Cmd by introducing two new command palettes to further simplify the way you monitor your Akka.NET applications. The new command palettes will allow you to monitor your system connections with Petabridge.Cmd.Remote and allow you to obtain sharding information using Petabridge.Cmd.Cluster.Sharding.

If you have not heard of Petabridge.Cmd yet, please take a look at our introduction YouTube video for Petabridge.Cmd

Video Highlights

  • Description of Petabridge.Cmd.
  • How Petabridge.Cmd works.
  • How Petabridge.Cmd uses command palettes.
  • Demo of setting up Petabridge.Cmd and the use of commands.

Petabridge.Cmd.Remote Commands

Let’s cover the 4 new commands that we have introduced with Petabridge.Cmd.Remote. These can be used with applications using Akka.Remote or Akka.Cluster:

  1. remote stats - allows you to see the number of Association and Disassociation events seen by the node running your Petabridge.Cmd.Host;
  2. remote connections - provides you with the number of active remote connections to the local host;
  3. remote tail - writes all of the connections and disconnections events out to the console until stopped via Control + C and;
  4. remote history - shows the connection events witnessed by the node. Can show up to 1,000 events.

We encourage you to give these commands a try in your application. Below you can see the output of some of the commands in our remote command palette. We ran the remote connections on a cluster and you can see the connection of our host node to all of the remote nodes. We also ran our remote history command with...

Akka.NET vs. Kafka, RabbitMQ, and Other Messaging Systems

What's the difference between Akka.NET, Kafka, RabbitMQ, and other message-driven technologies? Can they work together?

A while back I created a thread on Twitter to attempt to explain the difference between Akka.NET and some other popular message-distribution and queuing technologies, such as Apache Kafka and RabbitMQ.

I’m going to cover that in some more detail in this post because it’s a common question asked by many developers who are just starting to look into Akka.NET.

Differences between Akka.NET, Kafka, RabbitMQ, and Other Messaging Systems

Message brokers, enterprise message buses, message queues, event hubs, and so on - for the sake of simplicity, which I’m certain will enrage developers in some corners of the Internet, I’m going to lump these technologies together into a single category: these are message distribution systems.

The manner in which they distribute messages varies and for our purposes those differences are totally immaterial (sorry, vendors.) The point is: producers write messages into these systems and the goal is, with varying degrees of reliability, concurrency, and asynchrony, to distribute these messages for processing to one or more downstream consumers.

Message distribution systems are transports - they aren’t involved in the act of creating or consuming messages. Only in delivering and routing these messages from their sources and to their destinations.

Where Akka.NET differs: Akka.NET actors are fundamentally message processing and message producing technologies.

Akka.NET actors are responsible for managing business state or executing commands, both of which occur when an actor receives a message.

Where the confusion occurs, however,...

Why You Should be Paying Attention to OpenTracing

How the Vendor-Neutral Tracing Standard Will Affect Frameworks and Application Development.

In a previous post we introduced distributed tracing and how it solves some of the worst DevOps problems that arise with the use of microservices. In this post we’re going to introduce the OpenTracing standard and talk how what this project does and why you should pay attention to it if you’re a product owner, software architect, or developer.

Middleware, Frameworks, and Instrumentation

Petabridge is committed to delivering world-class tools for developers to build high-performance, large scale software applications in .NET; thus, developing middleware and application programming frameworks, Akka.NET most notably, is a core part of our mission.

All web and Internet-connected applications depend on frameworks and middleware of some kind, because the overwhelming majority of companies who build software don’t want to be in the business of creating and maintaining infrastructure. This is why we have nice tools like ASP.NET Core, SignalR, Akka.NET, gRPC, DotNetty, NServiceBus, RabbitMQ, and so on.

These frameworks provide us with convenient abstractions that:

  • Significantly lower the cost of developing software in-house;
  • Improve the productivity of individual software developers; and
  • Allow for the formation and sharing of industry best practices and standardized designs.

In short, middleware makes the modern software economy possible.

How Asynchrony, Microservices, and Distributed Systems Changed the Economics

As the Internet has grown, there has been a tremendous amount of pressure on server-side software developers to build systems that are available 24/7/365, capable of working across a larger range of devices (desktop, mobile, embedded, etc…,) able to capture and utilize ever-increasing amounts of data, and capable of responding faster and faster to end-user requests. This pressure is what’s driven the adoption of technologies like the actor model, pushed frameworks like .NET to prioritize the development of asynchronous programming constructs such as the TPL, and driven...