Since announcing Phobos 1.0 a year ago we’ve made many significant changes to it that should add many useful new ways to observe the run-time behavior of your Akka.NET applications in production.

Phobos is a commercial add-on for Akka.NET that allows developers to automatically record distributed traces and metrics from their actors without the need for any manual instrumentation code.

In this post we’re going to cover:

• Improvements that have been made to Phobos all the way up through the v1.3.2 version released on December 21st, 2021;
• Phobos 2.0, which will be built entirely upon OpenTelemetry and should be in beta soon; and
• Our ongoing support plan for Phobos 1.x and 2.x over the next two years.

Phobos 1.x Improvements

As of writing this the current version of Phobos is v1.3.2 and we’ve made several important changes to the product over the course of 2021:

Phobos 1.1

We made two major changes in Phobos 1.1 at the beginning of 2021:

• Automatically record all Akka.Cluster node membership data when using Phobos.Actor.Cluster and
• Added several built-in Phobos dashboards for platforms like Grafana, Datadog, and Application Insights to help visualize all automatically captured data from Phobos.

An example of what this cluster data looks like via our Prometheus + Grafana dashboard:

These changes make it significantly easier to set alerts when your cluster has unreachable nodes for a lengthy period of time, see how many nodes are currently joining / leaving, and it also makes it possible for us to filter our metrics by the Akka.Cluster role or node address.

Phobos 1.2

As of Phobos 1.2.5 we made several significant tracing improvements and fixed some key bugs:

1. Correctly traces all messages sent through routers, including remotely deployed routers or routers created using Akka.Cluster.Metrics;
2. Greatly reduced memory pressure and chattiness when tracing high throughput areas of Akka.NET, such as the DistributedPubSub.Mediator;
3. Added support for tracing Ask<T> operations, including being able to trace their timeouts and observed duration; and
4. Added some additional tooling to trace filtering which allows developers to ignore spans that are already part of an active trace, if necessary.

Phobos 1.3

We’re still actively working on Phobos 1.3, with 1.3.2 just being released yesterday, and there are some major improvements in this release too:

1. Correlates async operations that are awaited inside an actor’s ReceiveAsync block during traces;
2. PipeTo operations are now traced correctly so long as the actor initiating the detached Task<T> passes a reference to itself as the sender on the PipeTo method; and
3. Added a new akka.messages.latency metric that automatically tracks the latency requires to process each message / actor type combination.

We also updated all of Phobos’ built-in dashboards to include the akka.messages.latency metrics, and here’s what that looks like in Grafana:

If you’re a current Phobos customer we’d strongly encourage you to upgrade to the latest to take advantage of these changes - and if you’re not a Phobos customer then you should consider purchasing a license.

Phobos 2.0 and OpenTelemetry

Over on my personal blog I published “An Overview of Distributed Tracing with OpenTelemetry in .NET 6,” which describes some of the new tracing + metrics capabilities that are becoming mainstream in .NET application programming as a result of the new OpenTelemetry standard becoming a first-class citizen within the platform.

In Phobos 1.x we opted to use a combination of OpenTracing and App.Metrics to provide observability to our customers because back in June 2020, when we first developed Phobos 1.0, OpenTelemetry was in a very rough alpha state. As of .NET 6, released in November 2021, OpenTelemetry tracing is now production ready and OpenTelemetry metrics should be released to market in early 2022 - therefore, we feel confident that we can build future versions of Phobos on top of OpenTelemetry going forward.

And that’s exactly what we plan to do with Phobos 2.0 and beyond.

We’re going to replace OpenTracing and App.Metrics with OpenTelemetry tracing and metrics respectively.

In terms of timeframe, we’re hoping to have the first rough beta of Phobos 2.0 ready by 12/31/2021 - that’s what we’ve been telling our customers who’ve asked and even though there isn’t much 2021 left on the calendar we’re going to try to hit that target.

Update: 12/31/21 - as of today we’ve not delivered a beta of Phobos OTel just yet; as it turns out, rewriting our verification test suite for tracing in OpenTelemetry is a much bigger project than we anticipated and we don’t want to release the library (which is fully ported) without completing this milestone. We’ll provide an update on the state of things in our webinar on January 27th, 2022 if not sooner.

In terms of a full-fledged Phobos 2.0 implementation with full tutorials, documentation, dashboards, and metrics? That will depend on when all of the OpenTelemetry packages are released to market, but we feel confident that our work on this front should be complete by end of April, 2022.

Phobos 1.x and 2.x Lifetimes

So what’s in-store for Phobos 1.x in the context of 2.x coming around the corner?

Our plan is to concurrently support Phobos 1.x and 2.x all the way through the Akka.NET v1.4 and 1.5 lifecycles, which should last through the end of 2023.

We will continue to add updates to Phobos 1.x for fixing core actor tracing and metrics, but we will likely not address any APM-vendor-specific issues. I.e. “my graphs in Datadog look different than they do in Jaeger.” We’re going to direct users to Phobos 2.0 as OpenTelemetry enjoys official, ongoing support in ways that App.Metrics and older driver implementations will not. New customers should use Phobos 2.0 by default going forward once it becomes available.

Webinar: Observability in Akka.NET Applications with OpenTelemetry and Phobos

If you’re interested in learning how to observe the dynamic, run-time behavior of your Akka.NET applications using OpenTelemetry then we have an upcoming webinar on Thursday, January 27th 2022 at 10:30am CST that will be right up your alley.

Click here to register for “Observability in Akka.NET Applications with OpenTelemetry and Phobos.”

This webinar will last about an hour and in it we’ll cover:

• OpenTelemetry concepts and basics;
• How to consume OpenTelemetry through APM providers such as DataDog, Prometheus, and so on;
• A preview of Phobos 2.0; and
• What types of behavior you can observe in your Akka.NET applications through it.

If you aren’t able to attend live you should still register anyway as we’ll email out a recording to all attendees after the Webinar airs.

Thanks for all of your support this year and we’ll see you all in 2022!

If you liked this post, you can share it with your followers or follow us on Twitter!