• Using the New Akka.Hosting Health Checks
  • Enabling Health Checks
  • Customizing Health Check Responses
  • Filtering Health Checks by Tag
  • What About Health Checks in Non-Web Applications?
• Why We Deprecated Akka.HealthChecks
  • Design Problems
  • Why a New Library?
• Migrating from Akka.HealthChecks to Akka.Hosting’s Health Checks
  • Upcoming Health Check Functionality from Akka.Hosting

We have just recently shipped Akka.Hosting v1.5.48.1 and newer, all of which now have built-in Microsoft.Extensions.HealthCheck integration that is significantly easier to configure, fewer packages to install, and easier to customize than what we had with Akka.HealthChecks.

This post consists of three parts:

1. How to use the new Akka.Hosting health checks;
2. Why we deprecated Akka.HealthChecks; and
3. Migration recommendations for existing Akka.HealthChecks users - because the new Akka.Hosting health checks cause conflicts with the older ones.

Let’s dive in.

Using the New Akka.Hosting Health Checks

Here’s an example of what a new health check looks like in Akka.Hosting, using the AkkaConfigurationBuilder:

builder
    .WithActorSystemLivenessCheck() // have to opt-in to the built-in health check
    .WithHealthCheck("FooActor alive", async (system, registry, cancellationToken) =>
{
    /*
     * N.B. CancellationToken is set by the call to MSFT.EXT.DIAGNOSTICS.HEALTHCHECK,
     * so that value could be "infinite" by default.
     *
     * Therefore, it might be a really, really good idea to guard this with a non-infinite
     * timeout via a LinkedCancellationToken here.
     */
    try
    {
        var fooActor = await registry.GetAsync<FooActor>(cancellationToken);

        try
        {
            var r = await fooActor.Ask<ActorIdentity>(new Identify("foo"), 
            	cancellationToken: cancellationToken);
            if (r.Subject.IsNobody())
                return HealthCheckResult.Unhealthy("FooActor was alive but is now dead");
        }
        catch (Exception e)
        {
            return HealthCheckResult.Degraded("FooActor found but non-responsive", e);
        }
    }
    catch (Exception e2)
    {
        return HealthCheckResult.Unhealthy("FooActor not found in registry", e2);
    }

    return HealthCheckResult.Healthy("fooActor found and responsive");
});

The WithActorSystemLivenessCheck() does exactly what you think it does: registers a liveness check that returns Unhealthy if the ActorSystem is terminated. But the more interesting thing is the second call - WithHealthCheck.

This method allows users to define custom health checks using either an IAkkaHealthCheck implementation or a delegate function that takes an ActorSystem, an ActorRegistry, and a CancellationToken.

Enabling Health Checks

You can add health checks to your AkkaConfigurationBuilder via the WithHealthCheck method, but in order to get these to be effective you need to enable the HealthCheckService, part of Microsoft.Extensions.HealthChecks. This is easy:

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHealthChecks(); // enable the health check service

// Akka.NET config, other services, etc

var app = builder.Build();

// other HTTP Routes

app.MapHealthChecks("/healthz"); // health check endpoint

app.Run();

This will give you an HTTP endpoint at http(s)://0.0.0.0:{port}/healthz which will return a 200 if all health checks pass and a 400/500 error code if some of the health checks are “unhealthy.”

See Akka.Hosting.Asp.LoggingDemo for the full source code.

Customizing Health Check Responses

Let’s say we want to get a full report of everything that is causing our service’s health to report back as degraded or unhealthy - we can do that by customizing the MapHealthChecks call:

app.MapHealthChecks("/healthz", new HealthCheckOptions
{
    Predicate = _ => true, // include all checks
    ResponseWriter = async (ctx, report) =>
    {
        ctx.Response.ContentType = "application/json; charset=utf-8";

        var payload = new
        {
            status = report.Status.ToString(),
            totalDuration = report.TotalDuration,
            checks = report.Entries.Select(e => new
            {
                name = e.Key,
                status = e.Value.Status.ToString(),
                duration = e.Value.Duration,
                description = e.Value.Description,
                tags = e.Value.Tags,
                data = e.Value.Data   // anything you added via context.Registration
            })
        };

        await ctx.Response.WriteAsync(JsonSerializer.Serialize(payload, new JsonSerializerOptions { WriteIndented = true }));
        // or in .NET 8+: await ctx.Response.WriteAsJsonAsync(payload);
    }
});

Here’s what you’ll see when running the Akka.Hosting.Asp.LoggingDemo sample:

{
  "status": "Healthy",
  "totalDuration": "00:00:00.0163664",
  "checks": [
    {
      "name": "akka.cluster.join",
      "status": "Healthy",
      "duration": "00:00:00.0058347",
      "description": "Observed successful cluster join after [0:01:57.4184716] - actual join duration was probably faster, but this is how quickly the health check observed it.",
      "tags": [
        "ready",
        "akka.cluster",
        "akka"
      ],
      "data": {}
    },
    {
      "name": "akka.actorsystem",
      "status": "Healthy",
      "duration": "00:00:00.0022848",
      "description": "ActorSystem is running.",
      "tags": [
        "akka"
      ],
      "data": {}
    },
    {
      "name": "di-test",
      "status": "Healthy",
      "duration": "00:00:00.0011939",
      "description": "Test is healthy",
      "tags": [
        "test",
        "custom",
        "akka"
      ],
      "data": {}
    }
  ]
}

If you have other non-Akka.NET health checks, such as Entity Framework Core or other custom ones, they will also appear in this list.

Filtering Health Checks by Tag

This is covered in the official ASP.NET Core health check documentation, but you can also filter health checks by tag if you wanted to have separate endpoints for “liveness” vs. “readiness” checks¹, which is common in the Kubernetes ecosystem for instance:

app.MapHealthChecks("/healthz/ready", new HealthCheckOptions
{
    Predicate = healthCheck => healthCheck.Tags.Contains("ready")
});

app.MapHealthChecks("/healthz/live", new HealthCheckOptions
{
    Predicate = healthCheck => healthCheck.Tags.Contains("live")
});

app.MapHealthChecks("/healthz/akka", new HealthCheckOptions
{
    Predicate = healthCheck => healthCheck.Tags.Contains("akka")
});

What About Health Checks in Non-Web Applications?

Many users run Akka.NET as headless services, which typically means execution as a simple console application (with process supervision on top) or a Windows / Linux service without exposing any sort of public network communications interface.

How can you publish health checks inside these non-ASP.NET Core services?

There are two practical ways of accomplishing this:

1. Convert your headless application to use the WebApplication instead of a generic IHostBuilder so you can expose a /healthz HTTP route for health checks. This is my preferred option for applications that have to run in public cloud environments like Azure.
2. Write a custom IHealthCheckPublisher that can push health check notifications to other sources. This is part of the Microsoft.Extensions.HealthChecks library.

Why We Deprecated Akka.HealthChecks

Akka.HealthCheck is a package I wrote in early 2019 either during, or shortly after, an onsite training with one of our customers in the UK. They were running a large real-time graph search system built on top of Akka.NET and wanted Kubernetes-style liveness checks and readiness checks - and this included support for non-HTTP health check transports, such as file-based or socket-based health checks.

In the span of a few days I had built:

1. Built-in Akka.Cluster and Akka.Persistence health checks;
2. Configurable transports: HTTP / socket / cmd or file; and
3. Made the system reasonably extensible.

A few years later we introduced Akka.Hosting, which made it possible for us to tie into Microsoft.Extensions.HealthChecks. Thus, we integrated Akka.HealthChecks with Akka.Hosting as a major feature of our Akka.NET v1.5 roll-out- which led to a massive surge in adoption for Akka.HealthChecks.

Design Problems

It was after this massive surge of adoption that we discovered some pretty fundamental flaws in how Akka.HealthChecks is designed:

1. Most health checks were way too aggressive - this is especially true of our clustering health checks, which would interfere with the ability of tools like the Split Brain Resolver to help Akka.NET applications recover from intermittent failures.
2. Many health checks were fragile and complex - our Akka.Persistence health checks were to blame here. The intent was good: test all of the Akka.Persistence actor functionality periodically and report back if any of those operations failed. But this resulted in numerous bugs, such as our SuicideProbe actor filling users’ journals and snapshot stores with junk data (which we fixed), the probe never resetting back to “healthy” after recovering, and so on.
3. Difficult to customize - one of the things I really appreciate about Microsoft.Extensions.HealthCheck is how unopinionated it is. Do you want to treat HealthStatus.Degraded as a mere warning or do you want it to trigger a restart? You can customize that! Akka.HealthChecks made this type of customization difficult to do.
4. Difficult to configure - configuring Akka.HealthChecks requires 3 separate API calls for full Microsoft.Extensions.HealthCheck integration: the IServiceCollection, the AkkaConfigurationBuilder, and the WebApplication. All of these have custom configuration options and it is not immediately clear how they interact with each other.
5. Too many different NuGet packages - the full Akka.HealthChecks integration with ASP.NET Core health checks requires 5-6 different packages to be installed, in addition to Akka.Hosting, which is already taken as a dependency.

Why a New Library?

We wanted to pivot from this and offer an experience that “just works” by default. Akka.Hosting’s health checks are designed to be:

• Much simpler to configure;
• Easier to customize;
• Less opinionated / aggressive about signaling failure; and
• Require no additional NuGet packages beyond the ones you already need to configure your application with Akka.Hosting.

Introducing this functionality directly into Akka.Hosting achieves all of these goals with a minimal amount of fuss.

Migrating from Akka.HealthChecks to Akka.Hosting’s Health Checks

The migration process is really simple:

1. Uninstall all of the Akka.HealthCheck NuGet packages from your project - otherwise, these will cause conflicts that will cause your application to crash at startup if you upgrade to Akka.Hosting v1.5.48.1 or later.
2. Use the new built-in health checks inside Akka.Hosting or port your own custom ones, if you had any before.
3. Ensure that you are exposing the Microsoft.Extensions.HealthCheck output at the appropriate destination, either using the MapHealthChecks method for inbound HTTP routes or a custom IHealthCheckPublisher for push-based health checks.

Upcoming Health Check Functionality from Akka.Hosting

This section will be subject to change because the two issues I’m going to describe here are works in progress.

1. Akka.Persistence health checks - these were a very popular feature from Akka.HealthChecks that are currently not implemented in Akka.Hosting. This is being addressed here: https://github.com/akkadotnet/akka.net/issues/7840 and should ship as part of Akka.Hosting v1.5.51.
2. IAkkaHealthChecks with dependency injection support - this isn’t something we even supported in Akka.HealthChecks, but it was requested by end-users, and that is already merged in thanks to third party contributors: https://github.com/akkadotnet/Akka.Hosting/pull/659. This will also ship in Akka.Hosting for v1.5.51.

Please let us know if there’s anything else we need to include!

Observe and Monitor Your Akka.NET Applications with Phobos

Did you know that Phobos can automatically instrument your Akka.NET applications with OpenTelemetry?

Click here to learn more.