Beyond HTTP

Open Source Distributed Systems in .NET

Akka.NET v1.5: No Hocon, No Lighthouse, No Problem

Exploring Akka.Hosting, Akka.HealthCheck, and Akka.Management

In our previous post we covered the Akka.NET v1.5 release and in particular, we focused on the changes made to the core Akka.NET modules.

In this blog post we’re going to cover the three new libraries we’ve added to Akka.NET as part of the v1.5 development effort:

  • Akka.Hosting - a new library that eliminates the need for HOCON configuration; offers deep + seamless integration with Microsoft.Extensions.Logging / Hosting / DependencyInjection / and Configuration; makes dependency injection a first-class citizen in Akka.NET with the ActorRegistry and IRequiredActor<T> types; implements ActorSystem life-cycle management best practices automatically; and makes it much easier to standardize and scale Akka.NET development across large development teams.
  • Akka.Management - a new library that provides automatic Akka.Cluster bootstrapping and environment-specific service discovery capabilities. Akka.Management eliminates the need for things like Lighthouse - clusters can instead be formed by querying environments like Kubernetes, Azure Table Storage, or Amazon Web Services EC2/ECS. This also enables Akka.Cluster to run in much lighter-weight PaaS environments such as Akka.NET on Azure App Service or Akka.NET on Azure Container Apps.
  • Akka.HealthCheck - the Akka.HealthCheck library has actually been around for a few years, but it’s been modernized to support Microsoft.Extensions.HealthCheck and includes automated liveness and readiness checks for Akka.Persistence and Akka.Cluster out of the box. It’s also very easy to write your own custom healthchecks.

Akka.Templates

Before we get into some of the details of Akka.Hosting / Management / HealthCheck, it’s worth noting that we just recently shipped Akka.Templates - a set of dotnet new templates for Akka.NET that will work not just on the dotnet command line interface, but will also act as “New Project” templates in any .NET IDE like Visual Studio or JetBrains Rider.

All of these templates use Akka.Hosting and the Akka.WebApi template uses everything. If you want to learn these new libraries, get started by installing Akka.Templates on your local machine:

dotnet new install "Akka.Templates::*"

And then use them to create a new project:

dotnet new akkawebapi -n "your project name"

Akka.Hosting

The premise behind Akka.Hosting is to streamline Akka.NET with the Microsoft.Extensions.* ecosystem: hosting, logging, dependency injection, and configuration.

With Akka.Hosting it is now possible to configure Akka.NET applications without any HOCON - even fairly complex ones.

Let’s take a look at some POCO configuration classes we use in our Akka.WebApi template, for instance:

public class AkkaSettings
{
    public string ActorSystemName { get; set; } = "AkkaWeb";

    public bool UseClustering { get; set; } = true;

    public bool LogConfigOnStart { get; set; } = false;

    public RemoteOptions RemoteOptions { get; set; } = new()
    {
        // can be overridden via config, but is dynamic by default
        PublicHostName = Dns.GetHostName()
    };

    public ClusterOptions ClusterOptions { get; set; } = new ClusterOptions()
    {
        // use our dynamic local host name by default
        SeedNodes = new[] { $"akka.tcp://AkkaWebApi@{Dns.GetHostName()}:8081" }
    };

    public ShardOptions ShardOptions { get; set; } = new ShardOptions();

    public PersistenceMode PersistenceMode { get; set; } = PersistenceMode.InMemory;

    public AkkaManagementOptions? AkkaManagementOptions { get; set; }
}

Many of these types - such as the RemoteOptions, ClusterOptions, and ShardOptions types are built directly into Akka.Hosting and can be parsed directly from an appSettings.json file, like so:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "AkkaSettings": {
    "ActorSystemName": "AkkaWebApi",
    "UseClustering": true,
    "RemoteOptions": {
      "HostName": "0.0.0.0",
      "Port": 8081
    },
    "ClusterOptions": {
      "Roles": [
        "web-api"
      ],
    },
    "ShardOptions": {
      "StateStoreMode": "DData",
      "RememberEntities": false,
      "Role": "web-api"
    },
    "AkkaManagementOptions": {
      "Enabled": false,
      "PortName": "management",
      "ServiceName": "akka-management",
      "RequiredContactPointsNr": 3,
      "DiscoveryMethod": "Config"
    },
    "PersistenceMode": "InMemory"
  }
}

Back inside our Program.cs, we consume our configuration sources:

var builder = WebApplication.CreateBuilder(args);

var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Development";

/*
 * CONFIGURATION SOURCES
 */
builder.Configuration
    .AddJsonFile("appsettings.json")
    .AddJsonFile($"appsettings.{environment}.json", optional: true)
    .AddEnvironmentVariables();

And then we can call the Akka.Hosting APIs directly on the IServiceCollection:

builder.Services.ConfigureWebApiAkka(builder.Configuration, (akkaConfigurationBuilder, serviceProvider) =>
{
    // we configure instrumentation separately from the internals of the ActorSystem
    akkaConfigurationBuilder.ConfigurePetabridgeCmd();
    akkaConfigurationBuilder.WithWebHealthCheck(serviceProvider);
});

// in AkkaConfiguration.cs
public static class AkkaConfiguration
{
    public static IServiceCollection ConfigureWebApiAkka(this IServiceCollection services, IConfiguration configuration,
        Action<AkkaConfigurationBuilder, IServiceProvider> additionalConfig)
    {
        var akkaSettings = configuration.GetRequiredSection("AkkaSettings").Get<AkkaSettings>();
        Debug.Assert(akkaSettings != null, nameof(akkaSettings) + " != null");

        services.AddSingleton(akkaSettings);

        return services.AddAkka(akkaSettings.ActorSystemName, (builder, sp) =>
        {
            builder.ConfigureActorSystem(sp);
            additionalConfig(builder, sp);
        });
    }

    // other configuration methods
}

It’s this key line that allows us to parse all of the configuration values we’re going to use for our Akka.Cluster application:

var akkaSettings = configuration.GetRequiredSection("AkkaSettings").Get<AkkaSettings>();

This is functionality that’s built into Micrsoft.Extensions.Configuration - parsing configuration sections into POCOs. We can, from this point onward, simply use the parsed value to configure all of the various facets of our ActorSystem using Akka.Hosting:

public static AkkaConfigurationBuilder ConfigureNetwork(this AkkaConfigurationBuilder builder,
    IServiceProvider serviceProvider)
{
    var settings = serviceProvider.GetRequiredService<AkkaSettings>();
    var configuration = serviceProvider.GetRequiredService<IConfiguration>();


    if (!settings.UseClustering)
        return builder;


    var b = builder
        .WithRemoting(settings.RemoteOptions);


    if (settings.AkkaManagementOptions is { Enabled: true })
    {
        // need to delete seed-nodes so Akka.Management will take precedence
        var clusterOptions = settings.ClusterOptions;
        clusterOptions.SeedNodes = Array.Empty<string>();


        b = b
            .WithClustering(clusterOptions)
            .WithAkkaManagement(hostName: settings.AkkaManagementOptions.Hostname,
                settings.AkkaManagementOptions.Port)
            .WithClusterBootstrap(serviceName: settings.AkkaManagementOptions.ServiceName,
                portName: settings.AkkaManagementOptions.PortName,
                requiredContactPoints: settings.AkkaManagementOptions.RequiredContactPointsNr);


        switch (settings.AkkaManagementOptions.DiscoveryMethod)
        {
            case DiscoveryMethod.Kubernetes:
                break;
            case DiscoveryMethod.AwsEcsTagBased:
                break;
            case DiscoveryMethod.AwsEc2TagBased:
                break;
            case DiscoveryMethod.AzureTableStorage:
            {
                var connectionStringName = configuration.GetSection("AzureStorageSettings")
                    .Get<AzureStorageSettings>()?.ConnectionStringName;
                Debug.Assert(connectionStringName != null, nameof(connectionStringName) + " != null");
                var connectionString = configuration.GetConnectionString(connectionStringName);


                b = b.WithAzureDiscovery(options =>
                {
                    options.ServiceName = settings.AkkaManagementOptions.ServiceName;
                    options.ConnectionString = connectionString;
                });
                break;
            }
            case DiscoveryMethod.Config:
                break;
            default:
                throw new ArgumentOutOfRangeException();
        }
    }
    else
    {
        b = b.WithClustering(settings.ClusterOptions);
    }


    return b;
}

Using Production Configuration Methods During Testing with Akka.Hosting.TestKit

One of the best practices that Akka.Hosting lends itself towards is creating strongly typed, resuable configuration methods that are implemented as extension methods of the AkkaConfigurationBuilder type:

public static AkkaConfigurationBuilder ConfigureCounterActors(this AkkaConfigurationBuilder builder,
    IServiceProvider serviceProvider)
{
    var settings = serviceProvider.GetRequiredService<AkkaSettings>();
    var extractor = CreateCounterMessageRouter();

    if (settings.UseClustering)
    {
        return builder.WithShardRegion<CounterActor>("counter",
            (system, registry, resolver) => s => Props.Create(() => new CounterActor(s)),
            extractor, settings.ShardOptions);
    }
    else
    {
        return builder.WithActors((system, registry, resolver) =>
        {
            var parent =
                system.ActorOf(
                    GenericChildPerEntityParent.Props(extractor, s => Props.Create(() => new CounterActor(s))),
                    "counters");
            registry.Register<CounterActor>(parent);
        });
    }
}

This method configures the CounterActor type that we ship with the “AkkaWebApi” dotnet new template - and it does some things that would have been very difficult to do in Akka.NET with traditional HOCON configuration.

For starters, notice the settings.UseClustering call? We can toggle major Akka.NET features on or off and change the way we run our application using the POCO classes we defined earlier - this means that I can use this exact same ConfigureCounterActors extension method during unit testing with the Akka.Hosting.TestKit and disable clustering in order to make test execution and setup significantly easier:

public class CounterActorSpecs : TestKit
{
    public CounterActorSpecs(ITestOutputHelper output) : base(output:output)
    {
    }
    
    
    protected override void ConfigureServices(HostBuilderContext context, IServiceCollection services)
    {
        var settings = new AkkaSettings() { UseClustering = false, PersistenceMode = PersistenceMode.InMemory };
        services.AddSingleton(settings);
        base.ConfigureServices(context, services);
    }

    protected override void ConfigureAkka(AkkaConfigurationBuilder builder, IServiceProvider provider)
    {
        builder.ConfigureCounterActors(provider).ConfigurePersistence(provider);
    }
    
    [Fact]
    public void CounterActor_should_follow_Protocol()
    {
        // arrange (counter actor parent is already running)
        var counterActor = ActorRegistry.Get<CounterActor>();
        var counterId1 = "counterId";
        var counter1Messages = new IWithCounterId[]
        {
            new SetCounterCommand(counterId1, 3),
            new IncrementCounterCommand(counterId1, 10),
            new IncrementCounterCommand(counterId1, -5),
            new IncrementCounterCommand(counterId1, 2),
      
            new FetchCounter(counterId1)
        };

        // act

        foreach (var msg in counter1Messages)
        {
            counterActor.Tell(msg, TestActor);
        }

        // assert
        var counter = (Counter)FishForMessage(c => c is Counter);
        counter.CounterId.Should().Be(counterId1);
        counter.CurrentValue.Should().Be(3+10-5+2);
    }
}

Notice that we can just manually allocate an AkkaSettings instance, bind it to our test IHost’s IServiceCollection, and run the test without waiting to make sure Akka.Cluster.Sharding or Akka.Cluster is running? This helps trivialize configuration in Akka.NET - a good thing.

Actor Dependency Injection with IRequiredActor<T> and ActorRegistry

The ActorRegistry was introduced in the very first version of Akka.Hosting, but in the newest releases of Akka.Hosting we’ve complimented it with a new type: IRequiredActor<T>.

Suppose we have a class that depends on having a reference to a top-level actor, a router, a ShardRegion, or perhaps a ClusterSingleton (common types of actors that often interface with non-Akka.NET parts of a .NET application):

public sealed class MyConsumer
{
    private readonly IActorRef _actor;

    public MyConsumer(IRequiredActor<MyActorType> actor)
    {
        _actor = actor.ActorRef;
    }

    public async Task<string> Say(string word)
    {
        return await _actor.Ask<string>(word, TimeSpan.FromSeconds(3));
    }
}

The IRequiredActor<MyActorType> will cause the Microsoft.Extensions.DependencyInjection mechanism to resolve MyActorType from the ActorRegistry and inject it into the IRequired<Actor<MyActorType> instance passed into MyConsumer.

The IRequiredActor<TActor> exposes a single property:

public interface IRequiredActor<TActor>
{
    /// <summary>
    /// The underlying actor resolved via <see cref="ActorRegistry"/> using the given <see cref="TActor"/> key.
    /// </summary>
    IActorRef ActorRef { get; }
}

By default, you can automatically resolve any actors registered with the ActorRegistry without having to declare anything special on your IServiceCollection:

using var host = new HostBuilder()
  .ConfigureServices(services =>
  {
      services.AddAkka("MySys", (builder, provider) =>
      {
          builder.WithActors((system, registry) =>
          {
              var actor = system.ActorOf(Props.Create(() => new MyActorType()), "myactor");
              registry.Register<MyActorType>(actor);
          });
      });
      services.AddScoped<MyConsumer>();
  })
  .Build();
  await host.StartAsync();

Adding your actor and your type key into the ActorRegistry is sufficient - no additional DI registration is required to access the IRequiredActor<TActor> for that type.

Akka.HealthCheck

Akka.HealthCheck has existed in some shape for a few years - but we gave it a massive facelift in Akka.NET v1.5 so we could integrate with the Microsoft.Extensions.HealthCheck ecosystem. It goes without saying: you should really use Akka.HealthCheck in combination with Akka.Hosting.

We can do this by installing the Akka.HealthCheck.Hosting.Web NuGet package

dotnet add package Akka.HealthCheck.Hosting.Web

And then wiring it up into our Akka.NET application via the following:

// Add services to the container.
builder.Services.WithAkkaHealthCheck(HealthCheckType.All);
builder.Services.ConfigureWebApiAkka(builder.Configuration, (akkaConfigurationBuilder, serviceProvider) =>
{
    // we configure instrumentation separately from the internals of the ActorSystem
    akkaConfigurationBuilder.ConfigurePetabridgeCmd();
    akkaConfigurationBuilder.WithWebHealthCheck(serviceProvider);
});

builder.Services.AddControllers();
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment() || app.Environment.EnvironmentName.Equals("Azure"))
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();
app.MapAkkaHealthCheckRoutes(optionConfigure: (_, opt) =>
{
    // Use a custom response writer to output a json of all reported statuses
    opt.ResponseWriter = Helper.JsonResponseWriter;
}); // needed for Akka.HealthCheck
  • builder.Services.WithAkkaHealthCheck(HealthCheckType.All); (required) - registers either built-in or custom Akka.NET healthchecks into the Microsoft.Extensions.HealthCheck HealthCheckService.
  • akkaConfigurationBuilder.WithWebHealthCheck(serviceProvider); (required) - configures the actors inside Akka.HealthCheck that will actually run the appropriate checks.
  • app.MapAkkaHealthCheckRoutes (optional) - if you’re using ASP.NET Core, this will map the Akka.HealthCheck probes’ output to the http://{host:port}/healthz/akka route. If any of the healthchecks are failing you’ll receive an HTTP error code and a JSON response indicating which of the possible healthchecks are failing.

There are other healthcheck transports available other than HTTP - see the Akka.HealthCheck documentation for details.

Built-in HealthChecks

Akka.HealthCheck operates on a Kubernetes-style liveness / readiness health check distinction:

  • Liveness check - failure means the process is inoperable and must be terminated + restarted;
  • Readiness check - failure means the process is not ready to receive traffic; it’s the equivalent of a load-balancer healthcheck.

And we offer the following built-in healthchecks, and it’s quite easy to add your own custom Akka.NET healthchecks:

  • AkkaLiveness - ensures that the ActorSystem is running;
  • AkkaReadiness - same as the liveness probe;
  • ClusterLiveness - Akka.Cluster has started;
  • ClusterReadiness - we have joined a cluster and all nodes are Reachable;
  • PersistenceLiveness - we can successfully read and write from Akka.Persistence.

Akka.Management

Lastly, we have Akka.Management. Akka.Management designed to help assist Akka.Cluster issues with cluster formation and discovery by replacing the akka.cluster.seed-nodes approach to using a data-driven service discovery approach instead.

Akka.Discovery

The first component from the Akka.Management ecosystem we should consider is Akka.Discovery - a generic API that allows nodes to self-report their own cluster connectivity status + address and query the same data for others.

By default we expose a configuration-based approach to discovery that is not too dissimilar to how akka.cluster.seed-nodes works, but the real value is in the more automated plugins:

Akka.Discovery is just one piece of this - it’s the source of the data we use to run the Akka.Cluster bootstrapping process built into the Akka.Management NuGet package itself.

Cluster Bootstrapping with Akka.Management

To get any value out of Akka.Management, we have to install the Akka.Management NuGet package itself as well as at least one Akka.Discovery plugin - per our Akka.WebApi template:

<PackageReference Include="Akka.Cluster.Hosting" />
<PackageReference Include="Akka.Discovery.Azure" />
<PackageReference Include="Akka.Management" />

Ok, that’ll do it.

Now how does it work? You should really, really watch “No Lighthouse, No Hocon, No Problem (27:41)” to get an answer to that question, but I’ll do my best to summarize here.

  • Akka.Management exposes what should be a non-public HTTP port and accepts HTTP requests that arrive from other nodes - this is essential to the bootstrapping process, as it’s used to determine whether or not there is already a cluster, who is in it, and which nodes we’ve found via Akka.Discovery are still alive and responsive right now;
  • We query our Akka.Discovery source and retrieve a list of potential nodes to contact; and
  • We contact each node via the public HTTP API they expose - and we will form a cluster as long as there are at least 3 nodes (configurable) waiting to join. If any of the nodes who’ve replied back are already in a cluster, we will join theirs instead.

Here’s what the configuration for this looks like inside the WebApi template:

public static AkkaConfigurationBuilder ConfigureNetwork(this AkkaConfigurationBuilder builder,
    IServiceProvider serviceProvider)
{
    var settings = serviceProvider.GetRequiredService<AkkaSettings>();
    var configuration = serviceProvider.GetRequiredService<IConfiguration>();

    if (!settings.UseClustering)
        return builder;

    var b = builder
        .WithRemoting(settings.RemoteOptions);

    if (settings.AkkaManagementOptions is { Enabled: true })
    {
        // need to delete seed-nodes so Akka.Management will take precedence
        var clusterOptions = settings.ClusterOptions;
        clusterOptions.SeedNodes = Array.Empty<string>();

        b = b
            .WithClustering(clusterOptions)
            .WithAkkaManagement(hostName: settings.AkkaManagementOptions.Hostname,
                settings.AkkaManagementOptions.Port)
            .WithClusterBootstrap(serviceName: settings.AkkaManagementOptions.ServiceName,
                portName: settings.AkkaManagementOptions.PortName,
                requiredContactPoints: settings.AkkaManagementOptions.RequiredContactPointsNr);

        switch (settings.AkkaManagementOptions.DiscoveryMethod)
        {
            case DiscoveryMethod.Kubernetes:
                break;
            case DiscoveryMethod.AwsEcsTagBased:
                break;
            case DiscoveryMethod.AwsEc2TagBased:
                break;
            case DiscoveryMethod.AzureTableStorage:
            {
                var connectionStringName = configuration.GetSection("AzureStorageSettings")
                    .Get<AzureStorageSettings>()?.ConnectionStringName;
                Debug.Assert(connectionStringName != null, nameof(connectionStringName) + " != null");
                var connectionString = configuration.GetConnectionString(connectionStringName);

                b = b.WithAzureDiscovery(options =>
                {
                    options.ServiceName = settings.AkkaManagementOptions.ServiceName;
                    options.ConnectionString = connectionString;
                });
                break;
            }
            case DiscoveryMethod.Config:
                break;
            default:
                throw new ArgumentOutOfRangeException();
        }
    }
    else
    {
        b = b.WithClustering(settings.ClusterOptions);
    }

    return b;
}

The key to making this work correctly is ensuring that the settings.AkkaManagementOptions.ServiceName is the same in both Akka.Management and WithAkkaManagement and WithClusterBootstrap - this ensures that Akka.Discovery and Akka.Management are both looking for the same thing. If these two names are different, the service will never form!

Learn More

Once again, I’ll refer to our “No Lighthouse, No Hocon, No Problem” video on Petabridge’s YouTube channel - it’s 40+ minutes long, organized into chapters, and very detailed.

In addition to that - the Akka.Hosting / Akka.HealthCheck / Akka.Management GitHub repositories themselves are chock-full of information.

Lastly, you can always contact us on the Akka.NET Discord chat with questions!

If you liked this post, you can share it with your followers or follow us on Twitter!
Written by Aaron Stannard on March 23, 2023

 

 

Observe and Monitor Your Akka.NET Applications with Phobos

Phobos - instant OpenTelemetry observability for Akka.NET

Did you know that Phobos can automatically instrument your Akka.NET applications with OpenTelemetry?

Click here to learn more.


Petabridge

Call us: 1-866-418-3140

Follow us on…

follow us in feedly

© 2015-2024 Petabridge, LLC