// AKKA.STREAMS.KAFKA

85% Less Code Than Raw Confluent.Kafka

Akka.Streams.Kafka sits on top of Confluent.Kafka and eliminates the distributed systems boilerplate. Built-in backpressure, supervision strategies for error handling, transparent partition rebalancing, and compositional pipeline design.

NuGet Package Full Comparison

// WHY_NOT_RAW_CONFLUENT

What You Get Automatically

Built-in Backpressure

Reactive Streams implementation automatically pauses Kafka polling when downstream processing (database writes, API calls) can't keep up. No manual semaphores or flow control needed.

Supervision Strategies

Declarative error policies — retry, skip, or stop — replace scattered try-catch blocks with manual retry logic. Each stage can have independent error handling.

Transparent Rebalancing

Automatically handles in-flight message invalidation, offset commits during partition rebalancing, and race condition prevention. Dozens of lines of complex callback code in raw Confluent.Kafka become automatic.

Compositional Pipelines

Build processing graphs by composing sources, flows, and sinks. Parallel processing, batching, and database writes — each stage with independent parallelism settings.

// SIDE_BY_SIDE

350 Lines vs 50 Lines

For identical order processing functionality, raw Confluent.Kafka requires approximately 350 lines of code. Akka.Streams.Kafka does it in 50. Explore the demo repo →

Concern	Confluent.Kafka	Akka.Streams.Kafka
Backpressure	Manual	✓ Automatic
Error retries	Manual	✓ Automatic
Dead letter queue	Manual	✓ Automatic
Partition rebalancing	Manual	✓ Automatic
Graceful shutdown	Manual	✓ Automatic
Offset management	Manual	✓ Automatic

// IN_CODE

Parallel Processing with Ordered Offset Commits

4 parallel workers, partition order preserved, offsets committed only after processing succeeds. If processing fails, the offset doesn't advance and the message gets retried.

KafkaConsumer.cs

KafkaConsumer.CommittableSource(consumerSettings,
        Subscriptions.Topics("orders"))
    .SelectAsync(4, async msg =>              // 4 parallel workers
    {
        var order = Deserialize(msg.Record.Value);
        await ProcessOrder(order);            // async processing
        return msg.CommitableOffset;          // carry offset through
    })                                        // preserves partition order
    .ToMaterialized(
        Committer.Sink(committerSettings),    // commit only after processing
        Keep.Both)
    .Run(materializer);

// DEEP_DIVE

Watch the Full Walkthrough

// ARCHITECTURE

Akka.Streams.Kafka does not replace Confluent.Kafka — it sits on top of it.

Akka.Streams.Kafka → Confluent.Kafka → librdkafka → Kafka Protocol

It abstracts away the "how" so you can focus on the "what."