beave-rs

Kafka adapter

The optional kafka feature provides typed Kafka source and sink adapters backed by rdkafka.

beavers = { version = "0.1", features = ["kafka"] }

Configuration

A Kafka handle holds the bootstrap brokers and the librdkafka settings that every client built from it uses. property sets a librdkafka property on every consumer and producer of the handle, producer_property on its producers only, and sasl_plain configures SASL/PLAIN authentication, over TLS unless security.protocol is set otherwise. handle.source::<C>(group_id, topics) builds a source and handle.sink::<C>(topic) a sink that publishes every output to topic; source_with_codec and sink_with_codec take a codec value as their first argument instead of using the codec's default. Optional settings are builder methods on the source and sink, such as property on a source for consumer-only settings like auto.offset.reset, or max_pending on a sink.

use beavers::{Utf8, adapters::kafka::{Kafka, KafkaPublish}};

let kafka = Kafka::new("localhost:9092").property("client.id", "orders-service");
let source = kafka
    .source::<Utf8>("orders-workers", ["orders"])
    .property("auto.offset.reset", "earliest");
let sink = kafka.sink::<Utf8>("processed-orders").max_pending(500);

let output = KafkaPublish::new("created".to_owned());

Every non-transactional sink built from one handle publishes through one shared producer, created on the first publication of any of them. Each source has a consumer of its own, because a consumer belongs to one consumer group, and each transactional sink has a producer of its own. Each builder method on the handle returns a handle with a producer of its own, so configure the handle before building sources and sinks from it.

The handle, the source, and the sink each have validate, which checks the values without contacting a broker. Sources and sinks validate their settings when first used, then create their Kafka clients lazily, so an application can construct its pipeline before a broker is reachable. The handle's Debug output shows property names only, because their values can hold credentials.

Records

KafkaRecord<T> is the decoded input, including immutable delivery metadata; KafkaPublish<T> is the user-controlled output accepted by the sink. The value is decoded and encoded by the selected codec. key and header values remain bytes. A Kafka null payload becomes KafkaRecord { value: None, .. }, which supports tombstones without inventing a sentinel value. Records decoded from a source carry KafkaMetadata with topic, partition, offset, and timestamp. KafkaPublish has no source metadata, so input metadata is never inherited implicitly. The sink always publishes to its configured topic and lets Kafka choose a partition from the explicit key. It does not copy a source partition, offset, timestamp, or topic into output.

The raw form of a KafkaMessage is a KafkaRecord<Vec<u8>> with the undecoded value bytes. Dead letters carry it, so the original value, key, headers, and delivery metadata survive even when decoding fails. KafkaRecord implements Serialize when its value does.

Acknowledgements and ordering

The source disables Kafka auto-commit and auto-offset-store. A successful ACK records a completed delivery and commits up to the first unfinished delivery of that topic partition, or past the last received delivery when all are finished. Because Kafka delivers a partition in offset order, offsets missing between received deliveries, such as compacted records or transaction markers, do not hold the commit back. A completion after an earlier in-flight offset cannot advance the commit. Broker commit failures leave completed local progress in place, so a later acknowledgement can retry the same position.

Each delivery's ordering key is its topic partition. Under the default distribution, SourceOrder, a subscription processes one record at a time per partition, in offset order, while different partitions run in parallel up to the subscription's concurrency. KafkaDistribution names the choices for Kafka records:

KafkaDistribution Processed one at a time
Partition (default) The records of one topic partition, as SourceOrder
MessageKey The records of one record key within a topic partition; records with a null key one at a time per partition
use beavers::adapters::kafka::KafkaDistribution;

Subscription::new("orders", source, sink, handler)
    .concurrency(16)
    .distribution(KafkaDistribution::MessageKey);

A producer's default partitioner sends the records of one key to one partition, so MessageKey keeps each key in offset order while different keys of a busy partition run in parallel. Under MessageKey, Parallel, or a distribution by a value of the payload, records of one partition can complete out of order; the commit still never skips an unfinished record.

Each partition assignment has its own generation and revocation token. When Kafka revokes a partition, or the adapter detects that its assignment was lost, the token is cancelled. The runtime abandons that partition's running and queued deliveries without ACK or subscription failure; handlers are not notified. Acknowledgements from an old generation are rejected, so they never commit into a newer assignment. Records of a partition that is not currently assigned, such as records fetched before a revoke, are skipped by receive. A publication that completed before the revoke is not undone, and the new owner reprocesses the record from the last committed offset.

A source with a graceful handoff lets deliveries that already started finish before the revoke completes instead.

Source::pause pauses the partition an ordering key names, so a subscription's max_in_flight_per_key stops one busy partition without stopping the others, and Source::resume resumes it. A paused partition stays paused when it is revoked and assigned to the source again, until the runtime resumes it. A record the source handed to receive before the pause is still delivered.

The consumer is polled on a thread of its own, which hands one record at a time to receive and runs the consumer's rebalance callbacks. It polls only as fast as the subscription receives, so a subscription that stops receiving for longer than max.poll.interval.ms makes the consumer leave its group. Dropping a pending source receive does not consume a record. Dropping a received KafkaMessage does not acknowledge it.

Publication

prepare encodes the nullable value once; publication retries reuse that prepared record.

KafkaSink::submit returns once the producer has queued the record, with a completion that resolves on the record's delivery report. The runtime frees the job's concurrency slot at submission and acknowledges the input when the report arrives, so a subscription keeps several records in the producer while each input is still acknowledged only after Kafka has its output. publish queues the record and waits for the report.

max_pending bounds the records queued by submit on one sink whose report has not arrived; the default is 1000. The limit is per sink and shared by its clones, not by the other sinks of the handle. submit waits while that many are outstanding, and also while librdkafka's own queue (queue.buffering.max.messages, queue.buffering.max.kbytes) is full. A completion dropped before its report, as when the input's partition is revoked, frees its slot; the record stays queued and may still be written.

librdkafka retries a failed produce request internally until delivery.timeout.ms. A delivery report that still carries an error fails the completion, which stops the subscription without acknowledging the input. The input is then redelivered from the last committed offset. Publish retries apply to queueing only, such as a record rejected before it reaches the queue.

Records queued for one partition keep their order when the producer does not reorder retried batches. Set enable.idempotence=true so that a retry cannot move a record behind a later record with the same key. close stops new submissions on that sink and flushes the shared producer for at most close_timeout, 30 seconds by default. The flush also waits for records the handle's other sinks have queued, and the producer stays open for them. Because the producer is shared, a cluster that stays unreachable past delivery.timeout.ms fails the queued records of every sink of the handle, and each subscription with a failed record stops.

Metadata inheritance

Register KafkaInherit on a Kafka-to-Kafka subscription to forward the received key and headers:

use beavers::adapters::kafka::KafkaInherit;

Subscription::new("orders", kafka_source, kafka_sink, handler)
    .middleware(KafkaInherit::new())

Explicit output fields take precedence. The received key is used only when the output key is None; because an unset key and an intentionally absent key are both None, use KafkaInherit::new().without_key() to publish keyless records. Received headers are placed before the output's own headers, except headers whose name the output already sets and dead-letter headers starting with beavers-dlq-. without_headers() disables header inheritance. The source topic, partition, offset, and timestamp are never inherited: the sink chooses the topic, Kafka chooses the partition and timestamp, and the source offset is only used for the source's own commits.

A handler that returns a plain value between a Kafka source and sink gets KafkaInherit::new() automatically. See the runtime guide.

Inherited headers include trace-context headers such as traceparent. Register TraceContext after KafkaInherit to replace them with the processing span's context; see trace-context propagation.

Tombstones

A Kafka producer sends a null value as a tombstone, typically to delete a key in a compacted topic. Use the platform-neutral Tombstones middleware to reject, skip, or propagate them before the handler runs; see tombstones. Propagation publishes KafkaPublish::tombstone with the received key and rejects a tombstone without a key.

Dead letters

KafkaPublish::from_dead_letter forwards a dead letter to a Kafka topic with its original record and failure headers, and KafkaDeadLetter::from_record reads them back; see forwarding dead letters.

Health and tracing

The source reports its connection state to health checks from librdkafka statistics, enabling statistics.interval.ms at 5000 ms unless the handle or source sets it. Consumer lag comes from the broker; see the monitoring guide. Each delivery's message span carries the topic, partition, offset, and group ID as messaging attributes.

Delivery guarantees

With a KafkaSink, producer publication and source offset commits are separate operations, so a failure or rebalance between them can produce duplicates. Because publication is pipelined, a crash can leave records whose reports had not arrived; their inputs are uncommitted and are reprocessed. Use a transactional subscription to make them atomic. Without a graceful handoff, a rebalance abandons in-flight work at once.

Transactions

KafkaSink::transactional(transactional_id) converts a sink into a KafkaTransactionalSink, which publishes records in Kafka producer transactions. Registered with Subscription::transactional() behind a KafkaSource, each batch of deliveries is completed by one transaction that contains the outputs of every delivery in it and, for each partition, the consumer offset after the batch's last delivery of that partition:

begin transaction → produce outputs → send consumer offsets → commit transaction
use beavers::{Subscription, Utf8, adapters::kafka::Kafka};

let kafka = Kafka::new("localhost:9092");
let source = kafka.source::<Utf8>("orders-workers", ["orders"]);
let sink = kafka
    .sink::<Utf8>("processed-orders")
    .transactional("orders-workers-1");
let subscription = Subscription::new("orders", source, sink, |order: String| async move {
    Ok(order.to_uppercase())
})
.transactional();

KafkaTransactionalSink is a separate type, so only a transactional sink can be registered with Subscription::transactional().

Consumers that read the output with isolation.level=read_committed, the librdkafka default, see a batch's outputs only once its offsets are committed with them. A commit failure aborts the transaction, so neither the outputs nor the offsets take effect, and the runtime retries the batch with the same prepared outputs. Deliveries that complete without output, such as discarded or dead-lettered ones, contribute their offset without records. Dead letters are published outside the transaction.

The transactional ID becomes the producer's transactional.id. Give each running instance of the application its own ID and keep it across restarts of that instance: a new producer with the same ID fences the previous one and aborts its unfinished transaction. The adapter sets the ID itself, so the handle's validate rejects transactional.id set with property or producer_property. KafkaTransactionalSink::transaction_timeout bounds each blocking transaction call: initialization, sending offsets, commit, and abort; the default is 60 seconds. Its close_timeout defaults to the sink's. The transactional sink publishes through a producer of its own, created with the handle's properties and producer properties, and its transactions are initialized on first use.

The source and sink must use the same Kafka cluster, because the offsets are committed through the group coordinator of the sink's cluster. Before the first delivery is processed, the sink compares the cluster ID reported by the source's consumer with the one reported by its brokers, read with a separate non-transactional client, and a mismatch stops the subscription. A source and sink built from different handles, or with different broker lists for the same cluster, are accepted. Each cluster ID request waits up to transaction_timeout.

A producer runs one transaction at a time, so transactions from every clone of the sink are serialized, while handlers still run in parallel up to the subscription's concurrency. Batching amortizes that serialization: one transaction's commit latency covers up to TransactionBatch::max_deliveries deliveries from every partition. Each transaction runs in its own task: an abandoned batch does not leave a transaction half-finished or release the producer while it is in use. A commit that fails with a retriable error is retried within the same transaction. After an error that cannot be aborted, or a fatal one such as fencing, the producer is discarded and the next attempt initializes a new one with the same transactional ID.

Offsets are sent with the consumer's current group metadata, so the group coordinator rejects them from a consumer that has left the group generation. The offset check and commit run while the source holds its revoke callback back: a revoke waits for a commit in progress, and a transaction that contains a delivery of a revoked or lost assignment is aborted without committing. The runtime then retries the batch without the revoked deliveries. A batch holds the deliveries of one source; a batch whose deliveries come from different consumers is rejected. Offsets are committed only through transactions; the source's own commits are never used in a transactional subscription.

Used as a plain Sink, KafkaTransactionalSink commits each publication in a transaction of its own without consumer offsets. That makes each record visible to read_committed consumers atomically, but the subscription remains at-least-once.

Graceful handoff

KafkaSource::graceful_handoff(timeout) makes a revoke wait for the revoked partitions' deliveries before the partitions go to their next owner. It is off by default.

use beavers::{Json, adapters::kafka::Kafka};
use std::time::Duration;

let kafka = Kafka::new("localhost:9092");
let orders = kafka
    .source::<Json<String>>("billing", ["orders"])
    .graceful_handoff(Duration::from_secs(10));

When Kafka revokes partitions, the source cancels each delivery's handoff token and stops accepting records of those partitions. The runtime abandons deliveries that have not started, while started ones continue and are acknowledged as usual. The revoke waits until every delivery of the partitions was acknowledged or dropped, or until timeout passes, then commits the acknowledged offsets and cancels the revocation tokens, abandoning whatever is still running. The next owner starts after the acknowledged records instead of reprocessing them.

The rebalance callback runs on the consumer's polling thread, so the consumer receives no records of any partition while it waits. Kafka also waits for the revoke before it completes the rebalance, for at most max.poll.interval.ms, so validate requires timeout to be shorter than that property (5 minutes by default). A lost assignment is not handed off, because its offsets can no longer be committed. In a transactional subscription, the revoke waits for the transactions that contain the started deliveries, and the source commits no offsets of its own.

Markdown · Source on GitHub