beave-rs

beave-rs — Design plan

This document contains planned work and unresolved design decisions. Current behavior belongs in the architecture, runtime, adapter, and codec documentation. When planned work is implemented, remove it from this document and document the resulting contract in the appropriate guide.

Direction

beave-rs is a lightweight Rust message-processing framework for Kafka, Pulsar, NATS JetStream, SQS, Azure Service Bus, and other transports:

Source → Handler → Sink

A Subscription connects one source, one handler, and one sink.

Application code should primarily contain typed Input → Output handlers. The framework should own connectivity, serialization, acknowledgements, retries, concurrency, shutdown, and observability while describing delivery guarantees according to the actual source and sink capabilities.

Handlers can keep simple per-key state committed with each delivery. The project does not aim to provide stateful stream processing such as windows, joins, watermarks, changelog-backed local state stores, distributed scheduling, SQL, or general DAG execution.

Middleware

The asynchronous pre_handler and post_handler hooks are described in the runtime guide. Remaining decisions:

Serialization and codecs

Potential codec work includes:

Schema Registry support must keep wire framing separate from raw Avro datum encoding and define schema lookup caching, compatibility failures, and behavior when the registry is unavailable.

Delivery semantics and transactions

Kafka-to-Kafka transactions are described in the runtime guide and the Kafka guide, including the batching of deliveries into one transaction. Remaining work:

The ignored transactional_pipeline_commits_outputs_with_offsets test passes against a single-node Kafka 3.9 broker with the default TransactionBatch. Failure and rebalance paths of batched commits have not been verified against a live broker. Verify with Kafka:

Pulsar transactions were removed. magnetar-driver 1.7 reuses the consumer's session-wide batch bitset for transactional acknowledgements of batched messages, so the broker rejects all but the first message of a batch, does not repeat its transaction coordinator handshake after a broker restart, and fails every retry of a redelivered delivery whose original transaction committed. Reintroducing them needs a client that fixes these defects.

Multiple sinks within one subscription remain deferred because partial publish success makes retry and acknowledgement behavior ambiguous. Any future design must define atomicity or explicit partial-failure semantics.

Failure defaults

ErrorPolicy::default() stops the subscription without ACK on a decode or encode failure, and on a handler rejection or exhausted retry when no dead-letter sink is configured. For a durable broker source, one poison record then stops the application on every restart until it is handled. For a source fed by untrusted clients, such as the HTTP source, one bad request would stop the server; the HTTP source avoids this for decode failures by decoding before a request becomes a delivery, but handler rejections without a dead-letter sink still stop it. Remaining decisions:

Dead-letter handling

Forwarding dead letters with failure headers is described in the runtime guide, and examining, reprocessing, parking, and dropping them in the dead-letter handling guide. The library deliberately sends no notifications and has no redrive command. Remaining decisions:

Dead-letter headers and properties have not been verified against live brokers.

Per-key backpressure

max_in_flight_per_key pauses and resumes keys through Source::pause, as described in the runtime guide. Only the Kafka source can pause a key. Remaining decisions:

State

Stateful handlers and InMemoryState are described in the state guide. Planned stores, each in its own change:

Remaining decisions:

Chained subscriptions

channel chains subscriptions, as described in the channel adapter guide. Remaining decisions:

Upstream revocation and redelivery through a channel are covered by local tests only. Verify with Kafka that a partition revocation during a downstream stage abandons the downstream work and that the next owner reprocesses it.

Pipelined broker publication

Kafka and Pulsar sinks accept a record when the producer queues it and complete it on the delivery report or broker receipt, as described in the Kafka and Pulsar guides. Remaining decisions:

Only the offline acceptance bound of the Kafka sink is covered by default tests. The ignored live tests check that completions resolve on delivery and that max_pending holds. Verify with the brokers:

Shared connection follow-ups

Sources and sinks share their adapter handle's connection; the adapter documents describe what each handle shares. Remaining work:

Kafka rebalance behavior

The graceful handoff and partition pause are covered by unit, runtime, and ignored live tests. The live tests pause and resume one partition, and hand off a partition with an eager rebalance while its delivery is acknowledged, against single-node Kafka 3.9 and 4.1 brokers. Verify with Kafka:

Partition scheduling and revocation are covered by unit and runtime tests but have not been verified against a live broker. Verify with Kafka:

Verify with Pulsar that Failover and Key_Shared deliveries carry the partition index and ordering key expected by the adapter.

Pulsar client

The Pulsar adapter uses magnetar-driver. Its consumers and producers reattach after topic unloads and broker restarts, while the pulsar crate adapter it replaced stalled its consumer after a topic unload and ended the pipeline on a broker restart. Other client limitations:

Observability

Spans, events, metrics, messaging attributes, and trace-context propagation are described in the runtime guide. Remaining decisions:

Health checks, source connection state, and stall detection are described in the runtime guide. Consumer lag is left to broker metrics. Remaining decision:

The Pulsar probe's is_connected has not been verified across a broker restart, and neither probe across a multi-broker cluster in which one broker fails.

Trace-context propagation and the Prometheus export path have not been verified against live brokers or an OpenTelemetry Collector.

The monitoring and alerting guide has not been verified against live cloud accounts. In particular, check the CloudWatch dimension sets produced by the awsemf exporter, the job label that the Azure Monitor PodMonitor assigns, and the customMetrics values that the azuremonitor exporter writes after cumulativetodelta.

Testing utilities

The testing feature is described in the testing guide. Remaining decisions:

Future adapters

Candidate adapters are:

The HTTP source answers each request with a status only. Remaining decisions:

Its server limits are listed under failure defaults.

The HTTP sink sends one request per output and classifies responses with a fixed rule. Remaining decisions:

The RabbitMQ adapter consumes one queue and publishes to one exchange. Remaining decisions:

The ignored RabbitMQ live tests cover publication with confirmations, ACK, redelivery after close, revocation after a connection is closed by the broker, unroutable messages, and a forwarding pipeline against a single RabbitMQ 4.1 node. Verify with RabbitMQ:

The SQS adapter consumes and sends to one queue with one request per message. Remaining decisions:

The ignored SQS live tests cover sending with attributes, deletion on ACK, visibility extension, release on drop and close, FIFO group ordering, and a forwarding pipeline against ElasticMQ. Verify with Amazon SQS:

The Pub/Sub adapter consumes one subscription with the official client's streaming pull and publishes to one topic. Remaining decisions:

The ignored Pub/Sub live tests cover publication with attributes, ACK, lease extension, revocation at the maximum lease, redelivery after a drop and close, ordering keys, and a forwarding pipeline against the emulator. Verify with Google Cloud Pub/Sub:

The Azure Service Bus adapter receives from one queue or subscription in peek-lock mode and sends to one queue or topic over AMQP. Remaining decisions:

The ignored Service Bus live tests cover sending with application properties, completion, lock renewal, abandonment on drop, release of prefetched messages on close, the dead-letter subqueue, topic subscriptions, scheduled messages, and a forwarding pipeline against the Service Bus emulator. The emulator serves plain AMQP only and cannot create topic subscriptions through its management API. Verify with an Azure namespace:

Each broker adapter must define its native record and publish types, ACK model, redelivery behavior, ordering scope, cancellation behavior, connection lifecycle, and mapping of trace-context propagation fields (NATS headers) before implementation.

Release preparation

The crate is published as beavers; versioning and the compatibility contract are described in versioning and releases. Remaining work before the first crates.io release:

Implementation order

Priority Scope
1 Observability refinements
2 Live verification of batched Kafka transactions, shared connections, and Kafka rebalance handoff
3 NATS JetStream adapter
4 Schema Registry and additional codecs

The order may change when a concrete application requires a later capability. When work begins, update this document with any newly resolved decisions. When the work is complete, remove its planning details and place the stable behavior in the relevant durable documentation.

Open questions

  1. Further refinement of source, message, and sink generics, lifetimes, and error types.
  2. Handler ergonomics for implicit versus explicit Emit registration.
  3. Compile-time versus startup validation of broker capabilities.
  4. Shutdown deadlines and cancellation policy.
  5. Adapter and codec crate boundaries as optional dependencies grow.
  6. Isolation of async handler futures, which can block an executor thread, from communication and control execution.

Core philosophy

Application code should process events. beave-rs should manage the surrounding flow without promising capabilities that the underlying platform cannot provide.

Markdown · Source on GitHub