Adapters
Adapters connect the framework's Source and Sink contracts
to concrete inputs and outputs. Local adapters and the channel adapter are
available by default. Kafka, Pulsar, RabbitMQ, SQS, Pub/Sub, Azure Service Bus, and HTTP adapters are optional features and are
available through beavers::adapters; the default local adapters and channel
types are also re-exported at the crate root.
| Adapter | Role | Typical use |
|---|---|---|
IterSource<T> |
Typed input | Fixtures, examples, finite jobs |
StdinSource<C> |
Line-delimited input | Files, pipes, interactive input |
InMemorySink<T> |
Typed output collection | Assertions and local inspection |
StdoutSink<C> |
Line-delimited output | JSON output, files, pipes |
ChannelSource<T> / ChannelSink<T> |
Chained subscriptions and application ends | Pipeline stages whose upstream ACK waits for the downstream stage; in-process workers fed and read by application code |
KafkaSource<C> / KafkaSink<C> |
Kafka records and publishes | Durable broker pipelines (kafka feature) |
KafkaTransactionalSink<C> |
Kafka publishes in producer transactions | Exactly-once Kafka-to-Kafka pipelines (kafka feature) |
PulsarSource<C> / PulsarSink<C> |
Pulsar records and publishes | Durable broker pipelines (pulsar feature) |
RabbitMqSource<C> / RabbitMqSink<C> |
RabbitMQ queue deliveries and exchange publishes | Work queues and routed messaging (rabbitmq feature) |
SqsSource<C> / SqsSink<C> |
Amazon SQS queue messages and sends | Managed work queues on AWS (sqs feature) |
PubSubSource<C> / PubSubSink<C> |
Google Cloud Pub/Sub subscription messages and topic publishes | Managed messaging on Google Cloud (pubsub feature) |
ServiceBusSource<C> / ServiceBusSink<C> |
Azure Service Bus queue and subscription messages and sends | Managed work queues and topics on Azure (servicebus feature) |
HttpSource<C> |
HTTP POST requests answered on ACK |
Webhooks and push ingestion (http feature) |
HttpSink<C> |
HTTP requests that succeed on a 2xx response |
Delivering events to HTTP services (http feature) |
NATS JetStream is planned. Local and channel adapters do not provide durable delivery guarantees.
Enable a broker adapter in the application manifest:
[dependencies]
beavers = { version = "0.1", features = ["kafka"] }
# or: features = ["pulsar"], ["rabbitmq"], ["sqs"], ["pubsub"], ["servicebus"], or ["http"]
The broker-specific guides document configuration, metadata, ACK behavior, and delivery boundaries: Local adapters, Channel, Kafka, Pulsar, RabbitMQ, SQS, Pub/Sub, Azure Service Bus, and HTTP.
Implementing another adapter
Implement Source with an associated Message: SourceMessage for input, or
Sink<T> with an associated prepared representation for output. Adapters own
transport-specific settings and resource cleanup. Codecs own serialization;
subscription runtime owns retries, concurrency, and ACK sequencing.
Keep these contracts explicit:
- A canceled receive must not silently lose a delivery.
- Decoding and preparing output must not publish or ACK.
- Dropping a received message must not ACK it.
- Successful publication must reach the sink's documented confirmation boundary.
- A sink whose destination refuses an output permanently, where sending it
again cannot succeed, marks the error with
PublishRejected::wrapso the error policy routes the delivery instead of retrying it. - Broker-specific commit order and assignment validity belong to the adapter.
SourceMessage::RawimplementsDebugandSerialize, so dead letters can be published as they are and channels can keep it for the next stage.- A source with ordered delivery scopes, such as partitions, returns an
OrderingKeyfromSourceMessage::ordering_key. - A source whose deliveries can move to another consumer returns a revocation
CancellationTokenfromSourceMessage::revocationand cancels it when the delivery is no longer owned. A source that lets started work finish first also returns a handoff token fromSourceMessage::handoffand cancels it when the handoff begins. - A source that can stop receiving one ordering key implements
Source::pauseandSource::resume, which the runtime calls for a subscription'smax_in_flight_per_key, and keeps a paused key paused when the key's scope is revoked and assigned again. - A source that accepts input from clients, such as a server, stops accepting
it in
Source::stop_receiving, which the runtime calls when it stops receiving on shutdown or failure, before draining. - A broker source describes each delivery with
SourceMessage::messaging_attributesfor its trace span, and can report its connection state throughSource::probefor health checks. A probe answers from state the source already holds, without network access.
Delivery::with_ordering_key, Delivery::with_revocation, and
Delivery::with_handoff provide these for custom sources built on
Delivery<T>, and Delivery::with_raw sets the raw
form their dead letters carry.
See architecture for ownership boundaries and the runtime guide for retry, failure, and shutdown behavior.
Broker adapters
Kafka, Pulsar, RabbitMQ, SQS, Pub/Sub, and Azure Service Bus keep their
platform-specific message models explicit below beavers::adapters::kafka,
beavers::adapters::pulsar, beavers::adapters::rabbitmq,
beavers::adapters::sqs, beavers::adapters::pubsub, and
beavers::adapters::servicebus. Each decodes source
messages into a broker-specific *Record<T> containing delivery metadata and
accept a separate *Publish<T> type for user-controlled output. Prepared
outputs hold encoded bytes for retries; their fields are private, and
payload() returns the encoded bytes. See the dedicated guides for each
broker's configuration and delivery behavior.
Each broker adapter starts from a connection handle: Kafka, Pulsar,
RabbitMq, Sqs, PubSub, or ServiceBus. The handle holds the broker
address and credentials, and its source and sink methods build sources and
sinks that share its connection, so they authenticate once; the broker guides
list the clients an adapter keeps per source or sink, such as Kafka consumers.
The connection opens on first use. A lost shared connection affects every
subscription that uses it, while each subscription still fails fast on its own
receive, publish, and ACK failures. HttpClient builds HTTP sinks that share
one connection pool, and HttpSource::new(addr) binds a server.
use beavers::{Json, adapters::kafka::Kafka};
let kafka = Kafka::new("localhost:9092").sasl_plain("user", "password");
let orders = kafka.source::<Json<Order>>("billing", ["orders"]);
let invoices = kafka.sink::<Json<Invoice>>("invoices").max_pending(500);
Builder methods consume and return the value, and every handle, source, and
sink has a validate() that checks its settings without network access. A
builder method on a handle returns a handle with a connection of its own, so
configure the handle before building sources and sinks from it.
source and sink use C::default() for a codec. source_with_codec(codec, ...) and sink_with_codec(codec, ...) take an existing codec instance first,
which is required for configured codecs such as Avro; HttpSource has
with_codec(codec, addr). StdinSource and StdoutSink take a codec with
with_codec(codec).