beave-rs

Adapters

Adapters connect the framework's Source and Sink contracts to concrete inputs and outputs. Local adapters and the channel adapter are available by default. Kafka, Pulsar, RabbitMQ, SQS, Pub/Sub, Azure Service Bus, and HTTP adapters are optional features and are available through beavers::adapters; the default local adapters and channel types are also re-exported at the crate root.

Adapter Role Typical use
IterSource<T> Typed input Fixtures, examples, finite jobs
StdinSource<C> Line-delimited input Files, pipes, interactive input
InMemorySink<T> Typed output collection Assertions and local inspection
StdoutSink<C> Line-delimited output JSON output, files, pipes
ChannelSource<T> / ChannelSink<T> Chained subscriptions and application ends Pipeline stages whose upstream ACK waits for the downstream stage; in-process workers fed and read by application code
KafkaSource<C> / KafkaSink<C> Kafka records and publishes Durable broker pipelines (kafka feature)
KafkaTransactionalSink<C> Kafka publishes in producer transactions Exactly-once Kafka-to-Kafka pipelines (kafka feature)
PulsarSource<C> / PulsarSink<C> Pulsar records and publishes Durable broker pipelines (pulsar feature)
RabbitMqSource<C> / RabbitMqSink<C> RabbitMQ queue deliveries and exchange publishes Work queues and routed messaging (rabbitmq feature)
SqsSource<C> / SqsSink<C> Amazon SQS queue messages and sends Managed work queues on AWS (sqs feature)
PubSubSource<C> / PubSubSink<C> Google Cloud Pub/Sub subscription messages and topic publishes Managed messaging on Google Cloud (pubsub feature)
ServiceBusSource<C> / ServiceBusSink<C> Azure Service Bus queue and subscription messages and sends Managed work queues and topics on Azure (servicebus feature)
HttpSource<C> HTTP POST requests answered on ACK Webhooks and push ingestion (http feature)
HttpSink<C> HTTP requests that succeed on a 2xx response Delivering events to HTTP services (http feature)

NATS JetStream is planned. Local and channel adapters do not provide durable delivery guarantees.

Enable a broker adapter in the application manifest:

[dependencies]
beavers = { version = "0.1", features = ["kafka"] }
# or: features = ["pulsar"], ["rabbitmq"], ["sqs"], ["pubsub"], ["servicebus"], or ["http"]

The broker-specific guides document configuration, metadata, ACK behavior, and delivery boundaries: Local adapters, Channel, Kafka, Pulsar, RabbitMQ, SQS, Pub/Sub, Azure Service Bus, and HTTP.

Implementing another adapter

Implement Source with an associated Message: SourceMessage for input, or Sink<T> with an associated prepared representation for output. Adapters own transport-specific settings and resource cleanup. Codecs own serialization; subscription runtime owns retries, concurrency, and ACK sequencing.

Keep these contracts explicit:

Delivery::with_ordering_key, Delivery::with_revocation, and Delivery::with_handoff provide these for custom sources built on Delivery<T>, and Delivery::with_raw sets the raw form their dead letters carry.

See architecture for ownership boundaries and the runtime guide for retry, failure, and shutdown behavior.

Broker adapters

Kafka, Pulsar, RabbitMQ, SQS, Pub/Sub, and Azure Service Bus keep their platform-specific message models explicit below beavers::adapters::kafka, beavers::adapters::pulsar, beavers::adapters::rabbitmq, beavers::adapters::sqs, beavers::adapters::pubsub, and beavers::adapters::servicebus. Each decodes source messages into a broker-specific *Record<T> containing delivery metadata and accept a separate *Publish<T> type for user-controlled output. Prepared outputs hold encoded bytes for retries; their fields are private, and payload() returns the encoded bytes. See the dedicated guides for each broker's configuration and delivery behavior.

Each broker adapter starts from a connection handle: Kafka, Pulsar, RabbitMq, Sqs, PubSub, or ServiceBus. The handle holds the broker address and credentials, and its source and sink methods build sources and sinks that share its connection, so they authenticate once; the broker guides list the clients an adapter keeps per source or sink, such as Kafka consumers. The connection opens on first use. A lost shared connection affects every subscription that uses it, while each subscription still fails fast on its own receive, publish, and ACK failures. HttpClient builds HTTP sinks that share one connection pool, and HttpSource::new(addr) binds a server.

use beavers::{Json, adapters::kafka::Kafka};

let kafka = Kafka::new("localhost:9092").sasl_plain("user", "password");
let orders = kafka.source::<Json<Order>>("billing", ["orders"]);
let invoices = kafka.sink::<Json<Invoice>>("invoices").max_pending(500);

Builder methods consume and return the value, and every handle, source, and sink has a validate() that checks its settings without network access. A builder method on a handle returns a handle with a connection of its own, so configure the handle before building sources and sinks from it. source and sink use C::default() for a codec. source_with_codec(codec, ...) and sink_with_codec(codec, ...) take an existing codec instance first, which is required for configured codecs such as Avro; HttpSource has with_codec(codec, addr). StdinSource and StdoutSink take a codec with with_codec(codec).

Markdown · Source on GitHub