beave-rs

Pulsar adapter

The optional pulsar feature provides typed Apache Pulsar source and sink adapters backed by the magnetar-driver 1.7 client.

beavers = { version = "0.1", features = ["pulsar"] }

Configuration

A Pulsar handle holds the service URL, using the pulsar:// or pulsar+ssl:// scheme, and optional static authentication. handle.source::<C>(topic, subscription) builds a source and handle.sink::<C>(topic) a sink; source_with_codec and sink_with_codec take a codec value as their first argument instead of using the codec's default. Optional settings are builder methods on the handle, the source, and the sink.

use beavers::Utf8;
use beavers::adapters::pulsar::{Pulsar, PulsarSubscriptionType};

let pulsar = Pulsar::new("pulsar://localhost:6650");
let source = pulsar
    .source::<Utf8>("persistent://public/default/orders", "orders-workers")
    .subscription_type(PulsarSubscriptionType::Failover);

All sources and sinks built from one handle share one client. The client connects on the first receive or publication of any of them, which also validates the settings, and reconnects their producers and consumers after the broker connection drops. It closes once the handle, every source and sink built from it, and every delivery that can still acknowledge through it have been dropped. Closing a source or sink closes only its own consumers or producers. Each builder method on the handle returns a handle with a client of its own, so configure the handle before building sources and sinks from it.

The handle, the source, and the sink each have validate, which checks the values without contacting a broker. A partitioned topic is consumed through one consumer per partition, all in the same subscription; the partition list is read when the source connects.

subscription_type selects a PulsarSubscriptionType and defaults to Shared. buffer_size is the number of messages each partition's consumer prefetches, 100 by default. A new subscription starts at the latest message.

Pulsar::token supplies a JWT, while Pulsar::authentication accepts a general PulsarAuthentication with a Pulsar authentication method name and credential bytes. Leave both unset for an unauthenticated broker.

Build a sink from the same handle. prepare accepts a PulsarPublish<T>, encodes the typed value once, and returns a cloneable PulsarPrepared payload. Set the key, ordering key, event time, and user properties before preparation:

use beavers::{Sink, Utf8};
use beavers::adapters::pulsar::{Pulsar, PulsarPublish};

let pulsar = Pulsar::new("pulsar://localhost:6650").token("eyJhbGciOi...");
let sink = pulsar.sink::<Utf8>("persistent://public/default/processed-orders");
let mut output = PulsarPublish::new("processed".to_owned());
output.key = Some(b"customer-42".to_vec());
output.properties.insert("kind".into(), "order".into());
let prepared = sink.prepare(output)?;
sink.publish(&prepared).await?;
# Ok::<(), beavers::BoxError>(())

producer_name names each partition's producer; by default the broker assigns names.

Records

The codec applies to the payload. PulsarMessage::decode returns a PulsarRecord<T> with top-level value, key, properties, and event_time fields. These application fields map directly to PulsarPublish<T>. Its metadata contains read-only delivery facts: topic, message ID, and publish time. For a partitioned topic, the metadata topic is the partition's topic, such as persistent://public/default/orders-partition-2. The source reads the key, properties, event time, and null marker of a message inside a producer batch from that message's own metadata.

The raw form of a PulsarMessage is a PulsarRecord<Vec<u8>> with the undecoded payload bytes, or value: None for a tombstone. Dead letters carry it, so the original payload, key, properties, event time, and delivery metadata survive even when decoding fails.

Acknowledgements and ordering

receive waits for the next message of any partition, starting with the partition after the one that delivered last so that a busy partition cannot starve the others. A message leaves its consumer's queue only when a receive completes, so dropping a pending receive future does not consume a delivery. Dropping a PulsarMessage leaves it unacknowledged. Call ack only after the handler has completed successfully. An individual ACK is sent with the message's ID and waits for the broker's response; the adapter does not use cumulative acknowledgements. Broker receive errors are reported as ReceiveError::Retry, while connection, configuration, and malformed message metadata errors are ReceiveError::Fatal. The client reconnects after the broker connection drops and reattaches its consumers.

A topic unload, as the broker's load balancer performs routinely, or a broker restart closes the consumers. An acknowledgement in flight at that moment fails, and one sent before the consumer has reattached times out after the client's 30-second operation timeout. ack therefore retries a failed acknowledgement under the source's ack_retry, five attempts with backoff from 100 ms to 2 s by default, and an attempt on the reattached consumer succeeds. Only an exhausted retry fails the acknowledgement and stops the subscription. After reattaching, the broker redelivers every unacknowledged message, including messages the consumer had already prefetched and still returns, so deliveries received around an unload can be processed twice and out of their original order.

Call close during shutdown to close the consumers. A delivery acknowledged after its source closed fails with an error. The client stays open for the handle's other sources and sinks.

Each delivery's ordering key follows the order Pulsar guarantees for the configured subscription type:

Subscription type Ordering key
Exclusive, Failover Topic partition
Key_Shared Ordering key, or the message key when no ordering key is set, within the topic partition
Shared None

Under the default SourceOrder distribution, deliveries with the same ordering key are processed one at a time in receive order. Key_Shared messages without a key and all Shared deliveries run in parallel up to the subscription's concurrency. PulsarDistribution names choices that apply the same way under every subscription type:

PulsarDistribution Processed one at a time
Partition The deliveries of one topic partition, as SourceOrder under Exclusive and Failover
MessageKey The deliveries of one key within a topic partition: the ordering key, or the message key when no ordering key is set; deliveries with neither one at a time per partition
use beavers::adapters::pulsar::PulsarDistribution;

Subscription::new("orders", source, sink, handler)
    .concurrency(16)
    .distribution(PulsarDistribution::MessageKey);

Under Exclusive or Failover, MessageKey runs different keys of a busy partition in parallel while keeping each key in receive order. Under Shared, it orders only the deliveries this consumer receives, since the broker may deliver one key's messages to several consumers. The Pulsar client does not report consumer failover or key-range reassignment, so Pulsar deliveries have no revocation token. When the broker moves a subscription to another consumer, unacknowledged messages are redelivered there and can be processed twice.

Publication

The sink connects lazily on the first publish, creating one producer per partition of a partitioned topic. A keyed message is routed like the Java client's default: the Java String.hashCode of its base64 key, modulo the partition count, so the same key always reaches the same partition. Keyless messages rotate across partitions. Keys are always sent in Pulsar's base64 representation because they are arbitrary bytes. Each submission clones the prepared bytes and metadata into a fresh producer message. Retrying a prepared value does not rerun the codec.

PulsarSink::submit returns once the partition's producer has queued the message, with a completion that resolves on Pulsar's broker receipt. The runtime frees the job's concurrency slot at submission and acknowledges the input when the receipt arrives. publish queues the message and waits for the receipt. A producer sends queued messages in order and replays unacknowledged ones in order after a reconnect, so outputs of one partition keep their submission order unless the broker rejects a send, as described below.

max_pending bounds the messages queued by submit on one sink whose receipt has not arrived; the default is 1000. submit waits while that many are outstanding. A completion dropped before its receipt, as when the input is abandoned, frees its slot; the message stays queued and may still be written, but it is not sent again after a rejection. Publish retries apply to queueing only.

The client replays sends that lose their connection. A send the broker answers with an error is not replayed by the client, and a broker that is shutting down or unloading the topic, as during a restart or a routine load-balancer unload, rejects the sends in flight with a persistence error. The sink therefore sends a rejected message again under the sink's send_retry, ten sends with backoff from 100 ms to 5 s by default, measured from the previous send. A task per producer observes the receipts in submission order, so the messages of one rejected run are sent again in their original order and reach the topic's next owner. Like the Java client, the sink does not send a message again after a NotAllowedError or TopicTerminatedError rejection. A send error that is not sent again, or a message rejected on every attempt, fails the completion, which stops the subscription without acknowledging the input.

A message sent again is written after later messages of its partition that the client replayed or the broker accepted in the meantime, so a rejection can reorder the outputs of one partition. A message the broker wrote despite answering with an error is written twice.

close stops new submissions, waits for every outstanding receipt, then closes the sink's producers. The client stays open for the handle's other sources and sinks.

Metadata inheritance

Register PulsarInherit on a Pulsar-to-Pulsar subscription to forward the received key, properties, and event time:

use beavers::adapters::pulsar::PulsarInherit;

Subscription::new("orders", pulsar_source, pulsar_sink, handler)
    .middleware(PulsarInherit::new())

Explicit output fields take precedence. The key and event time are inherited only when the output leaves them None, and a received property is added only when the output does not already set that name. Dead-letter properties starting with beavers-dlq- are never inherited. without_key(), without_properties(), and without_event_time() disable the corresponding field. The source topic, message ID, and publish time are never inherited, and no ordering key is derived from the input.

A handler that returns a plain value between a Pulsar source and sink gets PulsarInherit::new() automatically. See the runtime guide.

Inherited properties include trace-context properties such as traceparent. Register TraceContext after PulsarInherit to replace them with the processing span's context; see trace-context propagation.

Tombstones

A null value is a tombstone. Pulsar topic compaction deletes a key when it sees a message with that key and an empty payload, and producers can also mark a value as null explicitly. The source's empty_payload_is_tombstone, enabled by default, follows the compaction rule: an empty payload or a null-marked message becomes PulsarRecord { value: None, .. } and is not passed to the codec. When disabled, only a null-marked message becomes None, and an empty payload is decoded by the codec. Disable it only for topics that carry meaningful empty values and are not compacted.

The sink publishes PulsarPublish { value: None, .. } as an empty payload with the null marker set, so readers see a null value with or without empty_payload_is_tombstone. PulsarPublish::tombstone(key) builds one, and Tombstones::propagate() supports Pulsar-to-Pulsar forwarding. The sink publishes non-null values as encoded, even when the encoding is empty; readers that treat empty payloads as tombstones, including topic compaction, will read such a value as a deletion. See tombstones.

Dead letters

PulsarPublish::from_dead_letter forwards a dead letter to a Pulsar topic with its original message and failure properties, and PulsarDeadLetter::from_record reads them back; see forwarding dead letters.

Health and tracing

The source reports whether every partition's consumer is connected to health checks; a consumer reconnecting after a topic unload or broker restart makes the subscription unready until it reconnects. The subscription backlog comes from the broker's pulsar_subscription_back_log metric. Each delivery's message span carries the configured topic, partition, subscription, and message ID as messaging attributes.

Delivery guarantees

A source ACK and a sink publication are separate broker operations, so a process failure between them can produce a duplicate on redelivery. The adapter provides no Pulsar transactions: magnetar-driver 1.7 mishandles transactional acknowledgements of batched messages and transactions after a broker restart, so Pulsar pipelines are at-least-once.

Markdown · Source on GitHub