beave-rs

Errors and retries

beavers::Result<T> uses HandlerError, whose variants hold a BoxError. Ordinary errors propagated with ?, from any std::error::Error + Send + Sync type, String, &str, or anyhow::Error, become Reject: the handler is not retried, and the input is dead-lettered by default. The runtime cannot tell a transient failure from a deterministic one, so the handler requests other outcomes at the call site with the Classify extension trait. .reject()? states the default explicitly, .retry()? reruns the handler under its retry policy and dead-letters the input once that is exhausted; .fatal()? stops the subscription:

use beavers::{Classify, Result};

async fn handle(order: Order) -> Result<Output> {
    validate(&order).reject()?; // invalid input: dead-letter now (same as `?`)
    let user = db.find_user(order.user_id).await.retry()?; // transient: retry, then dead-letter
    Ok(build_output(order, user))
}

Error policy

ErrorPolicy decides what happens to a delivery after one of five routable failures, identified by FailureKind:

FailureKind Cause Default action
Decode SourceMessage::decode failed Stop
Rejected The handler or a middleware returned Reject, including errors propagated with ? DeadLetter
RetryExhausted The handler returned Retry on its final permitted attempt DeadLetter
Encode Sink::prepare failed for an emitted output Stop
PublishRejected The sink's destination refused an output permanently DeadLetter

Each failure maps to one FailureAction:

Rejections, exhausted handler retries, and rejected publications without a configured dead-letter sink stop without ACK: the input could not be processed and nothing can receive it. DeadLetter for decode or encode failures requires a dead-letter sink and fails validation otherwise. ErrorPolicy::dead_letter_all() routes every kind to the dead-letter sink. Set single actions with the method of the same name, such as ErrorPolicy::default().decode(FailureAction::DeadLetter).

Encoding happens for all emitted outputs before any publication, so an Encode failure never follows a partial publish; dead-lettering the input after it does not duplicate outputs.

A sink reports a permanent refusal, such as an HTTP 4xx response, by marking its publish, submit, or commit error with PublishRejected::wrap. The runtime does not retry a rejected output and submits no later outputs of the delivery. Outputs submitted before it stay published, and their completions are awaited before the delivery is routed, so a DeadLetter or Discard action acknowledges it only after they reached the sink's acknowledgement boundary. Errors without the marker are retried under publish_retry, and an exhausted retry stops the subscription. A failed completion is never routed. A transactional subscription cannot route a rejected commit either.

Dead letters

Subscription::dead_letter(sink) accepts a Sink<DeadLetter<Input, Raw>>, where Raw is the source message's undecoded form (SourceRaw<S>). Subscription::dead_letter_with(sink, convert) converts each envelope into the sink's own output type first, for example to forward the original payload to a broker topic with failure details as headers. Handlers never see dead letters.

Field Meaning
subscription Name of the failing subscription
failure The FailureKind
error Error message including its context chain
attempts Handler attempts made; zero for decode failures
input Decoded handler input; None after a decode failure
raw The delivery as received, with its broker metadata

SourceMessage::Raw defines raw per source, so broker metadata keeps its native shape instead of passing through a universal structure:

Source Raw
Kafka KafkaRecord<Vec<u8>>: value bytes, key, headers, and delivery metadata
Pulsar PulsarRecord<Vec<u8>>: payload bytes, key, properties, event time, and delivery metadata
RabbitMQ RabbitMqRecord<Vec<u8>>: body bytes, headers, properties, and delivery metadata
SQS SqsRecord<Vec<u8>>: body bytes, message attributes, and message metadata
Pub/Sub PubSubRecord<Vec<u8>>: data bytes, attributes, ordering key, and message metadata
Azure Service Bus ServiceBusRecord<Vec<u8>>: body bytes, application properties, message properties, and message metadata
Stdin Vec<u8>: the received line
Delivery (IterSource) (): input is already typed
Channel ChannelRaw: the raw form of the upstream delivery that produced the value

SourceMessage::Raw implements Serialize, so DeadLetter implements it whenever its input does and a JSON sink can publish it directly.

Conversion and preparation run once. Publication then retries under the dead_letter_retry policy. A conversion, preparation, or exhausted publication failure stops without ACK; dead-letter failures are never routed again.

Forwarding dead letters to a broker topic

KafkaPublish::from_dead_letter, PulsarPublish::from_dead_letter, RabbitMqPublish::from_dead_letter, SqsPublish::from_dead_letter, PubSubPublish::from_dead_letter, and ServiceBusPublish::from_dead_letter convert a dead letter into a record for a dead-letter topic or queue of the same platform. The record keeps the original key, value bytes, and headers or properties (and the Pulsar event time), so the payload can be inspected with ordinary broker tools and decoded by the same codec as the original topic. Failure details are added as text headers (Kafka), properties (Pulsar), string headers (RabbitMQ), string application properties (Azure Service Bus), or attributes (Pub/Sub, whose 1024-byte value limit cuts a longer error message). SQS accepts at most 10 message attributes, so the SQS adapter writes the same names and values as one JSON object into the beavers-dlq-details string attribute:

Name Value
beavers-dlq-subscription Subscription that dead-lettered the payload
beavers-dlq-failure FailureKind name, such as rejected
beavers-dlq-error Error message including its context chain
beavers-dlq-attempts Handler attempts made
beavers-dlq-count Times the payload has been dead-lettered, including this one
beavers-dlq-origin-topic Kafka or Pulsar topic the payload was first received from
beavers-dlq-origin-partition, -offset, -timestamp Kafka location and record timestamp of the first receipt
beavers-dlq-origin-message-id, -publish-time Pulsar message ID (ledger:entry:partition:batch) and publish time of the first receipt
beavers-dlq-origin-queue, -exchange, -routing-key RabbitMQ queue of the first receipt, and the exchange and routing key it was published with
beavers-dlq-origin-queue-url, -message-id SQS queue URL and message ID of the first receipt
beavers-dlq-origin-subscription, -message-id Pub/Sub subscription and message ID of the first receipt
beavers-dlq-origin-entity-path, -sequence-number Azure Service Bus entity path and sequence number of the first receipt

The sink must publish raw bytes:

let dead_letters = kafka.sink::<RawBytes>("orders-dlq");

Subscription::new("orders", source, sink, handle)
    .dead_letter_with(dead_letters, |dead_letter| Ok(KafkaPublish::from_dead_letter(dead_letter)));

When the dead-lettered record was itself received from a dead-letter topic, the origin headers are kept and the count is incremented, while the other details describe the latest failure. Previous dead-letter headers that are malformed are replaced as if the payload had never been dead-lettered. KafkaDeadLetter::from_record, PulsarDeadLetter::from_record, RabbitMqDeadLetter::from_record, SqsDeadLetter::from_record, PubSubDeadLetter::from_record, and ServiceBusDeadLetter::from_record read the details back from a received record and return None for a record without them. The inheritance middleware of each adapter never copies names starting with DEAD_LETTER_HEADER_PREFIX into outputs.

The dead-letter record's own timestamp or publish time is when it was dead-lettered. A subscription downstream of a channel receives a ChannelRaw; DeadLetter::try_map_raw reads it as the upstream record before conversion.

Reprocessing dead letters

beavers has no dedicated redrive command. A dead-letter topic is an ordinary topic, so dead letters are reprocessed by a subscription whose source reads the dead-letter topic and whose handler is the original one, or a wrapper around it that inspects KafkaDeadLetter::from_record first, for example to stop after a limit on details.count. The dead-letter handling guide describes how to find the cause of dead letters, how to reprocess, park, or drop them on log and queue brokers, and complete examples.

Other failures

Failure Behavior
Handler Retry Retry the handler with cloned input, then apply RetryExhausted routing
Handler Fatal Stop without ACK
Receive Retry Back off and retry receive; reset the failure count after receiving a message
Receive Fatal or exhausted retry Stop receiving, drain outstanding work, return an error
Value-only input unavailable (tombstone) Rejected routing without invoking the handler
Middleware Reject Rejected routing with the decoded input; publish no outputs
Middleware Retry or Fatal Stop without ACK; never rerun the handler or middleware
Publish failure Retry the prepared output; never rerun handler, middleware, or encoding
Publish rejected by the destination PublishRejected routing without retry
Exhausted publish Stop without ACK; infrastructure failures are never dead-lettered
ACK failure Return an error; do not claim successful completion

Publication, dead-letter publication, and ACK results are never assumed. An interrupted or failed attempt is not treated as success.

Sources and sinks built from one broker handle share its connection. A lost shared connection therefore affects every subscription that uses it at once: each one sees its own receive, publish, or ACK failure and handles it as above, so a subscription still stops on its own fatal or exhausted failure.

Markdown · Source on GitHub