Runtime guide
This document describes how the current implementation registers subscriptions and processes each delivery. Future broker behavior and proposed APIs are documented in the design plan.
Topic guides cover the rest of the runtime:
| Guide | Covers |
|---|---|
| Handlers | Handler signatures, emitting outputs, and blocking handlers |
| Middleware | Hooks around the handler and metadata inheritance |
| Errors and retries | Retry policies, the error policy, and dead letters |
| Tombstones | Handling null-value records |
| State | Per-key state kept across deliveries and committed before acknowledgement |
| Transactions | Batched transactional publish and acknowledgement |
| Observability | Spans, events, metrics, and trace-context propagation |
| Health checks | Liveness and readiness reports and endpoints |
Registration and configuration
Use App::subscribe(name, source, sink, handler) for defaults, or register a configured
Subscription. The following sketch assumes application-specific components and
retry policies have already been constructed:
App::new()
.subscription(
Subscription::new("orders", source, sink, handler)
.concurrency(16)
.max_in_flight(64)
.max_in_flight_per_key(16)
.distribution(SourceOrder)
.handler_retry(handler_retry)
.receive_retry(receive_retry)
.publish_retry(publish_retry)
.dead_letter_retry(dead_letter_retry)
.drain_timeout(Duration::from_secs(30))
.middleware(MapMetadata::new(|_input, output| Ok(output)))
.dead_letter(dead_letter_sink)
.error_policy(ErrorPolicy::dead_letter_all()),
)
.run()
.await?;
Every subscription has a required name, passed as the first argument of
Subscription::new, Subscription::new_emitting, and App::subscribe. The name appears in subscription errors, dead letters, spans,
and metric labels, so it must be non-empty and unique within an App; an empty
or duplicate name fails validation before any subscription starts.
Registration constructs the application. Processing starts in run().
| Setting | Default | Meaning |
|---|---|---|
concurrency |
1 | Maximum concurrent message jobs, and maximum deliveries decoding at once |
max_in_flight |
64 | Maximum received but unfinished deliveries, including decoding and queued ones |
max_in_flight_per_key |
none | Unfinished deliveries of one ordering key at which the source pauses that key |
distribution |
SourceOrder |
Which deliveries are processed one at a time; see distribution |
| Retry attempts | 3 | Includes the first attempt |
| Initial retry delay | 100 ms | Exponential backoff starting delay |
| Maximum retry delay | 5 s | Backoff cap |
| Retry jitter | Jitter::None |
Randomization of each backoff delay |
| Error policy | ErrorPolicy::default() |
Dead-letter handler failures and rejected publications; stop on decode and encode failures |
| Drain timeout | 30 s | Bound on draining; cleanup has a separate timeout of the same duration |
| Stall timeout | Disabled | Liveness fails while deliveries are in flight without progress for this long; see stall detection |
Receive, handler, output publish, and dead-letter publish retries have independent policies. Invalid zero concurrency, in-flight limits, or retry attempt counts, and a zero stall timeout fail validation before subscriptions start, as does an error policy that dead-letters decode or encode failures without a dead-letter sink.
Each RetryPolicy doubles its delay after every failed attempt, starting at
initial_delay and capped at max_delay. Set them with the methods of the same
name, such as RetryPolicy::default().max_attempts(5). jitter randomizes the capped
delay: Jitter::Full waits a uniformly random duration up to it, and
Jitter::Equal waits half of it plus a random share of the other half. Jitter
spreads retries from many consumers that failed at the same moment.
Decoding and distribution
Each subscription receives deliveries from its source one at a time. The runtime then:
- decodes each delivery on the blocking thread pool, up to
concurrencydeliveries at once, so a slow codec holds up neither receiving nor handlers; - passes the decoded deliveries to the subscription's
Distributionin the order they were received, even when a later delivery finished decoding first; - queues each delivery behind earlier deliveries with the same distribution
key, and runs deliveries with different keys, or without a key, in parallel
up to
concurrency.
A delivery whose key is busy waits in a queue and starts when the earlier delivery finishes. Outputs of deliveries with different keys have no ordering relationship with each other. A delivery that fails to decode has no key: the runtime routes it by the error policy as soon as a job slot is free, even while earlier deliveries from the same source scope still run.
A source can give each delivery an OrderingKey, the scope in which the source
delivers in order. The Kafka source uses the topic partition. The Pulsar source
uses the topic partition or message key, depending on its subscription type.
The RabbitMQ source uses its queue when configured as ordered. The SQS source
uses the message group of a FIFO queue. The Pub/Sub source uses the ordering
key of a message published with one. The Azure Service Bus source and local
adapters provide no key.
The distribution decides the key:
| Distribution | Key |
|---|---|
SourceOrder (default) |
The delivery's OrderingKey, so each source scope is processed in the order the source delivers it |
Parallel |
None: every delivery runs independently |
An adapter's distribution: KafkaDistribution, PulsarDistribution, ServiceBusDistribution |
A choice the adapter offers, such as the record key within its partition or the Service Bus session ID |
A closure Fn(&Item) -> K |
A value of the decoded input, such as a customer ID |
Your own Distribution implementation |
Anything the delivery or its decoded input carries |
Subscription::new("orders", source, sink, handler)
.concurrency(16)
.distribution(|order: &KafkaRecord<Order>| order.value.as_ref().map(|o| o.customer));
A subscription whose stateful handler keeps keyed state,
such as InMemoryState, distributes by its state key instead and takes no
distribution of its own.
Under Parallel, or any distribution finer than the source scope, deliveries
from one partition can run and complete out of order. Sources remain
responsible for safe acknowledgement; Kafka still commits only the contiguous
completed prefix of each partition. A transactional
subscription requires a distribution that keeps the source order, such as
SourceOrder or KafkaDistribution::Partition.
A key taken from the payload orders deliveries within one process only. The broker does not read the payload, so with several consumers the deliveries of one payload key can reach different processes. Distribute by a key the broker also routes by, such as the Kafka record key, when one process must see all of a key's deliveries, as in-memory state does.
Backpressure
The runtime receives only while a job slot is free, fewer than concurrency
deliveries are decoding, and fewer than max_in_flight deliveries are
unfinished. Decoding and queued deliveries count toward max_in_flight, so
consumption stays bounded when a handler or sink is slow. Deliveries without a
key never queue: each one starts as soon as it is distributed.
Without a per-key limit, a busy partition can fill max_in_flight and stop
receiving for all of them. max_in_flight_per_key bounds the unfinished
deliveries of each source OrderingKey, whatever the distribution. When a key reaches the limit,
the runtime calls Source::pause for it, and calls Source::resume once the
key's unfinished deliveries drop to half the limit, so a key near the limit
does not pause and resume on every delivery. Other keys keep flowing while one
is paused. Deliveries the source fetched before the pause still arrive, so a
key can exceed the limit by what the source buffers. The Kafka source pauses
the partition. Sources that cannot pause single keys, which is every other
adapter, return false from pause; the runtime then stops counting and only
max_in_flight applies.
Revocation
A source can also give each delivery a revocation CancellationToken. The
source cancels it when it stops owning the delivery, for example when Kafka
revokes the partition. The runtime then drops the delivery's job at its next
await point, whether it is decoding, in the handler, retrying, publishing, or
acknowledging. The delivery is not acknowledged and its outcome is not a
subscription failure, including an ACK that the source rejected because of the
revocation. Queued deliveries of the same assignment are dropped the same way.
Handlers do not observe revocation. Cancellation is cooperative, so work that never yields continues until it does. An output publication that was already accepted by the sink is not retracted; the new owner reprocesses the delivery and can produce a duplicate.
A source can let started work finish before it gives a delivery up by also
returning a handoff CancellationToken from SourceMessage::handoff. When the
source cancels it, the runtime drops the delivery without ACK if it has not
started and lets it continue otherwise; the source cancels the revocation token
when the handoff ends. The Kafka source does this during a
graceful handoff.
Adapters
See the local adapters for IterSource, StdinSource,
InMemorySink, and StdoutSink, and the Channel adapter
that chains subscriptions, including examples, EOF behavior, ACK guarantees,
and I/O limits. These components do not provide durable
redelivery after process exit. Optional broker adapters provide
broker-specific sources and sinks built from a connection handle, such as
Kafka or RabbitMq; their documentation covers configuration and
acknowledgement semantics.
Per-message lifecycle
receive → decode → pre_handler → handler → post_handler → prepare all outputs → submit → complete → ACK
Use Subscription::new_emitting to register a handler returning
Result<Emit<T>>. Emit::None emits nothing, One emits one value, and Many
emits several. A regular Vec<T> remains a single payload. With Many, outputs
are published sequentially and the input is acknowledged only after all succeed.
With no outputs, successful processing can proceed directly to ACK.
Decode and preparation failures follow the error policy.
Sink::submit hands an output to the sink and returns a Completion once the
sink has accepted it. Sinks that keep the default complete at submission. A sink whose
acceptance precedes its acknowledgement boundary, such as a
channel drained by another subscription, a
Kafka sink waiting for a delivery report, or a
Pulsar sink waiting for a broker receipt, returns a
pending completion: the job ends and frees its concurrency slot, and the
delivery is acknowledged after every completion succeeds. Deliveries waiting
for completion do not count toward max_in_flight; the sink bounds them by
making submit wait, as a channel does for capacity. They are abandoned
without ACK when revoked and are drained on shutdown. A failed completion stops the
subscription without acknowledging the delivery. Publish retries apply to
submission only.
End of input and shutdown
Receive::End explicitly means no more messages will arrive. A temporarily empty
source waits instead of reporting End. After End, a subscription drains work and
closes resources; failures during draining or cleanup still produce an error.
One subscription ending normally does not stop the others. App::run() returns
success after all subscriptions finish. A failure requests shutdown across the
application, which ultimately returns an error.
SIGINT / SIGTERM or App::run_until(CancellationToken) stops new receives and
starts draining running jobs. Deliveries still decoding or queued behind a key are
dropped without ACK. After Receive::End, queued deliveries still run. A source whose
stops_on_shutdown returns false, such as a ChannelSource created by channel,
keeps receiving until its upstream subscriptions close it; see
channel shutdown. When a subscription stops
receiving before its source ended, the runtime calls Source::stop_receiving
before draining, so a source such as the HTTP source
refuses new input instead of queuing it. A drain
timeout cancels unfinished tasks, then cleanup runs with its own deadline. In-progress publish or ACK can have an uncertain result if
interrupted; a durable broker may redeliver and cause duplicates.
Async handlers and middleware hooks share the Tokio executor with runtime work. They must not block worker threads; use blocking handlers for synchronous work. Cancellation and timeouts are cooperative and cannot forcibly interrupt arbitrary synchronous code.
Implementation limits
Kafka, Pulsar, RabbitMQ, SQS, Pub/Sub, and Azure Service Bus adapters, broker
record types, Protobuf, and Avro
codecs are implemented. Kafka commits up to the first unfinished offset per partition,
schedules work per partition or per record key, and abandons revoked work. Kafka-to-Kafka
subscriptions can publish and commit offsets in batched Kafka transactions;
other pairs are at-least-once. Pulsar uses individual
acknowledgements, schedules work by its subscription type's ordering scope, and
provides no transactions or exactly-once processing. RabbitMQ uses individual
acknowledgements and revokes the deliveries of a lost channel. SQS deletes each
acknowledged message, extends the visibility of held messages, and revokes a
delivery whose visibility could not be kept. Pub/Sub acknowledges through the
client's streaming pull, extends the leases of held messages, and revokes a
delivery held past its maximum lease. Azure Service Bus completes each
acknowledged message, renews the locks of held messages, and revokes a delivery
whose lock was lost. NATS JetStream and adapter pause/resume backpressure are not implemented. Async handler
futures run on the shared Tokio executor without isolation. Metadata inheritance
is limited to same-platform middleware;
cross-platform mappings are application-written MapMetadata functions.
Inputs currently require Clone + Send + Sync.
See architecture for extension contracts and the
roadmap for the intended sequence.