Transactions
Subscription::transactional() replaces the final publish and ACK steps with
sink transactions that each commit a batch of deliveries:
first delivery: verify_source(delivery)
… → prepare all outputs → enlist in a batch → commit(batch)
Subscription::new("orders", kafka_source, kafka_transactional_sink, handler)
.transactional()
.transaction_batch(TransactionBatch::new(100, Duration::from_millis(10)))
The method only compiles when the sink implements
TransactionalSink<Source::Message, Output>, which an adapter provides for the
source messages whose acknowledgement can join its transactions. The
Kafka adapter implements it for a KafkaSource and a KafkaTransactionalSink,
which kafka.sink(topic).transactional(id) builds;
see the Kafka guide. The runtime never calls
the delivery's own ACK in a transactional subscription.
Batches
A delivery whose outputs are prepared joins the open batch, and its job ends
and frees its concurrency slot and ordering key, as a
pending completion does. The delivery counts as
acknowledged when its batch commits. A batch closes when it holds
max_deliveries deliveries or max_linger after its first delivery joined,
whichever comes first:
TransactionBatch field |
Default | Meaning |
|---|---|---|
max_deliveries |
100 | Maximum deliveries committed in one transaction |
max_linger |
10 ms | Maximum wait of a batch's first delivery for more deliveries |
Set it with Subscription::transaction_batch; a zero max_deliveries fails
validation. TransactionBatch::single() commits every delivery in its own
transaction without waiting. Other subscriptions ignore the setting.
Batches of one subscription commit one at a time, in a task of their own, so a
commit finishes or aborts even when its jobs are abandoned. Deliveries that
complete while a batch commits form the next batch, which commits as soon as
the previous one ends if it is already full or its linger has passed. At most
one further batch of deliveries waits for a commit: when it is full, the next
delivery's job waits to join, which bounds deliveries waiting for a commit
outside max_in_flight. At the end of input and on shutdown, the last batch
commits after its linger, within the drain timeout.
commit publishes every output of the batch and acknowledges every delivery
in it atomically. A failed commit leaves none in effect, and the whole batch is
retried with the same prepared outputs under the subscription's
publish_retry policy. Before each attempt, deliveries that were revoked or
abandoned leave the batch; an attempt that fails while a delivery of the batch
is revoked is retried without it and does not count against the policy. An
exhausted retry stops the subscription without acknowledging any delivery of
the batch, and later batches fail without committing, because a later
delivery's acknowledgement could cover a failed delivery of the same ordering
scope. A commit error marked with PublishRejected is not retried and fails the
batch the same way: it cannot be attributed to one delivery, so the error policy
does not route it.
A delivery that completes without output, including one that emitted nothing, was discarded, or was dead-lettered, joins a batch without outputs. Dead letters are published by the dead-letter sink before the delivery joins and are not part of the transaction, so they remain at-least-once.
Source verification
Types cannot tell whether a source and a sink of the same platform connect to
the same cluster, and a transaction can only acknowledge a delivery of its own
cluster. The runtime therefore passes the first delivery of a transactional
subscription to TransactionalSink::verify_source before processing it. The
check is retried under the publish_retry policy, and a failure stops the
subscription before any handler runs or anything is published or
acknowledged. Shutdown during the check leaves the delivery unacknowledged.
Ordering
A transactional subscription requires a
distribution that keeps the source
order, such as the default SourceOrder or KafkaDistribution::Partition.
Deliveries of one ordering scope then join batches in receive order, and a
batch lists them in that order, so the acknowledgement of a scope's last
delivery in a batch covers every earlier delivery of that scope. Other
distributions fail validation before the application starts.