SQS adapter
The optional sqs feature provides typed Amazon SQS source and sink adapters
backed by the AWS SDK for Rust (aws-sdk-sqs).
beavers = { version = "0.1", features = ["sqs"] }
Configuration
An Sqs handle holds how the adapter reaches SQS. region, endpoint_url,
and credentials are optional builder methods on it: unset, the SDK's default
chains resolve the region and credentials from the environment, shared
configuration files, web identity, or container and instance metadata. Set
endpoint_url to use an SQS-compatible emulator such as ElasticMQ or
LocalStack. SqsCredentials::new takes an access key ID and secret access key;
the Debug output of SqsCredentials hides the secret key and session token.
handle.source::<C>(queue_url) and handle.sink::<C>(queue_url) build a
source and a sink for one queue; source_with_codec and sink_with_codec take
a codec value as their first argument instead of using the codec's default. A
URL ending in .fifo names a FIFO queue.
use beavers::{Utf8, adapters::sqs::{Sqs, SqsPublish}};
let sqs = Sqs::new().region("eu-west-1");
let source = sqs.source::<Utf8>(
"https://sqs.eu-west-1.amazonaws.com/123456789012/orders",
);
let sink = sqs.sink::<Utf8>(
"https://sqs.eu-west-1.amazonaws.com/123456789012/processed-orders.fifo",
);
let output = SqsPublish::new("created".to_owned()).with_message_group_id("customer-7");
All sources and sinks built from one handle share one SDK client, with its
credential provider and HTTP connection pool. The client is created on the
first receive or publication of any of them, which also validates the
settings. Each builder method on the handle returns a handle with a client of
its own, so configure the handle before building sources and sinks from it.
The handle, the source, and the sink each have validate, which checks the
values without contacting AWS.
The adapters create no queues and set no queue attributes such as a redrive policy. Create them with infrastructure tooling before the subscription starts.
The source receives with long polling. Its max_messages (1 to 10, default
10) is the batch size of each ReceiveMessage call and wait_time
(at most 20 seconds, the default) how long a call waits for messages.
visibility_timeout, 30 seconds by default, is requested with every receive and
overrides the queue's own default.
Records
SqsRecord<T> is the decoded input and SqsPublish<T> the output. The codec
applies to the message body. SQS has no null body, so the value is always
present and SQS records support no tombstones. SQS bodies are text: prepare
rejects an encoded value that is not UTF-8, so binary codecs such as Protobuf
and Avro need a text representation, such as the
Base64 wrapper, before they can be sent.
attributes holds the message attributes as SqsAttributeValue::String,
Number in its decimal text form, or Binary. A custom type label such as
Number.float is read as its base type and not published again. SQS accepts at
most 10 attributes per message.
SqsMetadata contains read-only facts about a received message: the queue URL,
message ID, receive count, sent and first-receive timestamps, and for FIFO
queues the message group, deduplication ID, and sequence number. SqsPublish
has none of them except the FIFO message group and deduplication ID, which the
output sets itself, and an optional delivery delay.
The raw form of an SqsMessage is an SqsRecord<Vec<u8>> with the undecoded
body. Dead letters carry it, so the original body, attributes, and metadata
survive even when decoding fails.
Acknowledgements and ordering
A received message stays invisible to other consumers for the visibility
timeout. The source keeps every message it has received and not yet deleted
invisible: a background task extends a message's visibility with
ChangeMessageVisibilityBatch once a third of its timeout has passed, until it
is acknowledged or released. SQS limits visibility to 12 hours from the receipt;
a message held longer becomes visible again.
A successful ACK deletes the message with DeleteMessage, retried under the
source's ack_retry. Dropping an SqsMessage without acknowledging it,
or closing the source, changes the visibility of its remaining messages to zero,
so another receive takes them at once with an incremented receive count.
A receipt handle is valid only for the receipt that returned it. When an extension fails or arrives too late, SQS may already have handed the message to another consumer, so the adapter cancels that delivery's revocation token. The runtime abandons a revoked delivery without ACK or subscription failure. An acknowledgement that finds its visibility expired, or that SQS rejects as an invalid receipt handle, also revokes the delivery and fails.
A standard queue delivers messages in no particular order, so its deliveries
have no ordering key and run in parallel up to the subscription's
concurrency. A FIFO queue delivers each message group in order; each delivery's
ordering key is its message group, so the default SourceOrder distribution
processes one
message per group at a time while groups run in parallel.
ReceiveMessage runs as a task the source keeps: dropping a pending receive
leaves the call running, and the next receive takes its messages.
Publication
prepare encodes the value and converts the attributes once, so publish
retries reuse them. It requires a message group ID for a FIFO queue, rejects a
per-message delay there because FIFO queues accept only a queue-wide delay, and
limits the delay to 15 minutes elsewhere. A FIFO queue without content-based
deduplication also needs deduplication_id.
SqsSink::publish sends the message with SendMessage and succeeds once SQS
has stored it. The SDK retries throttling and transient errors itself.
Parameter errors such as an invalid attribute fail as PublishRejected, which
the subscription's error policy can dead-letter instead of retrying. The sink
keeps no pending messages, so close only stops new publications.
Metadata inheritance
Register SqsInherit on an SQS-to-SQS subscription to forward the received
message attributes:
use beavers::adapters::sqs::SqsInherit;
Subscription::new("orders", sqs_source, sqs_sink, handler)
.middleware(SqsInherit::new().with_message_group())
A received attribute is added only when the output does not set that name, and
dead-letter attributes starting with beavers-dlq- are never inherited.
with_message_group() also forwards the FIFO message group when the output
leaves it None, for pipelines between FIFO queues. The message ID,
deduplication ID, and delay are never inherited. without_attributes()
disables attribute inheritance.
A handler that returns a plain value between an SQS source and sink gets
SqsInherit::new() automatically; register
SqsInherit::new().with_message_group() explicitly to forward groups. See the
runtime guide. Register
TraceContext after SqsInherit to replace inherited trace-context
attributes; the source reads trace context from string attributes.
Dead letters
SqsPublish::from_dead_letter forwards a dead letter with its original body,
attributes, and FIFO message group. Because SQS accepts at most 10 attributes,
the failure details and the URL and message ID of the first receipt are written
as one JSON object into the beavers-dlq-details string attribute; a message
that already has 10 attributes is rejected by SQS. SqsDeadLetter::from_record
reads them back; see
forwarding dead letters.
A queue's redrive policy moves a message to its dead-letter queue after
maxReceiveCount receives without deletion. The subscription's error policy
decides what happens to a failed delivery first, so the redrive policy applies
to messages that are received again and again without being acknowledged, for
example because they crash the process every time. Keep maxReceiveCount high
enough that the releases of a source's shutdowns do not exhaust it.
Delivery guarantees
A source ACK and a sink publication are separate requests, so a failure between them, or a visibility timeout that expires before the deletion, can produce a duplicate. Standard queues may also deliver a message more than once. SQS pipelines are at-least-once; FIFO deduplication applies only to messages sent within its five-minute window.