Codec wrappers
Base64<C>, Gzip<C>, and Zstd<C> wrap another codec C and transform its
payload. Encoding runs the inner codec first and then the wrapper; decoding runs
the wrapper first and passes its output to the inner codec. They accept any
inner codec, including other wrappers, and implement Decoder or
Encoder whenever the inner codec does, with the inner codec's value type.
| Wrapper | Cargo feature | Payload |
|---|---|---|
Base64<C> |
base64 |
Standard Base64 text of the inner payload |
Gzip<C> |
gzip |
One gzip member containing the inner payload |
Zstd<C> |
zstd |
One Zstandard frame containing the inner payload |
Each wrapper implements Default when its inner codec does, so it works with
adapters that construct their codec with C::default(). Use the adapter
handle's source_with_codec or sink_with_codec to pass a wrapper with
non-default settings or a configured inner codec such as Avro.
use beavers::adapters::sqs::Sqs;
use beavers::{Base64, Gzip, Protobuf};
// Protobuf bytes, gzip-compressed, carried as text in an SQS body.
let sink = Sqs::new().sink::<Base64<Gzip<Protobuf<Event>>>>(queue_url);
The order of nested wrappers is the order of the encoding steps from the inside
out: Base64<Gzip<Json>> compresses JSON and then encodes the compressed bytes
as Base64. The reader of the payload must use the same nesting.
Wrappers do not record what they did in the payload or in message metadata. A decoder therefore expects every payload in a stream to use the same wrapping; a subscription that receives both compressed and uncompressed payloads needs a custom codec that tells them apart.
Choosing compression
Gzip is readable by almost every platform; Zstandard compresses better and
faster where both ends support it. Both decompress into memory and fail to
decode once the output would exceed with_max_decoded_len, 64 MiB by default,
so a small, highly compressible payload cannot exhaust memory.
On Kafka, prefer the producer's batch compression: set the librdkafka
compression.type property, such as zstd, with Kafka::producer_property,
and consumers decompress transparently. Batch compression sees repeated
content across records and keeps payloads readable by any Kafka client. The
Pulsar sink does not expose producer compression yet. The compression wrappers
suit brokers without built-in compression, such as SQS, Pub/Sub, RabbitMQ, and
Azure Service Bus, and payloads that must stay compressed end to end.
Compressed payloads are binary, so SQS also needs Base64 around them.