beave-rs

Codec wrappers

Base64<C>, Gzip<C>, and Zstd<C> wrap another codec C and transform its payload. Encoding runs the inner codec first and then the wrapper; decoding runs the wrapper first and passes its output to the inner codec. They accept any inner codec, including other wrappers, and implement Decoder or Encoder whenever the inner codec does, with the inner codec's value type.

Wrapper Cargo feature Payload
Base64<C> base64 Standard Base64 text of the inner payload
Gzip<C> gzip One gzip member containing the inner payload
Zstd<C> zstd One Zstandard frame containing the inner payload

Each wrapper implements Default when its inner codec does, so it works with adapters that construct their codec with C::default(). Use the adapter handle's source_with_codec or sink_with_codec to pass a wrapper with non-default settings or a configured inner codec such as Avro.

use beavers::adapters::sqs::Sqs;
use beavers::{Base64, Gzip, Protobuf};

// Protobuf bytes, gzip-compressed, carried as text in an SQS body.
let sink = Sqs::new().sink::<Base64<Gzip<Protobuf<Event>>>>(queue_url);

The order of nested wrappers is the order of the encoding steps from the inside out: Base64<Gzip<Json>> compresses JSON and then encodes the compressed bytes as Base64. The reader of the payload must use the same nesting.

Wrappers do not record what they did in the payload or in message metadata. A decoder therefore expects every payload in a stream to use the same wrapping; a subscription that receives both compressed and uncompressed payloads needs a custom codec that tells them apart.

Choosing compression

Gzip is readable by almost every platform; Zstandard compresses better and faster where both ends support it. Both decompress into memory and fail to decode once the output would exceed with_max_decoded_len, 64 MiB by default, so a small, highly compressible payload cannot exhaust memory.

On Kafka, prefer the producer's batch compression: set the librdkafka compression.type property, such as zstd, with Kafka::producer_property, and consumers decompress transparently. Batch compression sees repeated content across records and keeps payloads readable by any Kafka client. The Pulsar sink does not expose producer compression yet. The compression wrappers suit brokers without built-in compression, such as SQS, Pub/Sub, RabbitMQ, and Azure Service Bus, and payloads that must stay compressed end to end. Compressed payloads are binary, so SQS also needs Base64 around them.

Markdown · Source on GitHub