beave-rs

Codecs

Codecs convert between serialized bytes and typed values. They do not receive messages, publish output, route metadata, or acknowledge deliveries. Those responsibilities belong to adapters and the subscription runtime.

Contracts

A codec names the typed value it converts, and implements decoding, encoding, or both:

pub trait Codec: Send + Sync + 'static {
    type Value;
}

pub trait Decoder: Codec {
    fn decode(&self, bytes: &[u8]) -> Result<Self::Value, BoxError>;
}

pub trait Encoder: Codec {
    fn encode(&self, value: &Self::Value) -> Result<Vec<u8>, BoxError>;
}

Sources and sinks take their payload type from the codec, so the codec is their only type parameter. A format that serializes many types carries the type as its parameter, as in Json<Order>; a codec bound to one type, such as Utf8 for String, has none:

let orders: KafkaSource<Json<Order>> = KafkaSource::new(source_config);
let invoices: KafkaSink<Avro<Invoice>> = KafkaSink::new(sink_config);
let lines: KafkaSink<Utf8> = KafkaSink::new(lines_config);

When a handler parameter fixes the type, the codec parameter can be left to inference, as in StdinSource::<Json<_>>::new(). These are synchronous transformations; implementations should not perform blocking network or filesystem operations. Expensive serialization still occupies the executing thread; there is currently no dedicated codec execution pool.

The traits do not require Serde. Each codec chooses its own representation and payload bounds. Codec, Decoder, Encoder, Json, RawBytes, and Utf8 are available through beavers::codec and re-exported at the crate root. Protobuf, Avro, MessagePack, Cbor, and the Base64, Gzip, and Zstd wrappers are available when their corresponding Cargo features are enabled.

Lifecycle and failures

For the current stdin-to-stdout pipeline:

receive raw line
    → SourceMessage::decode (Decoder)
    → handler
    → output middleware
    → Sink::prepare (Encoder + line framing)
    → Sink::publish
    → ACK

Decode runs once per received message before handler attempts. Handler retries reuse cloned decoded input. The runtime prepares all emitted outputs before publishing any; publish retries reuse the same prepared representation and do not rerun encoding.

Codec errors return as BoxError, so an implementation can propagate any error type with ?. Neither decode nor preparation failures are retried, because the same bytes or value would fail again. By default both stop processing without ACK; the subscription's error policy can instead dead-letter or discard them as FailureKind::Decode and FailureKind::Encode. An invalid JSON line is a decode failure, whereas a failed stdin read is a receive failure.

Implementing a codec

Implement Codec to name the value type, then the relevant direction. For example, a UTF-8 string codec needs no Serde dependency:

use beavers::{BoxError, Codec, Decoder, Encoder};

#[derive(Default)]
struct Utf8;

impl Codec for Utf8 {
    type Value = String;
}

impl Decoder for Utf8 {
    fn decode(&self, bytes: &[u8]) -> Result<String, BoxError> {
        Ok(std::str::from_utf8(bytes)?.to_owned())
    }
}

impl Encoder for Utf8 {
    fn encode(&self, value: &String) -> Result<Vec<u8>, BoxError> {
        Ok(value.as_bytes().to_vec())
    }
}

A codec for a format that serializes many types takes the type as a parameter and holds it in PhantomData<fn() -> T>, as Json<T> does, so the codec stays Send, Sync, and Copy whatever T is.

Utf8 is an exported built-in codec; see Utf8 for its exact behavior and RawBytes for arbitrary payloads. The local stdin and stdout adapters default-construct their codec, so using a custom codec with them requires Default. The codec traits themselves do not require it. Broker adapter handles also provide source_with_codec(codec, ...) and sink_with_codec(codec, ...) for existing configured codec instances.

Choose framing-compatible codecs for line-based adapters. A binary encoding can contain newline bytes, and an arbitrary string may contain embedded newlines; neither automatically forms a safe one-record-per-line protocol. The UTF-8 example preserves all bytes, including any newline supplied by the source.

Available formats and broker integration

Json converts between JSON and any Serde type. See JSON.

RawBytes preserves arbitrary Vec<u8> payloads and Utf8 converts valid UTF-8 between bytes and String. See RawBytes and Utf8.

The optional protobuf feature provides Protobuf for raw prost::Message payloads. The optional avro feature provides schema-bound Avro values. Both formats leave framing to the adapter. See Protobuf and Avro.

The optional msgpack and cbor features provide MessagePack and Cbor, compact binary formats for the same Serde types as Json. See MessagePack and CBOR.

The optional base64, gzip, and zstd features provide Base64<C>, Gzip<C>, and Zstd<C>, which wrap another codec to carry its payload as text or compress it. See codec wrappers.

Kafka applies its selected codec to a record value; keys and headers remain raw bytes in KafkaRecord<T>, and a value can be null. Pulsar applies its selected codec to a non-null payload while exposing key and properties as metadata; by default an empty payload is a null value. A null value never reaches the codec. Neither adapter implicitly carries received metadata into outgoing records. Metadata mapping is explicit output middleware; see the runtime guide.

Markdown · Source on GitHub