beave-rs

HTTP adapter

The HTTP adapter connects pipelines to services whose interface is HTTP. Enable it with the http feature:

[dependencies]
beavers = { version = "0.1", features = ["http"] }

Source

use beavers::{
    App, Json, Result, StdoutSink,
    adapters::http::{HttpRecord, HttpSource},
};
use serde::{Deserialize, Serialize};

#[derive(Clone, Deserialize, Serialize)]
struct Order {
    id: u64,
}

async fn accept(record: HttpRecord<Order>) -> Result<Order> {
    Ok(record.body)
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let source = HttpSource::<Json<Order>>::new("0.0.0.0:8080".parse()?)?;
    App::new()
        .subscribe("orders", source, StdoutSink::<Json<_>>::new(), accept)
        .run()
        .await?;
    Ok(())
}
curl -i -X POST localhost:8080/orders -d '{"id":7}'

Server configuration

The HTTP source has no shared handle; each source is its own server.

Setting Default Meaning
bind required Listen address passed to new. Port 0 selects a free port.
max_body_bytes(n) 1 MiB Larger bodies are answered with 413 and never become deliveries.
response(timing) ResponseTiming::Ack When a received request is answered; see response timing.

HttpSource::new(bind) binds the listener immediately, so an address in use is reported by the constructor and local_addr() returns the bound address, including a port chosen for port 0. HttpSource::with_codec(codec, bind) accepts a configured codec. validate checks the settings, and the server validates them again when it starts accepting connections on the first receive.

use beavers::{Utf8, adapters::http::{HttpSource, ResponseTiming}};

let source = HttpSource::<Utf8>::new("0.0.0.0:8080".parse()?)?
    .max_body_bytes(64 * 1024)
    .response(ResponseTiming::Receive);
# Ok::<(), Box<dyn std::error::Error>>(())

Records

Handlers receive HttpRecord<T>:

Field Content
path Request path, such as /orders
query Query string without ?, or None
headers Lowercase names and byte values, one entry per received value
body Body decoded by the source codec
metadata.remote_addr Peer address of the TCP connection

header(name) returns the first value of a header, matched case-insensitively. Behind a reverse proxy, remote_addr is the proxy; forwarding headers such as X-Forwarded-For remain in headers. The source accepts every path; a handler or middleware that serves only some paths rejects the others.

The raw form carried by dead letters is HttpRecord<Vec<u8>> with the undecoded body. Header values that are valid UTF-8, including traceparent, are the delivery's propagation fields, so with the opentelemetry feature the delivery span continues the client's trace.

Decoding

The server decodes each body with the source codec as soon as the request is read, before the request becomes a delivery. A body that fails to decode is answered with 400 Bad Request and the codec error as a plain-text body. It never reaches the subscription, so the error policy and dead-letter sink do not see it, and one malformed request cannot stop the subscription. SourceMessage::decode of an HTTP delivery therefore always succeeds.

Responses and delivery

With the default ResponseTiming::Ack, the server answers each decoded request once its delivery has an outcome:

Outcome Status
ACK after processing, dead-lettering, or discarding 200 OK
Delivery dropped without ACK: subscription failure, shutdown, or drain timeout 503 Service Unavailable
Request not yet received when the subscription stops receiving 503 Service Unavailable
Body that fails to decode 400 Bad Request with the codec error
Method other than POST 405 Method Not Allowed
Body larger than max_body_bytes 413 Payload Too Large
Body not readable, such as an aborted upload 400 Bad Request

A success status therefore means the subscription took responsibility for the request: its outputs were published, or the error policy dead-lettered or intentionally discarded it. A client that retries on 503, on a timeout, and on a closed connection gets at-least-once processing. Its retry can duplicate a request whose response was lost, so outputs should tolerate duplicates.

A client that disconnects while its request is processed does not cancel the delivery. The delivery is acknowledged normally and the response is dropped.

Handler rejections are dead-lettered by default and then answered with 200; without a dead-letter sink they stop the subscription, and the server with it.

Response timing

ResponseTiming::Receive answers 202 Accepted as soon as the subscription receives the decoded request, before processing. Clients get a response without waiting for the pipeline, but a success status no longer means the request was processed:

Use it when the producer cannot wait for processing and occasional loss on a crash is acceptable. Keep the default when every accepted request must be processed at least once.

Backpressure and ordering

Requests wait in a queue of one until the subscription receives them, so a subscription at its max_in_flight limit makes new requests wait instead of failing. Clients should apply a timeout. Requests have no ordering key; with concurrency above one they are processed in parallel and complete in any order.

Closing

When the subscription stops receiving, on application shutdown or a failure, the runtime calls stop_receiving before draining. The source then stops accepting connections, closes idle keep-alive connections, and answers queued requests that were not received with 503. New connections are refused while received requests drain; their connections stay open until each delivery completes and its response is sent.

close runs after draining, within the drain timeout. It does the same and then waits until open connections have answered their current request. Dropping the source also stops the listener.

Behind a load balancer, serve health checks with the health feature. Readiness fails as soon as shutdown starts, so a Kubernetes Service stops routing new requests to an instance before its source refuses them.

Server metrics

The server records these metrics through the metrics facade, alongside the subscription metrics. Each carries a listener label with the bound address.

Metric Type Extra labels Meaning
beavers_http_requests_total counter status Answered requests by response status
beavers_http_request_duration_seconds histogram status Time from receiving the request head to the response
beavers_http_requests_in_flight gauge Requests being read, decoded, or awaiting their delivery
beavers_http_connections_open gauge Open client connections
beavers_http_request_body_bytes histogram Size of bodies read within max_body_bytes

A request whose client disconnects before the response is not counted in beavers_http_requests_total. Decode failures appear as status="400" and never reach the subscription's beavers_delivery_failures_total.

Server limitations

Sink

HttpSink<C> accepts HttpPublish<T> outputs, where T is the codec's value type. It encodes each body with its codec and sends one request per output to its URL. A typical use is handing events consumed from a broker to a service with an HTTP API:

use beavers::{
    App, Json, Result,
    adapters::{
        http::{HttpClient, HttpPublish},
        kafka::{Kafka, KafkaRecord},
    },
};

async fn to_request(record: KafkaRecord<Order>) -> Result<HttpPublish<Order>> {
    let metadata = record.metadata.clone();
    let order = record.value.ok_or_else(|| anyhow::anyhow!("tombstone"))?;
    // A stable key lets the service recognize a redelivered event.
    let key = format!("{}-{}-{}", metadata.topic, metadata.partition, metadata.offset);
    Ok(HttpPublish::new(order).with_header("idempotency-key", key))
}

let source = Kafka::new("localhost:9092").source::<Json<Order>>("orders-to-billing", ["orders"]);
let sink = HttpClient::new()
    .sink::<Json<Order>>("https://billing.internal/orders")
    .header("content-type", "application/json");
App::new().subscribe("orders", source, sink, to_request).run().await?;

Sink configuration

An HttpClient handle holds the settings and the connection pool that the sinks built from it share. Its sinks send through one pool per handle, which keeps idle connections to each destination for reuse. The handle creates separate pools for http and https destinations, each on its first request, and loads the platform's root certificates only for the https pool. Idle connections close once the handle and every sink built from it have been dropped. Each handle builder method returns a handle with a pool of its own, so configure the handle before building sinks from it. The Debug output of the handle shows header names only.

Handle method Default Meaning
header(name, value) none Header sent with every request of every sink, such as authorization
timeout(duration) 30 s Limit for one attempt, from connecting until the response body is read, for sinks that set none

sink::<C>(url) builds a sink with the codec's default value, and sink_with_codec(codec, url) with a configured codec. The URL is required; the other settings are sink builder methods:

Sink setting Default Meaning
url required Absolute http or https URL, including any path and query
method(method) HttpMethod::Post Post, Put, or Patch
header(name, value) handle headers Header sent with every request, such as content-type; replaces the handle's headers of the same name
timeout(duration) handle timeout Limit for one attempt, replacing the handle's timeout
use beavers::{Json, adapters::http::{HttpClient, HttpMethod}};
use std::time::Duration;

let http = HttpClient::new()
    .header("authorization", "Bearer token")
    .timeout(Duration::from_secs(10));
let orders = http
    .sink::<Json<String>>("https://billing.example.com/orders")
    .header("content-type", "application/json");
let refunds = http
    .sink::<Json<String>>("https://billing.example.com/refunds")
    .method(HttpMethod::Put)
    .timeout(Duration::from_secs(60));

The handle and the sink each have validate. The sink's returns an error for a URL without a host, a scheme other than http or https, credentials in the URL, invalid header names or values, or a zero timeout, and the sink validates its settings again when it prepares its first output. Clones of a sink share its closing state.

https servers are verified against the platform's root certificates; a platform without any fails every https publication. The codec does not set content-type, so set it with header.

Requests

HttpPublish<T> carries the body and optional headers. A header name set on an output replaces every handle and sink header of that name; repeating a name on the output sends each value. Header names are case-insensitive, and invalid names or values fail prepare, which the error policy routes as Encode. The prepared request keeps the encoded body and complete header set, so a retry sends the same bytes without encoding again.

HttpPublish implements PropagationCarrier, so with the opentelemetry feature the outgoing request carries the delivery's traceparent header. HTTP records have no tombstones, so HttpPublish does not implement TombstonePublish, and Tombstones middleware for an HTTP sink does not compile.

Responses and retries

Each output waits for its response before the next output of the delivery is sent, and the delivery is acknowledged after every output succeeded:

Result Outcome
2xx Published
408, 429, or 5xx Retried under publish_retry
Connection failure or timeout Retried under publish_retry
Any other status, including 3xx redirects PublishRejected without retry

An exhausted retry stops the subscription without ACK, so a destination that is down leaves the input unacknowledged for redelivery after restart. A rejected output is routed by the error policy as FailureKind::PublishRejected, which dead-letters the input by default and stops the subscription when no dead-letter sink is configured. The error text names the method, URL without its query, status, and the start of the response body.

Delivery is at-least-once. A request whose response is lost or times out is sent again, the whole delivery is processed again after a crash or a failure of a later output, and outputs published before a rejection stay published. The destination should treat requests as idempotent, for example by keying them on the source record's position, as the example does. Redirects are not followed and Retry-After is not honored; retry delays come from publish_retry.

Outputs of one delivery are sent in order. Deliveries processed concurrently send their requests concurrently over pooled HTTP/1.1 connections, so the subscription's concurrency bounds the requests in flight to one destination.

Closing the sink

close only refuses further publications from the sink and all its clones. Requests already in progress finish, and the handle's connection pool stays open for its other sinks.

Client metrics

Each attempt is recorded through the metrics facade with a url label holding the scheme, host, and path, without the query, and an outcome label holding the response status, error for a failed connection or response, or timeout.

Metric Type Meaning
beavers_http_client_requests_total counter Request attempts, including retried ones
beavers_http_client_request_duration_seconds histogram Time from sending the request to reading the response body

Sink limitations

Markdown · Source on GitHub