beave-rs

Observability

The runtime reports through the tracing and metrics facades. Both do nothing until the application installs a subscriber or recorder, such as tracing-subscriber or a Prometheus exporter. Metric handles are registered when a subscription starts, so install the recorder before App::run.

Spans

subscription                 subscription = <name>
  └── message                one per delivery
       ├── decode
       ├── handler           one per attempt, with `attempt`
       ├── encode            `Sink::prepare` for every output
       ├── publish           every output, including publish retries
       ├── dead_letter       conversion and dead-letter publication
       └── ack
  └── commit                 transactional subscriptions: every attempt to commit
                             a batch, with `deliveries` and `outputs`

A transactional subscription records no publish or ack span. Its commit spans belong to the subscription, not to a delivery, because one transaction commits several deliveries.

The message span opens when the delivery is received, so it also covers the time the delivery waits behind earlier deliveries of its distribution key before its handler starts. pre_handler and post_handler middleware run in the message span.

Messaging attributes

Broker sources describe each delivery through SourceMessage::messaging_attributes, and the runtime records the OpenTelemetry messaging semantic conventions on its message span. With an OpenTelemetry layer, the span is exported as a consumer span named process <destination>, such as process orders, and carries messaging.operation.type and messaging.operation.name set to process together with:

Adapter messaging.system Destination Other attributes
Kafka kafka Topic messaging.destination.partition.id, messaging.kafka.offset, messaging.consumer.group.name
Pulsar pulsar Configured topic messaging.destination.partition.id for a partitioned topic, messaging.destination.subscription.name, messaging.message.id
RabbitMQ rabbitmq Queue messaging.message.id when the publisher set one
SQS aws_sqs Queue name messaging.message.id
Pub/Sub gcp_pubsub None messaging.destination.subscription.name, messaging.message.id
Azure Service Bus servicebus Queue or topic messaging.destination.subscription.name for a topic subscription, messaging.message.id when set

Other sources report no attributes, and their span keeps the name message. A custom source returns a MessagingAttributes built from its own metadata. Metrics do not carry these attributes, because partitions and message IDs would multiply the number of series. Outputs get no producer span of their own; the trace context of the message span continues downstream.

Events

Event Level Fields
subscription started / subscription stopped INFO
subscription failed ERROR error
discarding delivery WARN failure, attempts, error
dead-lettered delivery WARN failure, attempts, error
retrying receive WARN attempt, delay, error
retrying handler / retrying publish DEBUG attempt, error
abandoned revoked delivery DEBUG

A discarded delivery is always logged and counted, because FailureAction::Discard leaves no other trace.

Metrics

Every metric carries a subscription label with the subscription name.

Metric Type Additional labels Meaning
beavers_deliveries_received_total counter Deliveries received from the source
beavers_deliveries_acknowledged_total counter Successful ACKs or transaction commits, including after discard and dead-lettering
beavers_delivery_failures_total counter failure, action Routable failures by FailureKind and the action taken
beavers_handler_retries_total counter Handler attempts that returned Retry and were retried
beavers_publish_failures_total counter sink (output, dead_letter) Failed publish or transaction commit attempts, including retried ones
beavers_receive_errors_total counter Failed receive attempts
beavers_deliveries_revoked_total counter Deliveries abandoned after revocation
beavers_deliveries_in_flight gauge Received deliveries that have not finished
beavers_source_connected gauge 1 while the source is connected to its broker, 0 while it is not; Kafka and Pulsar sources only
beavers_stage_duration_seconds histogram stage Duration of decode, handler, encode, publish, complete, dead_letter, ack, commit, and state_commit
beavers_transaction_deliveries histogram Deliveries in each committed transaction

failure is decode, rejected, retry_exhausted, encode, or publish_rejected. action is stop, dead_letter, or discard; a DeadLetter action without a dead-letter sink is reported as stop, which is what it does. A failure is counted when it is routed, before the action runs. handler durations are per attempt and exclude retry backoff; publish durations cover all outputs of a delivery, including backoff. commit durations are per successful transaction and cover the whole batch. The connection gauge copies the source's health probe every 5 seconds and is not reported until the source knows its state.

Alerting

beavers sends no notifications itself. Alert on the metrics with a monitoring system and let it route alerts, since per-message notifications flood a channel when many deliveries fail at once. The monitoring and alerting guide lists the signals worth alerting on, PromQL alert rules, and configurations for Kubernetes, AWS, Google Cloud, and Azure. A failed subscription also makes the liveness probe fail.

Exporting metrics to OpenTelemetry

beavers does not bridge the metrics facade to OpenTelemetry metrics. An application that sends metrics through an OpenTelemetry Collector exposes them in the Prometheus format with metrics-exporter-prometheus and lets the Collector scrape them.

Install the exporter before App::run:

use metrics_exporter_prometheus::{Matcher, PrometheusBuilder};

PrometheusBuilder::new()
    .with_http_listener(([0, 0, 0, 0], 9000))
    .set_buckets_for_metric(
        Matcher::Full("beavers_stage_duration_seconds".to_owned()),
        &[0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0],
    )?
    .install()?;

app.run().await?;

The listener serves /metrics over HTTP. Without configured buckets, the exporter writes histograms as Prometheus summaries, which arrive as OpenTelemetry summaries that many backends do not support; configure buckets for beavers_stage_duration_seconds and any histograms the application records itself.

Scrape the endpoint with the Collector's Prometheus receiver, which is part of the otelcol-contrib distribution, and forward the metrics to any exporter:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: beavers
          scrape_interval: 15s
          static_configs:
            - targets: ["my-app:9000"]

exporters:
  otlp:
    endpoint: my-backend:4317

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [otlp]

Counters arrive as cumulative OpenTelemetry sums, the in-flight gauge as a gauge, and bucketed histograms as histograms. Labels become data-point attributes, and the receiver maps the scrape's job and instance to the service.name and service.instance.id resource attributes.

Trace-context propagation

SourceMessage::propagation_fields exposes text-map fields received with a delivery, and PropagationCarrier lets an output record accept them. Each adapter maps the fields to its own metadata:

Adapter Received fields Output fields
Kafka Headers with UTF-8 values Headers; every header of the same name is replaced
Pulsar Properties Properties
RabbitMQ String headers String headers
SQS String message attributes String message attributes
Pub/Sub Attributes Attributes
Azure Service Bus String application properties String application properties
Channel The sender's trace context, captured when a value is enqueued Captured automatically; no carrier needed
Delivery Fields set with Delivery::with_propagation_fields Not supported
Other local adapters None Not supported

With the opentelemetry feature, the runtime extracts each message span's remote parent from those fields through the global text-map propagator, and the TraceContext middleware injects the message span's context into every output:

A channel needs no middleware. With the opentelemetry feature, ChannelSink and ChannelSender capture the current span's context when they enqueue a value, so the receiving subscription's message span continues the trace as a child of the sending stage.

use beavers::TraceContext;

opentelemetry::global::set_text_map_propagator(TraceContextPropagator::new());
// Install a tracing subscriber with a tracing-opentelemetry layer.

Subscription::new("orders", kafka_source, kafka_sink, handler)
    .middleware(TraceContext::new())

Inheritance middleware such as KafkaInherit copies every header, including traceparent, unchanged. Without TraceContext, an output therefore carries the received context, and the processing step is invisible to the trace. With TraceContext registered after the inheritance middleware, the injected context replaces the inherited one, so downstream consumers continue the trace as children of the processing step. Middleware runs in registration order, and a handler returning a plain value registers the inheritance middleware first.

Markdown · Source on GitHub