beave-rs

Health checks

App::health() returns a cloneable Health handle. Obtain it before App::run; it reports every subscription registered with the application. Health::report() returns a HealthReport for the current moment:

Field Meaning
live false once a subscription has failed, which stops the application, or while a subscription is stalled
ready The application is running, shutdown has not started, and every subscription is ready
shutting_down Shutdown was requested by a signal, the run_until token, or a failure
subscriptions One SubscriptionReport per subscription, in registration order

Each SubscriptionReport carries the subscription name, its status (Pending before the application runs, then Running, Stopping while it drains, and finally Stopped or Failed), receive_failures, the consecutive failed receives since the last successful one, receive_backoff, whether the runtime is waiting for a receive retry delay, publish_retries, the deliveries whose output publication or transaction commit failed and is waiting to be retried, connected, the source's connection state, and stalled. A subscription is ready while it is Running with no receive backoff, no publish retry, and a source that does not report itself disconnected. A subscription whose source ended and that stopped without an error stays ready, so a finite source next to a long-running one does not make the application unready.

A retried receive that waits for input counts as ready again, because the runtime cannot tell an idle broker from an unreachable one while a receive is pending; the source's connection state covers that case where the source reports it. Dead-letter publish retries do not affect readiness.

Connection state

A source can report whether it is connected to its broker through Source::probe, which the health report reads on every request:

Adapter connected
Kafka Any broker connection is up, from librdkafka statistics
Pulsar Every partition's consumer is connected
Other adapters Not reported

The value is null until the source has connected, and for Kafka until the first statistics arrive. The Kafka source sets statistics.interval.ms to 5000 unless the handle or source sets it; 0 disables statistics and leaves the state unreported. librdkafka delivers statistics while the source polls its consumer, which the runtime does whenever it has room to receive, so while every delivery slot stays busy the Kafka state keeps its last reading. A source reporting connected: false makes its subscription unready. The Kafka and Pulsar clients reconnect by themselves, and readiness returns when they do.

beavers does not report consumer lag. Alert on the broker's own measurement, listed in the monitoring guide, which covers every instance of a consumer group and an application that is not running.

Stall detection

Liveness reports that the runtime answers and that no subscription failed. With stall_timeout, it also fails while a subscription is stalled: deliveries are in flight, but no handler attempt, publish or commit attempt, acknowledgement, or newly received delivery has occurred for that duration.

Subscription::new("orders", source, sink, handler)
    .stall_timeout(Duration::from_secs(300))

An idle subscription with nothing in flight is never stalled, and each failed retry attempt counts as progress, so backoff does not trigger it. Stall detection is disabled by default; choose a duration longer than the slowest expected handler attempt, because a restart abandons the deliveries in progress. It detects a subscription in which nothing progresses. One stuck delivery while others still complete, such as one partition blocked under the default SourceOrder distribution, keeps the subscription live; its partition's commit stops advancing, which shows as growing consumer lag on the broker.

With the health feature, HealthServer answers probes over HTTP/1.1 while the application runs:

let server = HealthServer::bind("0.0.0.0:8080".parse()?)?;
App::new()
    .subscription(subscription)
    .health_server(server)
    .run()
    .await?;

GET /livez answers 200 OK when the report is live and GET /readyz when it is ready; otherwise each answers 503 Service Unavailable. Both return the report as a JSON body. Other paths answer 404 Not Found and other methods 405 Method Not Allowed. HealthServer::bind binds the listener immediately, so address errors surface before the application runs and local_addr reports the port chosen for port 0. The server starts with the application and stops after every subscription has finished. Readiness therefore turns 503 as soon as shutdown starts, which removes an instance serving an HTTP source from a Kubernetes Service while it drains, and liveness keeps answering until draining completes. Use the readiness probe for traffic routing and the liveness probe for restarts:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }

The health server does not export metrics; metrics go through the metrics facade and the recorder the application installs.

Markdown · Source on GitHub