Monitoring and alerting
beavers sends no notifications itself. It reports metrics through the
metrics facade, structured events through
tracing, and health through the /livez and
/readyz endpoints of the health server. This
guide explains which of those signals deserve an alert and how to wire them to
a platform that pages people, for teams that do not already run Prometheus and
Alertmanager.
| Platform | Guide |
|---|---|
| Kubernetes with a self-managed Prometheus Operator | Kubernetes |
| Amazon EKS, ECS, or EC2 | AWS: Amazon Managed Service for Prometheus or CloudWatch |
| GKE or Cloud Run | Google Cloud: Managed Service for Prometheus and Cloud Monitoring |
| AKS or Azure Container Apps | Azure: Azure Monitor managed service for Prometheus and Azure Monitor alerts |
The managed Prometheus services of all three clouds evaluate the alert rules below unchanged, so most of each platform guide is about collecting the metrics and routing the alerts.
What to expose
Every deployment needs the same three outputs from the application.
Metrics. Install a Prometheus exporter before App::run, as described in
Exporting metrics, and
configure histogram buckets for beavers_stage_duration_seconds. The examples
in this guide assume the exporter listens on port 9000 and serves /metrics.
Health. Enable the health feature and register a HealthServer. The
examples assume port 8080. Readiness turns 503 during shutdown, receive
backoff, output publish retries, and while a Kafka or Pulsar source is
disconnected; liveness turns 503 after a subscription fails, or while it is
stalled when a stall timeout is set.
JSON logs. Emit events as JSON so that cloud log services can match the
subscription failed event. With tracing-subscriber and its json
feature:
tracing_subscriber::fmt().json().init();
Each event becomes one line on standard output. The message is in
fields.message, so a failed subscription appears as:
{"timestamp":"…","level":"ERROR","fields":{"message":"subscription failed","error":"…"},"target":"beavers::subscription::runtime"}
What to alert on
A failed subscription stops the whole application: App::run returns the
error after the other subscriptions drain, and the process usually exits. The
last increment of beavers_delivery_failures_total{action="stop"} may
therefore never be scraped. Cover that case with at least one signal that
outlives the process: a restart alert from the orchestrator or a log-based
alert on subscription failed. Each platform guide shows both.
| Situation | Signal | Severity | Why |
|---|---|---|---|
| A subscription stopped on an unroutable failure | beavers_delivery_failures_total{action="stop"}, container restarts, the subscription failed log event |
critical | Consumption stopped; the instance restarts and may fail again on the same record |
| Dead letters cannot be published | beavers_publish_failures_total{sink="dead_letter"} |
critical | Failed deliveries cannot leave the subscription, so it stops making progress |
| Outputs cannot be published | beavers_publish_failures_total{sink="output"} for several minutes |
critical | Deliveries wait for publish retries; readiness is 503 meanwhile |
| The source keeps failing | beavers_receive_errors_total for several minutes |
critical | The broker is unreachable or rejects the client; readiness is 503 during backoff |
| Work is stuck | beavers_deliveries_in_flight above zero with no acknowledgement |
critical | A handler or acknowledgement never completes; liveness detects this only with a stall timeout |
| The source lost its broker | beavers_source_connected at 0 for several minutes |
critical | The Kafka or Pulsar client keeps reconnecting; readiness is 503 meanwhile |
| Metrics disappeared | The scrape target is down | critical | Nothing else on this list can fire |
| Deliveries were dead-lettered | beavers_delivery_failures_total{action="dead_letter"} |
warning | Someone needs to inspect and redrive them; see the dead-letter handling guide |
| Producers send undecodable records | Ratio of failure="decode" to received deliveries |
warning | Usually a schema or producer change |
| Handlers are slow | The 99th percentile of beavers_stage_duration_seconds{stage="handler"} |
warning | The backlog grows when processing falls behind the input rate |
| The backlog grows | Consumer lag or queue depth from the broker | warning | beavers does not report lag; see Backlog |
Send warning alerts to the owning team's channel and critical alerts to
on-call paging. A dead letter needs investigation during working hours, while a
stopped subscription or a failing dead-letter sink stops consumption until
someone acts.
Alert on counts over a window, never on individual failures. A burst of bad records produces one alert per subscription instead of one notification per record.
Alert rules
These rules use PromQL. Prometheus, Amazon Managed Service for Prometheus, Google Cloud Managed Service for Prometheus, and Azure Monitor managed service for Prometheus evaluate them as written; the platform guides show where each expects them. Thresholds and windows are starting points; tune them to the input rate and latency of each subscription.
groups:
- name: beavers
rules:
- alert: BeaversStoppedOnFailure
expr: sum by (subscription) (increase(beavers_delivery_failures_total{action="stop"}[15m])) > 0
labels:
severity: critical
annotations:
summary: "{{ $labels.subscription }} stopped on an unroutable failure"
- alert: BeaversDeadLetterPublishFailing
expr: sum by (subscription) (increase(beavers_publish_failures_total{sink="dead_letter"}[5m])) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.subscription }} cannot publish dead letters"
- alert: BeaversOutputPublishFailing
expr: sum by (subscription) (rate(beavers_publish_failures_total{sink="output"}[5m])) > 0
for: 10m
labels:
severity: critical
annotations:
summary: "{{ $labels.subscription }} has failed to publish outputs for 10 minutes"
- alert: BeaversReceiveFailing
expr: sum by (subscription) (rate(beavers_receive_errors_total[5m])) > 0
for: 10m
labels:
severity: critical
annotations:
summary: "{{ $labels.subscription }} has failed to receive from its source for 10 minutes"
- alert: BeaversStalled
expr: |
max by (subscription) (beavers_deliveries_in_flight) > 0
and sum by (subscription) (increase(beavers_deliveries_acknowledged_total[15m])) == 0
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.subscription }} has unfinished deliveries and acknowledged none in 15 minutes"
- alert: BeaversMetricsDown
expr: up{job="beavers"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.instance }} does not serve metrics"
- alert: BeaversDeadLetters
expr: sum by (subscription) (increase(beavers_delivery_failures_total{action="dead_letter"}[10m])) > 0
labels:
severity: warning
annotations:
summary: "{{ $labels.subscription }} dead-lettered {{ $value }} deliveries in 10 minutes"
- alert: BeaversDecodeFailureRatio
expr: |
sum by (subscription) (rate(beavers_delivery_failures_total{failure="decode"}[5m]))
/ sum by (subscription) (rate(beavers_deliveries_received_total[5m])) > 0.01
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.subscription }} cannot decode over 1% of its input; check producers and schemas"
- alert: BeaversHandlerSlow
expr: |
histogram_quantile(0.99,
sum by (subscription, le) (rate(beavers_stage_duration_seconds_bucket{stage="handler"}[10m]))
) > 5
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.subscription }} handler p99 is above 5 seconds"
BeaversMetricsDown matches the job label that the scrape configuration
assigns; replace beavers with that value. It fires while an instance runs
but does not answer scrapes; an instance that has gone away leaves no up
series, which the restart and backlog alerts cover. BeaversHandlerSlow needs
histogram buckets, and its threshold must lie within them. BeaversStalled
also fires for a subscription whose handler legitimately runs longer than 15
minutes; raise the window above the longest expected handler duration.
Backlog
beavers does not report consumer lag. Alert on the broker's own measurement of the backlog, which covers every instance of a consumer group and also an application that is not running at all:
| Broker | Metric |
|---|---|
| Kafka with kafka-exporter | kafka_consumergroup_lag by consumergroup and topic |
| Amazon MSK | MaxOffsetLag, SumOffsetLag, or EstimatedMaxTimeLag by consumer group and topic |
| Pulsar | pulsar_subscription_back_log from the broker's /metrics |
RabbitMQ with the rabbitmq_prometheus plugin |
rabbitmq_queue_messages_ready from the per-object endpoint |
| Amazon SQS | ApproximateAgeOfOldestMessage and ApproximateNumberOfMessagesVisible |
Alert when the backlog keeps growing rather than when it is above zero, for
example deriv(sum by (consumergroup) (kafka_consumergroup_lag)[15m:]) > 0
for 30 minutes, or when the age of the oldest message exceeds what the
consumers of the output can tolerate. Apply the same alerts to a dead-letter
topic consumed by a reprocessing subscription to see
how many dead letters remain unhandled.