beave-rs

Google Cloud

Google Cloud Managed Service for Prometheus stores the metrics in Cloud Monitoring, and Cloud Monitoring alerting policies evaluate the shared PromQL rules against them. Notifications go to Cloud Monitoring notification channels, such as email, Slack, PagerDuty, SMS, webhooks, or Pub/Sub, so no Alertmanager is needed.

Collect on GKE

Managed collection is enabled by default on recent GKE Autopilot and Standard clusters. Expose the ports and probes as in the Kubernetes Deployment, then select the pods with a PodMonitoring resource:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: beavers
  namespace: orders
spec:
  selector:
    matchLabels:
      app: orders-consumer
  endpoints:
    - port: metrics
      interval: 30s

The job label of the collected series is the PodMonitoring name, beavers, which the BeaversMetricsDown rule expects.

Collect on Cloud Run

A Cloud Run service that consumes from a broker must keep running between requests: use instance-based billing and at least one minimum instance. Run the OpenTelemetry Collector as a sidecar container that scrapes the exporter and writes to Managed Service for Prometheus:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: beavers
          scrape_interval: 30s
          static_configs:
            - targets: ["localhost:9000"]

processors:
  resourcedetection:
    detectors: [env, gcp]
  batch: {}

exporters:
  googlemanagedprometheus: {}

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [resourcedetection, batch]
      exporters: [googlemanagedprometheus]

The googlemanagedprometheus exporter is part of the otelcol-contrib distribution and of Google's built OpenTelemetry Collector image. The service account needs the roles/monitoring.metricWriter role. Add a liveness probe on /livez to the application container; Cloud Run then replaces an instance whose subscription failed.

Alerting policies

Create one alerting policy per rule. Each condition holds the PromQL expression of a shared rule unchanged:

# dead-letters.yaml
displayName: "beavers: dead letters"
severity: WARNING
combiner: OR
conditions:
  - displayName: Deliveries were dead-lettered
    conditionPrometheusQueryLanguage:
      query: sum by (subscription) (increase(beavers_delivery_failures_total{action="dead_letter"}[10m])) > 0
      duration: 0s
      evaluationInterval: 60s
      disableMetricValidation: true
notificationChannels:
  - projects/my-project/notificationChannels/1234567890
documentation:
  mimeType: text/markdown
  content: |
    Inspect the dead-letter topic and redrive the deliveries once the cause is fixed.
# stalled.yaml
displayName: "beavers: stalled subscription"
severity: CRITICAL
combiner: OR
conditions:
  - displayName: Unfinished deliveries without acknowledgements
    conditionPrometheusQueryLanguage:
      query: |
        max by (subscription) (beavers_deliveries_in_flight) > 0
          and sum by (subscription) (increase(beavers_deliveries_acknowledged_total[15m])) == 0
      duration: 300s
      evaluationInterval: 60s
      disableMetricValidation: true
notificationChannels:
  - projects/my-project/notificationChannels/9876543210
gcloud monitoring policies create --policy-from-file=dead-letters.yaml
gcloud monitoring policies create --policy-from-file=stalled.yaml

The rule's for becomes duration, and the policy's severity takes the value of the rule's severity label. A PromQL policy holds exactly one condition. disableMetricValidation lets the policy be created before the application has written its first metrics; without it, creation fails for a metric that does not exist yet. A PromQL incident closes on its own once the query stops matching, so these policies need no autoClose. Give critical policies the on-call channel and warning policies the team's channel. Each time series that matches the query opens its own incident, so a policy notifies once per failing subscription.

On GKE, restarts are available as a Cloud Monitoring system metric, which PromQL reads under its converted name:

increase(kubernetes_io:container_restart_count{namespace_name="orders", container_name="app"}[15m]) > 0

Log-based alert

GKE and Cloud Run parse JSON logs on standard output into jsonPayload, so a log-based alerting policy can fire on the subscription failed event even when the process exits before its last scrape:

displayName: "beavers: subscription failed"
severity: CRITICAL
combiner: OR
conditions:
  - displayName: subscription failed was logged
    conditionMatchedLog:
      filter: jsonPayload.fields.message="subscription failed"
      labelExtractors:
        error: EXTRACT(jsonPayload.fields.error)
alertStrategy:
  notificationRateLimit:
    period: 300s
  autoClose: 1800s
notificationChannels:
  - projects/my-project/notificationChannels/9876543210

Log-based policies require notificationRateLimit, which limits notifications to one per period while matching entries keep arriving.

Backlog

Scrape kafka-exporter or the Pulsar broker with a PodMonitoring resource, or a Collector sidecar outside GKE, and alert on the metrics listed in Backlog with the same kind of PromQL policy.

Markdown · Source on GitHub