Google Cloud
Google Cloud Managed Service for Prometheus stores the metrics in Cloud Monitoring, and Cloud Monitoring alerting policies evaluate the shared PromQL rules against them. Notifications go to Cloud Monitoring notification channels, such as email, Slack, PagerDuty, SMS, webhooks, or Pub/Sub, so no Alertmanager is needed.
Collect on GKE
Managed collection is enabled by default on recent GKE Autopilot and Standard
clusters. Expose the ports and probes as in the
Kubernetes Deployment, then select the pods with a
PodMonitoring resource:
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
name: beavers
namespace: orders
spec:
selector:
matchLabels:
app: orders-consumer
endpoints:
- port: metrics
interval: 30s
The job label of the collected series is the PodMonitoring name,
beavers, which the BeaversMetricsDown rule expects.
Collect on Cloud Run
A Cloud Run service that consumes from a broker must keep running between requests: use instance-based billing and at least one minimum instance. Run the OpenTelemetry Collector as a sidecar container that scrapes the exporter and writes to Managed Service for Prometheus:
receivers:
prometheus:
config:
scrape_configs:
- job_name: beavers
scrape_interval: 30s
static_configs:
- targets: ["localhost:9000"]
processors:
resourcedetection:
detectors: [env, gcp]
batch: {}
exporters:
googlemanagedprometheus: {}
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [resourcedetection, batch]
exporters: [googlemanagedprometheus]
The googlemanagedprometheus exporter is part of the otelcol-contrib
distribution and of Google's built OpenTelemetry Collector image. The service
account needs the roles/monitoring.metricWriter role. Add a liveness probe on
/livez to the application container; Cloud Run then replaces an instance
whose subscription failed.
Alerting policies
Create one alerting policy per rule. Each condition holds the PromQL expression of a shared rule unchanged:
# dead-letters.yaml
displayName: "beavers: dead letters"
severity: WARNING
combiner: OR
conditions:
- displayName: Deliveries were dead-lettered
conditionPrometheusQueryLanguage:
query: sum by (subscription) (increase(beavers_delivery_failures_total{action="dead_letter"}[10m])) > 0
duration: 0s
evaluationInterval: 60s
disableMetricValidation: true
notificationChannels:
- projects/my-project/notificationChannels/1234567890
documentation:
mimeType: text/markdown
content: |
Inspect the dead-letter topic and redrive the deliveries once the cause is fixed.
# stalled.yaml
displayName: "beavers: stalled subscription"
severity: CRITICAL
combiner: OR
conditions:
- displayName: Unfinished deliveries without acknowledgements
conditionPrometheusQueryLanguage:
query: |
max by (subscription) (beavers_deliveries_in_flight) > 0
and sum by (subscription) (increase(beavers_deliveries_acknowledged_total[15m])) == 0
duration: 300s
evaluationInterval: 60s
disableMetricValidation: true
notificationChannels:
- projects/my-project/notificationChannels/9876543210
gcloud monitoring policies create --policy-from-file=dead-letters.yaml
gcloud monitoring policies create --policy-from-file=stalled.yaml
The rule's for becomes duration, and the policy's severity takes the
value of the rule's severity label. A PromQL policy holds exactly one
condition. disableMetricValidation lets the policy be created before the
application has written its first metrics; without it, creation fails for a
metric that does not exist yet. A PromQL incident closes on its own once the
query stops matching, so these policies need no autoClose. Give critical policies the on-call channel and
warning policies the team's channel. Each time series that matches the
query opens its own incident, so a policy notifies once per failing
subscription.
On GKE, restarts are available as a Cloud Monitoring system metric, which PromQL reads under its converted name:
increase(kubernetes_io:container_restart_count{namespace_name="orders", container_name="app"}[15m]) > 0
Log-based alert
GKE and Cloud Run parse JSON logs on standard
output into jsonPayload, so a log-based alerting policy can fire on the
subscription failed event even when the process exits before its last
scrape:
displayName: "beavers: subscription failed"
severity: CRITICAL
combiner: OR
conditions:
- displayName: subscription failed was logged
conditionMatchedLog:
filter: jsonPayload.fields.message="subscription failed"
labelExtractors:
error: EXTRACT(jsonPayload.fields.error)
alertStrategy:
notificationRateLimit:
period: 300s
autoClose: 1800s
notificationChannels:
- projects/my-project/notificationChannels/9876543210
Log-based policies require notificationRateLimit, which limits notifications
to one per period while matching entries keep arriving.
Backlog
Scrape kafka-exporter or the Pulsar broker with a PodMonitoring resource, or
a Collector sidecar outside GKE, and alert on the metrics listed in
Backlog with the same kind of PromQL policy.