Kubernetes
This guide runs Prometheus and Alertmanager in the cluster with the Prometheus Operator. On EKS, GKE, or AKS, the cloud's managed Prometheus avoids operating them yourself; see AWS, Google Cloud, or Azure. The Deployment below applies to every cluster.
Deployment
Expose the metrics and health ports, probe the health server, and give draining enough time before Kubernetes kills the container:
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-consumer
namespace: orders
spec:
replicas: 2
selector:
matchLabels:
app: orders-consumer
template:
metadata:
labels:
app: orders-consumer
spec:
terminationGracePeriodSeconds: 60
containers:
- name: app
image: registry.example.com/orders-consumer:1.0.0
ports:
- name: health
containerPort: 8080
- name: metrics
containerPort: 9000
livenessProbe:
httpGet: { path: /livez, port: health }
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet: { path: /readyz, port: health }
periodSeconds: 5
failureThreshold: 2
Readiness removes an instance that serves an HTTP source from its Service while it drains or retries. For broker sources, readiness only affects rollouts: a rolling update waits until new pods are ready. Liveness restarts a container whose subscription failed and that has not exited yet.
terminationGracePeriodSeconds must cover the drain of in-flight deliveries,
including publish retries, after SIGTERM. A container killed before it
finishes draining leaves unacknowledged deliveries that the broker redelivers.
Install the monitoring stack
The kube-prometheus-stack Helm chart installs the operator, Prometheus,
Alertmanager, kube-state-metrics, and default Kubernetes alerts:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
--namespace monitoring --create-namespace \
--values monitoring-values.yaml
By default the chart's Prometheus selects only PodMonitor and
PrometheusRule resources labelled release: monitoring, the Helm release
name. The examples below carry that label.
Scrape the metrics
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: beavers
namespace: orders
labels:
release: monitoring
spec:
selector:
matchLabels:
app: orders-consumer
podMetricsEndpoints:
- port: metrics
interval: 30s
The operator sets the job label of a PodMonitor target to
<namespace>/<name>, here orders/beavers. Use that value in the
BeaversMetricsDown rule.
Alert rules
Wrap the shared rules in a PrometheusRule and add a
restart alert for the application, which also fires when the failed
subscription's last metrics were never scraped:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: beavers
namespace: orders
labels:
release: monitoring
spec:
groups:
- name: beavers
rules:
# The shared rules from README.md#alert-rules, with
# up{job="orders/beavers"} in BeaversMetricsDown.
- alert: BeaversRestarted
expr: increase(kube_pod_container_status_restarts_total{namespace="orders", container="app"}[15m]) > 0
labels:
severity: critical
annotations:
summary: "{{ $labels.pod }} restarted; check its previous logs for 'subscription failed'"
kube-prometheus-stack already alerts on crash loops with
KubePodCrashLooping, but only after repeated restarts. BeaversRestarted
fires on the first one, because a subscription that stops on a poison record
restarts and fails again on the same record until someone dead-letters or
skips it.
Route alerts
Configure Alertmanager in monitoring-values.yaml. This example sends every
alert to a Slack channel and pages on critical alerts:
alertmanager:
config:
route:
receiver: team-slack
group_by: [alertname, subscription]
routes:
- receiver: "null"
matchers: ['alertname="Watchdog"']
- receiver: on-call
matchers: ['severity="critical"']
continue: true
- receiver: team-slack
receivers:
- name: "null"
- name: team-slack
slack_configs:
- api_url_file: /etc/alertmanager/secrets/alertmanager-slack/url
channel: "#orders-alerts"
send_resolved: true
title: "{{ .CommonLabels.alertname }}"
text: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
- name: on-call
pagerduty_configs:
- routing_key_file: /etc/alertmanager/secrets/alertmanager-pagerduty/key
alertmanagerSpec:
secrets: [alertmanager-slack, alertmanager-pagerduty]
Store the Slack webhook URL and the PagerDuty integration key in the
alertmanager-slack and alertmanager-pagerduty Secrets of the monitoring
namespace. Grouping by alertname and subscription sends one notification
per failing subscription, however many deliveries failed.
Logs
Kubernetes keeps only the logs of the current and the previous container. Ship
the JSON logs to a log store, such as Loki or the
cloud's log service, to read the subscription failed event after a restart.
With Loki, a ruler rule alerts on the event directly:
groups:
- name: beavers-logs
rules:
- alert: BeaversSubscriptionFailed
expr: |
sum by (namespace, pod) (
count_over_time({namespace="orders"} | json | fields_message="subscription failed" [10m])
) > 0
labels:
severity: critical
annotations:
summary: "{{ $labels.pod }} logged 'subscription failed'"