AWS
Choose by where the application runs:
- Amazon EKS: collect metrics into Amazon Managed Service for Prometheus and evaluate the shared PromQL rules there.
- Amazon ECS or EC2: send metrics to CloudWatch with the AWS Distro for OpenTelemetry (ADOT) Collector and alert with CloudWatch alarms.
Both paths deliver alerts through an Amazon SNS topic. Subscribe an email address, an HTTPS endpoint, or an incident tool that accepts SNS, such as PagerDuty or Opsgenie. CloudWatch alarms can also reach Slack or Microsoft Teams through Amazon Q Developer in chat applications.
Amazon Managed Service for Prometheus
Collect
Create a workspace and an Amazon Managed Service for Prometheus collector, the agentless scraper, for the EKS cluster. Give the scraper a configuration that discovers the application's pods:
global:
scrape_interval: 30s
scrape_configs:
- job_name: beavers
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
regex: orders-consumer
action: keep
- source_labels: [__meta_kubernetes_pod_container_port_name]
regex: metrics
action: keep
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
The scraper needs the cluster role and access entry described in the Amazon Managed Service for Prometheus documentation. A cluster that already runs ADOT or Prometheus in agent mode can remote-write to the workspace instead. Expose the ports and probes as in the Kubernetes Deployment.
Alert rules
Save the shared rules as beavers-rules.yaml, add the
restart rule if the cluster runs
kube-state-metrics, and upload them as a rule groups namespace:
aws amp create-rule-groups-namespace \
--workspace-id ws-0123456789abcdef \
--name beavers \
--data fileb://beavers-rules.yaml
Use aws amp put-rule-groups-namespace with the same arguments to update it.
Route alerts
The workspace's alert manager accepts the Alertmanager configuration format,
wrapped in alertmanager_config, and delivers only to SNS. Route by severity
to two topics:
alertmanager_config: |
route:
receiver: team
group_by: [alertname, subscription]
routes:
- receiver: on-call
matchers: ['severity="critical"']
receivers:
- name: team
sns_configs:
- topic_arn: arn:aws:sns:ap-northeast-1:123456789012:beavers-warning
sigv4:
region: ap-northeast-1
subject: "{{ .CommonLabels.alertname }}"
message: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
- name: on-call
sns_configs:
- topic_arn: arn:aws:sns:ap-northeast-1:123456789012:beavers-critical
sigv4:
region: ap-northeast-1
subject: "{{ .CommonLabels.alertname }}"
message: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
aws amp create-alert-manager-definition \
--workspace-id ws-0123456789abcdef \
--data fileb://alertmanager.yaml
Each topic's access policy must let the service publish:
{
"Effect": "Allow",
"Principal": { "Service": "aps.amazonaws.com" },
"Action": ["sns:Publish", "sns:GetTopicAttributes"],
"Resource": "arn:aws:sns:ap-northeast-1:123456789012:beavers-critical",
"Condition": {
"ArnEquals": { "aws:SourceArn": "arn:aws:aps:ap-northeast-1:123456789012:workspace/ws-0123456789abcdef" },
"StringEquals": { "AWS:SourceAccount": "123456789012" }
}
}
CloudWatch
Collect
Run the ADOT Collector as a sidecar container in the ECS task, or as a service on the EC2 instance. It scrapes the exporter and writes CloudWatch metrics through the embedded metric format:
receivers:
prometheus:
config:
scrape_configs:
- job_name: beavers
scrape_interval: 30s
static_configs:
- targets: ["localhost:9000"]
processors:
batch: {}
exporters:
awsemf:
namespace: Beavers
log_group_name: /beavers/metrics
dimension_rollup_option: NoDimensionRollup
metric_declarations:
- dimensions: [[subscription]]
metric_name_selectors:
- "^beavers_deliveries_.*"
- "^beavers_receive_errors_total$"
- dimensions: [[subscription, action], [subscription, failure]]
metric_name_selectors:
- "^beavers_delivery_failures_total$"
- dimensions: [[subscription, sink]]
metric_name_selectors:
- "^beavers_publish_failures_total$"
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [awsemf]
awsemf converts cumulative counters to the increase since the previous
scrape, so the Sum statistic of a counter over an alarm period is the number
of events in that period. Only metrics matched by metric_declarations become
CloudWatch metrics, and each dimension set is published separately: declaring
[subscription, action] lets an alarm count dead letters of every failure kind
together. The task role needs logs:PutLogEvents, logs:CreateLogStream, and
logs:CreateLogGroup on the log group.
awsemf drops the first value of each counter after the collector or the
application restarts, because it has no previous value to subtract.
The handler latency histogram is left out because awsemf does not convert
the cumulative bucket counts of a scraped Prometheus histogram to deltas, so
CloudWatch statistics on it are not meaningful; time the handler with a
CloudWatch metric of its own if latency needs an alarm.
Alarms
This CloudFormation template creates the alarms for one subscription. Repeat the resources, or generate them, for each subscription:
Parameters:
Subscription:
Type: String
CriticalTopicArn:
Type: String
WarningTopicArn:
Type: String
Resources:
StoppedOnFailure:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub beavers-${Subscription}-stopped
Namespace: Beavers
MetricName: beavers_delivery_failures_total
Dimensions:
- { Name: subscription, Value: !Ref Subscription }
- { Name: action, Value: stop }
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions: [!Ref CriticalTopicArn]
DeadLetterPublishFailing:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub beavers-${Subscription}-dead-letter-publish
Namespace: Beavers
MetricName: beavers_publish_failures_total
Dimensions:
- { Name: subscription, Value: !Ref Subscription }
- { Name: sink, Value: dead_letter }
Statistic: Sum
Period: 300
EvaluationPeriods: 2
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions: [!Ref CriticalTopicArn]
OKActions: [!Ref CriticalTopicArn]
OutputPublishFailing:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub beavers-${Subscription}-output-publish
Namespace: Beavers
MetricName: beavers_publish_failures_total
Dimensions:
- { Name: subscription, Value: !Ref Subscription }
- { Name: sink, Value: output }
Statistic: Sum
Period: 300
EvaluationPeriods: 2
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions: [!Ref CriticalTopicArn]
OKActions: [!Ref CriticalTopicArn]
ReceiveFailing:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub beavers-${Subscription}-receive
Namespace: Beavers
MetricName: beavers_receive_errors_total
Dimensions:
- { Name: subscription, Value: !Ref Subscription }
Statistic: Sum
Period: 300
EvaluationPeriods: 2
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions: [!Ref CriticalTopicArn]
OKActions: [!Ref CriticalTopicArn]
Stalled:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub beavers-${Subscription}-stalled
Metrics:
- Id: inflight
ReturnData: false
MetricStat:
Metric:
Namespace: Beavers
MetricName: beavers_deliveries_in_flight
Dimensions:
- { Name: subscription, Value: !Ref Subscription }
Period: 900
Stat: Minimum
- Id: acked
ReturnData: false
MetricStat:
Metric:
Namespace: Beavers
MetricName: beavers_deliveries_acknowledged_total
Dimensions:
- { Name: subscription, Value: !Ref Subscription }
Period: 900
Stat: Sum
- Id: stalled
Expression: IF(inflight > 0 AND acked == 0, 1, 0)
Label: stalled
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions: [!Ref CriticalTopicArn]
DeadLetters:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub beavers-${Subscription}-dead-letters
Namespace: Beavers
MetricName: beavers_delivery_failures_total
Dimensions:
- { Name: subscription, Value: !Ref Subscription }
- { Name: action, Value: dead_letter }
Statistic: Sum
Period: 600
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions: [!Ref WarningTopicArn]
DecodeFailureRatio:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub beavers-${Subscription}-decode-ratio
Metrics:
- Id: decode
ReturnData: false
MetricStat:
Metric:
Namespace: Beavers
MetricName: beavers_delivery_failures_total
Dimensions:
- { Name: subscription, Value: !Ref Subscription }
- { Name: failure, Value: decode }
Period: 300
Stat: Sum
- Id: received
ReturnData: false
MetricStat:
Metric:
Namespace: Beavers
MetricName: beavers_deliveries_received_total
Dimensions:
- { Name: subscription, Value: !Ref Subscription }
Period: 300
Stat: Sum
- Id: ratio
Expression: FILL(decode, 0) / received
Label: decode failure ratio
EvaluationPeriods: 2
Threshold: 0.01
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions: [!Ref WarningTopicArn]
TreatMissingData: notBreaching keeps idle subscriptions quiet, but it also
keeps the alarms quiet when the collector stops sending metrics. Cover that
with the task and log alerts below.
Task and log alerts
A failed subscription makes the process exit, usually before its last metrics are scraped. ECS reports the stopped task to Amazon EventBridge; route it to the critical topic:
{
"source": ["aws.ecs"],
"detail-type": ["ECS Task State Change"],
"detail": {
"clusterArn": ["arn:aws:ecs:ap-northeast-1:123456789012:cluster/orders"],
"group": ["service:orders-consumer"],
"lastStatus": ["STOPPED"],
"stopCode": ["EssentialContainerExited"]
}
}
With the awslogs log driver and JSON logs, a
metric filter on the application's log group counts subscription failed
events. Alarm on Sum > 0 of the resulting metric, as for the other counters:
SubscriptionFailedFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName: /ecs/orders-consumer
FilterPattern: '{ $.fields.message = "subscription failed" }'
MetricTransformations:
- MetricNamespace: Beavers
MetricName: subscription_failed
MetricValue: "1"
DefaultValue: 0
Health checks
ECS has no HTTP probe of its own. Behind an Application Load Balancer, which
an HTTP source needs, set the target group health check
path to /readyz so that a draining or retrying task stops receiving
requests. A container health check that calls /livez needs an HTTP client in
the image; a minimal image without one can rely on the process exiting after a
subscription fails, which ECS replaces according to the service's desired
count.
Backlog
Amazon MSK publishes MaxOffsetLag, SumOffsetLag, and EstimatedMaxTimeLag
per consumer group and topic, and Amazon SQS publishes
ApproximateAgeOfOldestMessage. Alarm on them as described in
Backlog; with Amazon Managed Service for Prometheus, a
self-managed Kafka cluster can be scraped with kafka-exporter instead.