A financial services organization runs a mission-critical payment processing workload on Google Cloud. To improve operational reliability and incident response, the SRE team needs to implement an automated alerting and self-healing incident management architecture. The solution must fulfill two requirements:
1. Alert the SRE team based on consumed service level objective (SLO) error budget rates to catch both rapid budget exhaustion and slow, persistent errors without triggering false positives from brief transient spikes.
2. Automatically trigger downstream remediation workflows securely during severe incidents without violating least-privilege principles.
Which TWO actions should the SRE team take to achieve these objectives?
- Create Cloud Monitoring alerting policies utilizing multi-window multi-burn-rate conditions targeting the payment service SLO.Answer
- Configure Cloud Monitoring notification channels to publish alert events to a Cloud Pub/Sub topic that triggers a Cloud Function operating with fine-grained, minimal IAM roles.Answer
- CConfigure static metric threshold alerts evaluated over single 1-minute windows to trigger immediate incident escalations whenever latency spikes.
- DGrant the primitive Editor IAM role to the service account executing automated remediation scripts to guarantee access to all GCP infrastructure.
- EConfigure backend load balancer health checks to execute deep database queries on every health probe to proactively remove unhealthy instances.
Answer
The SRE team should implement multi-window multi-burn-rate alerting policies targeting the service SLO and route alert notifications to Cloud Pub/Sub to trigger a Cloud Function using fine-grained, minimal IAM roles.
The correct architecture uses multi-window multi-burn-rate alerting policies to monitor SLO error budget consumption accurately without alert fatigue, combined with Cloud Pub/Sub and Cloud Functions running under custom, least-privilege service accounts to execute automated self-healing workflows securely.
Step-by-Step Solution
Key Concept
Incident Management and Automated Alerting via SLO Burn Rates and Pub/Sub Remediation Workflows