Your Site Reliability Engineering (SRE) team is designing an end-to-end automated incident response and escalation workflow on Google Cloud to handle severe latency spikes in an enterprise web application. Place the operational steps in the correct chronological sequence from initial fault detection to incident resolution.
- 1Cloud Monitoring aggregates Service Level Indicator (SLI) metrics and evaluates that the latency SLO burn rate exceeds the defined critical alert threshold.
- 2The Cloud Monitoring alerting policy fires and dispatches an incident payload to a configured Pub/Sub notification channel.
- 3A subscriber Cloud Function ingests the Pub/Sub alert notification and executes automated remediation by updating traffic routing and scaling backend capacity.
- 4Cloud Monitoring observes that latency SLI values have returned within acceptable SLO limits for the required duration window and automatically resolves the incident.
Answer
The correct sequence starts with Cloud Monitoring evaluating the SLI burn rate threshold, followed by publishing an alert payload to Pub/Sub, triggering a Cloud Function to execute automated remediation, and concluding with Cloud Monitoring verifying metric recovery to auto-resolve the incident.
The workflow follows a standard event-driven incident lifecycle: Detection (evaluating SLI burn rate) -> Notification (publishing to Pub/Sub) -> Action (executing programmatic remediation via Cloud Function) -> Verification & Resolution (monitoring SLI stabilization and closing the alert).
Step-by-Step Solution
Key Concept
Automated Incident Remediation and Alert Escalation Workflow