Question

Difficulty: MediumIncident Management and Automated Alerting

An enterprise Cloud Architecture team is establishing an end-to-end automated incident detection and self-healing remediation workflow on Google Cloud for a microservice encountering resource exhaustion. Arrange the operational steps in the correct chronological order from initial alerting setup through incident resolution.

  1. 1Define a Cloud Monitoring alerting policy with metric threshold criteria and configure a Pub/Sub topic as the notification channel target.
  2. 2Cloud Monitoring detects a metric threshold breach and publishes a structured incident alert payload to the designated Pub/Sub topic.
  3. 3A serverless Cloud Function triggered by the Pub/Sub topic authenticates using a least-privilege service account to parse incident details.
  4. 4The Cloud Function executes Google Cloud API requests to gracefully restart degraded service instances and clear stuck resource locks.
  5. 5Cloud Monitoring observes metric recovery below threshold conditions and automatically clears the active incident status.

Answer

The correct sequence follows the full lifecycle of automated incident detection and remediation: configuring alerting policies with Pub/Sub notification targets, publishing alert payloads upon metric breach, triggering event-driven remediation functions, executing API remediation calls to restore workload health, and auto-closing the incident in Cloud Monitoring upon metric recovery.
The proper operational order follows a logical incident lifecycle: proactive monitoring configuration, metric breach detection with event publishing via Pub/Sub, serverless event ingestion and payload parsing, execution of automated API remediation scripts, and final auto-closure of the incident state by Cloud Monitoring upon verified service recovery.

Step-by-Step Solution

1
Configure Cloud Monitoring alert policies and notification targets
Sets baseline alert thresholds and establishes the Pub/Sub integration pathway.
Alerting policies and messaging channels must be provisioned prior to handling live operational incidents.
2
Emit incident event payload to Pub/Sub
Publishes structured incident metadata when performance thresholds are exceeded.
Cloud Monitoring decouples detection from remediation by delivering alerts via Pub/Sub messaging.
3
Ingest payload via event-driven Cloud Function
Parses resource identifiers and operational context securely using service account IAM roles.
Remediation functions require parsed message attributes to identify specific failing resources.
4
Execute programmatic remediation API calls
Restores instance health through automated lifecycle management commands.
Direct action against Cloud APIs mitigates operational outages without human intervention.
5
Validate metric recovery and resolve incident state
Updates incident telemetry and auto-closes open incident tickets in Monitoring.
Cloud Monitoring automatically closes incidents once consecutive healthy evaluation periods pass.

Key Concept

Automated Incident Lifecycle Management and Event-Driven Remediation in Google Cloud
Rate this question