Examkin

Question

Difficulty: HardIncident Management and Automated Alerting

A global retail organization runs an online checkout service hosted on Google Cloud. During peak promotional events, downstream service dependencies occasionally experience transient latency spikes that consume the service level objective (SLO) error budget. The SRE team needs to implement an incident management and automated alerting strategy that minimizes alert fatigue by notifying on-call engineers only when the error budget is burning at a rate that risks an SLO breach, while simultaneously initiating automated remediation to reroute non-critical traffic via a Cloud Run service. Which approach should the cloud architect recommend?

  1. A
    Configure an Application Load Balancer health check to directly query downstream database endpoints so that backend instances experiencing elevated latency are automatically drained from service.
  2. Configure Cloud Monitoring alerting policies using multi-window, multi-burn-rate conditions on the latency Service Level Indicator (SLI), sending notifications to an incident channel and publishing alert state changes to a Pub/Sub topic that triggers the Cloud Run remediation service.Answer
  3. C
    Configure static threshold alerts in Cloud Monitoring set to evaluate 99th percentile response latency over 1-minute alignment intervals, triggering immediate PagerDuty webhooks and Cloud Run remediation.
  4. D
    Configure Cloud Logging aggregate exclusion filters to drop latency logs during high-volume traffic events, and configure Compute Engine autoscaling based on CPU utilization to trigger Pub/Sub incident notifications.

Answer

Configure Cloud Monitoring alerting policies using multi-window, multi-burn-rate conditions on the latency Service Level Indicator (SLI), sending notifications to an incident channel and publishing alert state changes to a Pub/Sub topic that triggers the Cloud Run remediation service.
The solution using multi-window multi-burn-rate alert policies aligns with Google Cloud SRE best practices. It measures how rapidly the SLO error budget is being consumed over both short (e.g., 5 minutes) and long (e.g., 1 hour) lookback windows. This prevents alert fatigue from brief non-critical latency bursts while ensuring timely notification for true incidents. Routing notifications through Pub/Sub provides a reliable event-driven mechanism to trigger Cloud Run for automated incident mitigation.

Step-by-Step Solution

1
Analyze alerting criteria requirements
Identify that avoiding alert fatigue while guarding SLO budgets requires burn-rate alerting rather than static threshold monitoring.
Burn-rate alerting measures consumption rate of the error budget over multiple time windows (short and long), filtering out brief transient spikes while catching sustained degradations.
2
Design automated notification and remediation pipeline
Select Pub/Sub as the messaging notification channel from Cloud Monitoring alerting policy to decouple alert triggers from execution.
Cloud Monitoring alerting policies can natively publish incidents to Cloud Pub/Sub, which asynchronously invokes Cloud Run microservices for automated traffic rerouting.
3
Evaluate downstream health check design
Ensure health checks test local instance health rather than deep downstream dependencies.
Testing deep dependencies in health checks causes backend pools to drain unnecessarily during downstream dependency issues, worsening outages.

Key Concept

Multi-Window Multi-Burn-Rate Alerting & Automated Incident Response Integration
Rate this question

Topics

Incident Management and Automated Alerting

More questions from this topic