Examkin

Question

Difficulty: MediumIncident Management and Automated Alerting

A Site Reliability Engineering (SRE) team manages a real-time smart grid IoT telemetry ingestion service hosted on Google Cloud using Cloud Run behind an External HTTP(S) Load Balancer, backed by Cloud Bigtable. During recent peak load events, brief traffic micro-bursts caused transient CPU utilization spikes that triggered frequent, unactionable alerts. Furthermore, during a minor database latency degradation, backend instances were prematurely marked unhealthy by the load balancer, precipitating a cascading outage across the entire service. Which TWO architectural modifications should the SRE team implement to eliminate alert fatigue and prevent cascading service failures?

  1. Configure alerting policies using multi-window, multi-burn-rate conditions tied to the service error budget instead of static CPU utilization threshold alerts.Answer
  2. Modify the load balancer health check configuration to query a dedicated lightweight local endpoint that reports instance liveness without validating downstream database connectivity.Answer
  3. C
    Update the load balancer health check configuration to execute a read/write test query against Cloud Bigtable to guarantee true end-to-end service availability.
  4. D
    Reduce the alignment period and trigger duration of static metric threshold alerts to 1 minute so SREs are immediately notified of transient CPU spikes.
  5. E
    Implement a project-wide Cloud Logging exclusion filter for all application logs below ERROR severity to suppress low-priority log volume.

Answer

The SRE team should implement multi-window, multi-burn-rate alerting policies based on service SLO error budgets and configure load balancer health checks to use a lightweight local endpoint that does not query downstream dependencies.
Configuring multi-window burn rate alerting ensures SREs are notified based on sustained SLO consumption rather than transient CPU bursts, resolving alert fatigue. Additionally, separating load balancer health checks from downstream database checks (shallow health checks) prevents database latency from triggering cascading instance removals by the load balancer.

Step-by-Step Solution

1
Analyze the root cause of alert fatigue during transient traffic bursts.
Identified that static metric alerts on CPU utilization trigger on short, non-impacting spikes.
Static metric thresholds fail to distinguish between short benign load bursts and sustained user-impacting outages. Multi-window burn rate alerts calculate error budget consumption rate to trigger alerts only when an SLO breach is imminent.
2
Analyze the root cause of cascading failures during database latency.
Identified that health checks were evaluating downstream database responsiveness.
Health checks must assess local container/instance liveness (shallow health check). If health checks query downstream components like databases, a transient database delay causes the load balancer to mark all app instances unhealthy, triggering a full service blackout.

Key Concept

SLO Burn-Rate Alerting & Shallow Health Check Patterns
Estimated Time:2m 0s
Rate this question

Topics

Incident Management and Automated Alerting

More questions from this topic