Examkin

Question

Difficulty: HardIncident Management and Automated Alerting

An enterprise media organization operates a high-throughput video streaming platform hosted on Google Kubernetes Engine (GKE) behind a Global External Application Load Balancer. During peak live events, transient database latency causes load balancer health checks to fail across healthy GKE pods, resulting in cascading backend shutdowns and incident alert floods. Additionally, operations teams are overwhelmed by false-positive pages for short latency spikes while failing to detect slow, sustained error budget depletion. You need to re-architect the automated alerting and incident management strategy to improve system reliability and incident escalation. Which TWO architectural actions should you take? (Select TWO.)

  1. Reconfigure the load balancer health check endpoints to target a shallow application status check (/healthz) that only verifies local pod process health.Answer
  2. Configure Cloud Monitoring alerting policies using multi-window, multi-burn-rate conditions based on Service Level Objective (SLO) error budget consumption.Answer
  3. C
    Configure load balancer health checks to directly execute deep database validation queries every 5 seconds to drain instances immediately when database response times exceed SLAs.
  4. D
    Establish static metric threshold alerts set to trigger paging notifications whenever HTTP 5xx error rates exceed 2% over a 1-minute evaluation window.
  5. E
    Grant the primitive Owner role (roles/owner) to the automated incident response service account across the GCP project hierarchy to eliminate permission blocks during automated remediation.

Answer

The correct actions are to implement shallow health check endpoints for load balancing to prevent cascading failures, and to configure multi-window, multi-burn-rate alerting policies based on SLO error budget consumption in Cloud Monitoring.
The correct strategy combines shallow health checks with SLO burn-rate alerting. Shallow load balancer health checks (/healthz) verify local instance functionality without cascading failures when downstream databases slow down. Multi-window burn-rate alerts monitor the consumption rate of error budgets over long and short windows, dramatically decreasing false positives while prioritizing real outages.

Step-by-Step Solution

1
Analyze load balancer health check failure root cause
Identified that health checks hitting downstream database dependencies cause all backend pods to be marked unhealthy during transient database latency.
Health checks must isolate local container/process health from external service dependencies.
2
Redesign health check endpoint structure
Decoupled health checks by pointing them to a shallow endpoint (/healthz) that returns HTTP 200 based strictly on local application process readiness.
This prevents load balancing infrastructure from removing functional application pods when downstream services experience transient degradation.
3
Evaluate and replace alerting strategy
Replaced static metric threshold alerts with multi-window burn-rate alerting tied directly to SLO error budget consumption rates.
Burn-rate alerting accurately measures user-impacting trends across multiple time windows, avoiding false alarms from transient spikes while alerting promptly on critical outages.

Key Concept

Incident Management, Shallow Health Checks, and SLO Burn-Rate Alerting
Rate this question

Topics

Incident Management and Automated Alerting

More questions from this topic