Examkin

Soru

Zorluk: OrtaIncident Management and Automated Alerting

An enterprise SaaS platform deployed across Google Kubernetes Engine (GKE) clusters uses an Application Load Balancer to route user traffic. During a scheduled database maintenance window, the database experienced transient latency spikes, causing the Application Load Balancer to mark all backend pods as unhealthy and leading to a complete service outage. Subsequent analysis showed that the load balancer health checks were executing deep SQL queries against the database, and static CPU utilization alerts failed to warn the Site Reliability Engineering (SRE) team before the error budget was completely consumed. Which TWO architectural and alerting strategies should the SRE team implement to prevent cascading service failures and establish effective automated incident management?

  1. Reconfigure the load balancer health check to query a shallow endpoint on the application that returns HTTP 200 OK without evaluating deep database dependencies.Cevap
  2. Implement multi-window, multi-burn-rate alerting in Cloud Monitoring based on HTTP 5xx error rates and latency SLIs to trigger incidents before error budgets are exhausted.Cevap
  3. C
    Configure load balancer health checks to execute deeper synthetic database queries while increasing the health check timeout to 300 seconds.
  4. D
    Set up a static single-window threshold alert on backend instance CPU utilization set to 95% to trigger automated node pool redeployments.
  5. E
    Apply Cloud Logging exclusion filters on HTTP 5xx error logs during maintenance windows to prevent alerting policies from creating incident tickets.

Cevap

The SRE team should reconfigure the load balancer health checks to use a shallow application endpoint that does not query downstream databases, and establish multi-window error budget burn-rate alerts based on user-facing latency and HTTP 5xx SLIs in Cloud Monitoring.
To prevent cascading failures during database latency events, load balancer health checks should hit a shallow endpoint that confirms the container process is running without performing deep database operations. Additionally, configuring multi-window error budget burn-rate alerts in Cloud Monitoring ensures SRE teams receive timely alerts based on actual SLI impact (latency and 5xx errors) before the error budget is depleted.

Adım Adım Çözüm

1
Analyze the cause of the cascading health check failure.
Identified that deep health check probes querying the database caused healthy web pods to be marked dead when the database experienced transient latency.
Health checks for load balancers must only verify local application process health (shallow health check) to avoid removing valid backends during upstream dependency degradation.
2
Evaluate the alerting requirement for proactive incident response.
Determined that static CPU thresholds failed to alert SREs about user-impacting outages caused by database latency.
SRE best practices recommend multi-window, multi-burn-rate alerting on Service Level Indicators (SLIs) like request latency and error rates to detect rapid error budget depletion.

Anahtar Kavram

Shallow Health Checks and Multi-Window Error Budget Burn-Rate Alerting
Bu soruyu puanla

Konular

Incident Management and Automated Alerting

Bu konudan daha fazla soru