Examkin

Soru

Zorluk: ZorIncident Management and Automated Alerting

An enterprise e-commerce platform hosted on Google Cloud experiences high traffic volatility during promotional events. Transient micro-spikes in compute resource usage currently trigger dozens of high-severity alerts, causing severe alert fatigue among on-call engineers. Additionally, during a recent incident, the Google Cloud HTTP Load Balancer marked all backend instances as unhealthy because its health checks were configured to execute deep SQL queries against a database that was experiencing temporary connection pool exhaustion, resulting in a cascading platform outage. The Lead Reliability Engineer must redesign the operational alerting and health checking architecture to reflect true customer impact and prevent cascading failures. Which combination of architectural practices should be implemented?

  1. Configure Cloud Monitoring alert policies using multi-window burn-rate conditions based on Service Level Objectives (SLOs), and update the load balancer health checks to target a shallow endpoint that only validates local process status.Cevap
  2. B
    Configure Cloud Monitoring static threshold alerts on CPU and memory utilization with zero duration, and update the load balancer health checks to target a deep database endpoint with an increased response timeout.
  3. C
    Configure Cloud Monitoring alerts based on single-window immediate threshold breaches, and maintain deep health checks on the load balancer to guarantee that backend instances cannot accept traffic unless the database is healthy.
  4. D
    Define target Service Level Indicators (SLIs) as fixed metric alerts in Cloud Monitoring, and apply aggregate Cloud Logging exclusion filters to suppress high-severity application error logs during peak events.

Cevap

Configure Cloud Monitoring alert policies using multi-window burn-rate conditions based on Service Level Objectives (SLOs), and update the load balancer health checks to target a shallow endpoint that only validates local process status.
The combination of multi-window burn-rate alerts and shallow load balancer health checks directly addresses both issues. Multi-window burn-rate alerting ensures that alerts trigger only when the error budget is consumed at a rate that threatens the SLO over both short and long lookback windows, eliminating false alarms from micro-spikes. Shallow health checks verify that the specific application instance process is responsive without testing backend database endpoints, preventing cascading failures across the compute pool when database latency occurs.

Adım Adım Çözüm

1
Evaluate alerting strategy to eliminate alert fatigue from transient micro-spikes.
Transitioning from static metric thresholds to multi-window burn-rate alerting evaluates the rate of Error Budget consumption over multiple time windows (e.g., 1-hour and 6-hour windows), ensuring alerts fire only when sustained error rates threaten the overall Service Level Objective.
Sustained burn rates correlate directly with real user impact, preventing false alarms from brief operational spikes.
2
Evaluate load balancer health check configuration to prevent cascading service failures.
Replacing deep health checks (which query downstream databases or external services) with shallow health checks (which query a dedicated `/healthz` endpoint verifying only local process availability) isolates instance lifecycle management from dependency health.
If a downstream database becomes temporarily overloaded, removing healthy compute instances from load balancing exacerbates the outage instead of mitigating it.

Anahtar Kavram

Site Reliability Engineering (SRE) best practices for SLO-based multi-window burn-rate alerting and shallow load balancer health checks.
Bu soruyu puanla

Konular

Incident Management and Automated Alerting

Bu konudan daha fazla soru