Examkin

Question

Difficulty: MediumIncident Management and Automated Alerting

An SRE team at a global logistics firm is redesigning their incident response and alerting strategy for an inventory routing service running on Google Kubernetes Engine (GKE) behind an External HTTP(S) Load Balancer. During a recent database slowdown, the load balancer health checks repeatedly failed because they executed complex SQL queries, causing the load balancer to mark all healthy application pods as offline and creating a cascading outage. Furthermore, static CPU alerts generated hundreds of false-positive notifications during benign batch jobs. Which architectural modification should the team implement to prevent cascading failures while ensuring alerting reflects true user impact?

  1. Configure lightweight health check endpoints that verify application process responsiveness without querying downstream databases, and transition from static metric threshold alerts to multi-window SLO burn-rate alerting policies.Answer
  2. B
    Configure deep synthetic health check endpoints that execute end-to-end database queries to guarantee backend connectivity before routing traffic, while keeping static metric threshold alerts on CPU usage.
  3. C
    Maintain static metric threshold alerts on container CPU utilization, and grant primitive Owner IAM roles to automated remediation service accounts to allow rapid cluster node scaling.
  4. D
    Configure Cloud Logging exclusion filters to drop 5xx server error logs during active operational incidents, while keeping deep database health check probes active on the load balancer.

Answer

Configure lightweight health check endpoints that verify application process responsiveness without querying downstream databases, and transition from static metric threshold alerts to multi-window SLO burn-rate alerting policies.
The solution requires decoupling backend instance readiness from downstream database health by serving shallow health check probes. This keeps healthy compute instances active in the load balancer pool even when the database is struggling. Furthermore, adopting multi-window SLO burn-rate alerts ensures that notifications are generated only when the rate of error budget consumption poses a genuine risk to service reliability, suppressing noise from routine CPU utilization spikes.

Step-by-Step Solution

1
Analyze load balancer health check probe behavior
Identified that deep health check probes querying downstream databases cause load balancers to take healthy application instances offline when database latency rises.
Health checks must assess the health of the specific instance serving traffic, not the availability of shared downstream dependencies.
2
Redesign health check endpoint logic
Replace deep database checks with lightweight 'shallow' health check endpoints that verify local container process readiness.
Shallow checks prevent cascading outages by isolating backend container health from database performance spikes.
3
Upgrade alerting strategy to align with Site Reliability Engineering best practices
Replace static CPU metric threshold alerts with SLO multi-window burn-rate alerting policies in Cloud Monitoring.
SLO burn-rate alerts reduce false positives from routine CPU spikes and trigger incident notifications only when error budgets are being consumed at a rate threatening the SLA.

Key Concept

Shallow Load Balancer Health Checks and Multi-Window SLO Burn-Rate Alerting
Rate this question

Topics

Incident Management and Automated Alerting

More questions from this topic