Examkin

Question

Difficulty: MediumIncident Management and Automated Alerting

An enterprise application hosted on Compute Engine instances behind an External Application Load Balancer experiences cascading service outages whenever downstream database queries undergo brief, transient latency spikes. Diagnostic logs indicate that load balancer health checks are configured to execute deep database queries directly, causing healthy web instances to be misidentified as unhealthy and removed from serving pools. Additionally, the operations team suffers from alert fatigue caused by fixed threshold alerts firing during minor latency spikes. Which architecture modification should you recommend to improve operational reliability and incident management?

  1. Reconfigure load balancer health checks to query a lightweight local HTTP endpoint that validates local process status, and establish Cloud Monitoring alerting policies based on SLO error budget burn rates.Answer
  2. B
    Configure load balancer health checks to execute deeper synthetic database transactions at higher frequencies, and assign Project Owner primitive IAM roles to monitoring service accounts to restart unresponding VMs automatically.
  3. C
    Maintain existing health check paths but lower the threshold duration for static latency alert policies to 10 seconds to trigger immediate incident escalation webhooks.
  4. D
    Replace the load balancer health check with VPC Service Controls perimeter audit logging metrics to trigger automated incident response functions upon data access delays.

Answer

Reconfigure load balancer health checks to query a dedicated, lightweight local health endpoint on the web servers, and implement Cloud Monitoring alerting policies using SLO error budget burn rates.
The correct strategy decouples load balancer health probing from downstream database dependencies by using a lightweight local endpoint, preventing cascading server drops. Additionally, leveraging error budget burn-rate alerts in Cloud Monitoring alerts operations teams based on true impact to customer SLOs rather than short-lived performance noise.

Step-by-Step Solution

1
Analyze load balancer health check behavior
Identify that deep health checks probing downstream databases cause healthy web instances to be marked unhealthy during transient database slowness, leading to cascading failures.
Load balancer health checks must only measure whether the immediate instance can accept traffic.
2
Decouple health probing logic
Configure health checks to hit a lightweight local HTTP endpoint (e.g., /healthz) that tests local web process viability without executing deep database calls.
Isolating local health checks prevents transient database latency from taking down the web serving tier.
3
Implement effective automated alerting for incident management
Use Cloud Monitoring to create burn-rate alerting policies on Service Level Objectives (SLOs).
Burn-rate alerting evaluates the rate at which the error budget is consumed over multiple time windows, preventing alert fatigue from transient spikes while alerting promptly on significant outages.

Key Concept

Incident Management and Automated Alerting
Rate this question

Topics

Incident Management and Automated Alerting

More questions from this topic