Examkin

Question

Difficulty: HardIncident Management and Automated Alerting

An e-commerce platform running on Google Cloud processes order transactions through a fleet of Compute Engine virtual machines managed by a Managed Instance Group (MIG) behind an External HTTP(S) Load Balancer. During a flash sale event, downstream database contention caused transient query latency spikes. Because the load balancer health check was configured to query a deep database endpoint, the load balancer marked all backend instances as unhealthy simultaneously and initiated automated instance replacements, causing a complete service outage. Which architectural modification should you recommend to prevent this cascading operational failure and establish a resilient incident management and alerting workflow?

  1. Reconfigure the load balancer health check to target a lightweight local endpoint that validates only instance web server responsiveness, and implement Cloud Monitoring multi-window SLO burn-rate alerting to manage incident response.Answer
  2. B
    Configure the load balancer health check probes to execute deep database SELECT queries on every probe interval, and set an automated policy to restart instances immediately when health probes fail.
  3. C
    Assign the primitive Owner role to the automated incident remediation service account to grant unconstrained permissions to restart instances and alter VPC load balancer settings during an outage.
  4. D
    Establish static single-window threshold alerts on CPU usage, and apply a broad Cloud Logging exclusion filter to drop all 5xx HTTP application logs during traffic spikes to avoid notification throttling.

Answer

Reconfigure the load balancer health check to target a lightweight local endpoint that validates only instance web server responsiveness, and implement Cloud Monitoring multi-window SLO burn-rate alerting to manage incident response.
Reconfiguring health checks to monitor a local lightweight endpoint prevents load balancers from mistakenly terminating instances due to downstream dependency bottlenecks. Combining this with Cloud Monitoring SLO burn-rate alerting provides precise operational observability, allowing SRE teams to respond effectively to incident events.

Step-by-Step Solution

1
Analyze the root cause of the cascading outage
Identify that the health check endpoint tested downstream database availability rather than local instance responsiveness.
When downstream dependencies slow down, deep health checks fail across all instances simultaneously, leading to mass instance removal by the load balancer.
2
Isolate health probe concerns from application dependency testing
Modify the HTTP health check path to point to a shallow/lightweight status handler on the web server (e.g., returning HTTP 200 OK).
Shallow health checks ensure the load balancer only routes traffic away from genuinely dead or unresponsive compute instances.
3
Implement proper incident alerting for user-impacting performance degradation
Create Cloud Monitoring alerting policies based on multi-window SLO burn rates to notify SRE teams when database latency impacts user experience.
Burn-rate alerts trigger timely incident management workflows for sustained error budget consumption without causing unintended infrastructure restarts.

Key Concept

Incident Management and Health Check Architecture
Rate this question

Topics

Incident Management and Automated Alerting

More questions from this topic