Examkin

Soru

Zorluk: Çok zorIncident Management and Automated Alerting

An enterprise financial organization runs a mission-critical payment processing application on a Google Cloud Managed Instance Group (MIG) behind an External HTTP(S) Load Balancer. The SRE team faces two main operational challenges: first, frequent transient traffic bursts trigger alert storms that cause on-call engineer fatigue while failing to highlight sustained slow-burn reliability budget exhaustion; second, when specific corrupt payloads cause backend compute instances to fail, manual incident response causes elevated Mean Time to Resolution (MTTR).

The Site Reliability Engineering team needs an automated alerting and incident management pipeline that:
1. Accurately detects rapidly accelerating and slow-burn degradation of the request latency Service Level Objective (SLO) while preventing alert fatigue during temporary bursts.
2. Automatically isolates compromised VM instances for offline forensic analysis and replaces capacity without worsening backend load or risking cascading failure through deep load balancer health probes.

Which operational architecture should you recommend?

  1. Configure Cloud Monitoring Service Monitoring using multi-window, multi-burn-rate alert policies on latency SLIs. Route alerts through Pub/Sub to an automated Cloud Function that detaches failing instances from the MIG for forensic analysis, and maintain lightweight HTTP load balancer health check endpoints focused solely on local node health.Cevap
  2. B
    Configure static threshold Cloud Monitoring alert policies set to trigger when CPU utilization exceeds 85% or HTTP 5xx error rate exceeds 2% over a 5-minute window. Point the notification channel to a Webhook that executes a Cloud Function to automatically restart impacted Compute Engine instances.
  3. C
    Configure Load Balancer health checks to execute complex end-to-end database query checks on every backend endpoint. When health checks fail, configure a Cloud Logging log-based metric alert to publish an alert event directly to PagerDuty for manual operator triage.
  4. D
    Configure alert policies directly monitoring the static SLO target percentage (99.9%) as an instantaneous metric value. Attach a Pub/Sub notification channel executing a Cloud Function that operates using primitive Project Owner IAM roles to destroy and recreate the target VPC network.

Cevap

The optimal architecture uses Cloud Monitoring Service Monitoring with multi-window, multi-burn-rate alert policies tracking request latency SLIs, paired with Pub/Sub and Cloud Functions for automated instance isolation and lightweight load balancer health checks.
The solution employing multi-window, multi-burn-rate alerting on SLIs directly aligns with Google Cloud SRE practices. It prevents alert fatigue by requiring both short-window acceleration and long-window sustained consumption before alerting. Routing alert payloads via Pub/Sub to Cloud Functions enables automated cordoning, offline forensics, and MIG auto-healing, while lightweight health checks prevent cascading failures caused by downstream dependency degradation.

Adım Adım Çözüm

1
Analyze the alerting requirement to address alert fatigue and slow-burn detection.
Identify that static metrics or single short windows fail, whereas Google SRE multi-window multi-burn-rate alerts evaluate both short-term fast consumption and long-term slow consumption of the error budget.
Multi-burn-rate alerts reduce false positives from temporary spikes while ensuring critical SLO breaches trigger timely notifications.
2
Evaluate the automated incident response mechanism.
Publishing alert events to a Pub/Sub topic allows a decoupled serverless function (Cloud Functions) to isolate/detach failing compute instances for forensic inspection while the MIG provisions healthy replacements.
Automating instance cordoning and replacement minimizes MTTR without requiring manual operator intervention.
3
Assess load balancer health check configuration.
Keep health checks lightweight (testing basic endpoint/port responsiveness) rather than executing heavy downstream database dependencies.
Deep dependency health checks cause cascading cluster failures when a downstream database experiences transient load.

Anahtar Kavram

Multi-window multi-burn-rate alerting and decoupled automated incident remediation in GCP
Bu soruyu puanla

Konular

Incident Management and Automated Alerting

Bu konudan daha fazla soru