An e-commerce company operates a real-time recommendation engine microservice on Google Cloud Run. The Site Reliability Engineering (SRE) team defines a Service Level Objective (SLO) requiring 99.9% of valid requests to return successful HTTP responses over a rolling 30-day window. During flash sales, transient traffic spikes occasionally consume substantial portions of the monthly error budget within a few hours before static alerts trigger, leading to exhaustion of the budget without timely intervention. Which monitoring and alerting configuration should the team implement in Cloud Monitoring to detect fast budget consumption before the error budget is fully depleted?
- Configure multi-window, multi-burn-rate alerts in Cloud Monitoring that measure short-term (e.g., 1-hour) and long-term (e.g., 6-hour) burn rates against a defined percentage of error budget consumption.Answer
- BSet up a static threshold metric alert in Cloud Monitoring that triggers an incident whenever CPU utilization on the Cloud Run instances exceeds 85% for more than 5 minutes.
- CCreate a static threshold alert on the 5xx HTTP error count metric to trigger when the error count exceeds 50 errors within a 5-minute rolling window.
- DRedefine the Service Level Indicator (SLI) metric target from 99.9% to 99.0% whenever traffic spikes occur to prevent premature alerting.
Answer
Configure multi-window, multi-burn-rate alerts in Cloud Monitoring that calculate consumption rates over multiple time windows.
Implementing multi-window, multi-burn-rate alerting in Cloud Monitoring allows SRE teams to measure the rate at which the error budget is being consumed over both short (e.g., 1 hour) and longer (e.g., 6 hours) windows. This ensures rapid detection of critical outages while preventing alert fatigue caused by minor, temporary spikes.
Step-by-Step Solution
Key Concept
Error Budget Burn Rate Alerting