A financial data analytics platform runs microservices on Google Cloud that generate log entries during routine nightly database maintenance. These maintenance activities cause temporary error logs for up to 3 minutes, which currently trigger false alarms and wake up on-call engineers. Operations requirements mandate that genuine outages—where error log rates persist beyond 5 minutes—must automatically initiate incident workflows via Cloud Pub/Sub and notify PagerDuty, without losing error logs required for compliance auditing. How should you configure the Cloud Monitoring alerting policy to satisfy these operational requirements?
- Create a log-based metric for error logs, and set the alert policy condition duration to 5 minutes before triggering notifications to Pub/Sub and PagerDuty.Answer
- BConfigure Cloud Logging aggregate exclusion filters to drop all database error log entries automatically during the nightly maintenance window.
- CReconfigure the backend Load Balancer health checks to execute a deep query against the primary database every 3 seconds to auto-drain traffic during maintenance.
- DAssign the primitive Owner role to the monitoring service account and configure static zero-delay threshold alerts that trigger local shell scripts on a management VM.
Answer
Create a log-based metric for error logs, and set the alert policy condition duration to 5 minutes before triggering notifications to Pub/Sub and PagerDuty.
Configuring a log-based metric with an alert condition duration of 5 minutes ensures that transient error bursts occurring during 3-minute maintenance windows do not trigger on-call alerts, while genuine sustained issues immediately trigger incident response notifications through Pub/Sub and PagerDuty.
Step-by-Step Solution
Key Concept
Cloud Monitoring Alert Policy Duration & Log-based Metric Filtering