An enterprise DevOps team manages a critical microservice deployed on Google Kubernetes Engine (GKE). The microservice writes structured JSON payload logs containing a custom numerical field, `processing_latency_ms`. The Site Reliability Engineering (SRE) team needs to evaluate latency distributions and set up automated alerts whenever the 99th percentile latency exceeds 500 ms over a rolling 5-minute window across all clusters. How should the team configure Google Cloud Logging and Monitoring to fulfill this operational requirement while strictly adhering to security least-privilege principles?
- Create a distribution log-based metric in Cloud Logging extracting `processing_latency_ms` from incoming log entries, and configure a Cloud Monitoring alerting policy targeting the 99th percentile of this metric aggregated over a 5-minute alignment window.Answer
- BGrant the Editor primitive role to the service account executing log routing so that it can dynamically create log sinks, modify Cloud Monitoring metrics, and update alerting channels across the project without encountering permission errors.
- CGrant the Service Account Admin role (`roles/iam.serviceAccountAdmin`) to the alerting service account so it can impersonate the Logging service account and create metric descriptors on demand.
- DCreate a counter log-based metric for latency occurrences and submit a regional quota increase request for Cloud Logging API read limits to prevent alerting delays during high traffic bursts.
Answer
Create a distribution log-based metric in Cloud Logging extracting processing_latency_ms from incoming log entries, and configure a Cloud Monitoring alerting policy targeting the 99th percentile of this metric aggregated over a 5-minute alignment window.
The correct answer leverages a distribution log-based metric to extract numerical values (`processing_latency_ms`) directly from structured JSON log entries. Cloud Monitoring can then compute distribution statistics, such as the 99th percentile over a 5-minute alignment period, and trigger alerts accordingly without requiring elevated privileges or additional application code.
Step-by-Step Solution
Key Concept
Configuring distribution log-based metrics and operational alert threshold policies in Google Cloud Operations suite.
Estimated Time:2m 0s