Question

Difficulty: HardMonitoring and Logging

An Site Reliability Engineering (SRE) team is managing a fleet of GKE microservices that output custom application logs to Cloud Logging. The team needs to configure automated real-time incident routing whenever application logs register a severity of `ERROR`. The solution must trigger an alerting mechanism that sends notifications directly to a Cloud Pub/Sub topic for downstream automated remediation. Which TWO actions must the team perform to achieve this requirement using Google-recommended practices?

  1. Create a custom log-based metric in Cloud Logging with a log filter defined for severity `ERROR`.Answer
  2. Configure an alerting policy in Cloud Monitoring that monitors the log-based metric threshold and targets a Pub/Sub notification channel.Answer
  3. C
    Install and configure the legacy Stackdriver Logging Agent on all GKE cluster worker nodes to stream log records directly to Cloud Pub/Sub.
  4. D
    Grant the Project Editor (`roles/editor`) primitive role to the Pub/Sub service account to enable event publishing permissions.

Answer

To implement real-time incident routing based on error log entries, the SRE team must create a log-based counter metric in Cloud Logging filtered for `ERROR` severity, and then create a Cloud Monitoring alerting policy evaluating this metric that sends alerts to a Pub/Sub notification channel.
To trigger automated workflows from log events in Google Cloud, log entries must first be converted into metric data using a custom log-based metric in Cloud Logging. Once the metric exists, a Cloud Monitoring alerting policy can monitor the metric stream and publish alert notifications directly to a Cloud Pub/Sub topic channel.

Step-by-Step Solution

1
Define a log filter for `severity = ERROR` in Cloud Logging and create a log-based counter metric.
Cloud Logging aggregate count data becomes available as a metric stream inside Cloud Monitoring.
Alerting policies in Cloud Monitoring evaluate time-series metric data, which requires converting discrete log occurrences into a log-based metric.
2
Create an alerting policy in Cloud Monitoring based on the new log-based metric and assign a Pub/Sub notification channel.
When the metric crosses the specified threshold, Cloud Monitoring publishes an alert payload message to the designated Pub/Sub topic.
Cloud Monitoring natively supports Cloud Pub/Sub as a notification channel type for automated incident handling workflows.

Key Concept

Log-based Metrics and Cloud Monitoring Alerting Policies with Pub/Sub Notification Channels
Rate this question