Examkin

Question

Difficulty: HardCloudWatch Dashboards and Container Insights

A SysOps Administrator has deployed Amazon CloudWatch Container Insights on an Amazon EKS cluster running on Amazon EC2 worker nodes by running the CloudWatch agent as a DaemonSet. The Administrator notes that while the agent's diagnostic logs are successfully being written to CloudWatch, no Container Insights performance metrics (such as cluster, node, or pod CPU utilization) are appearing in the CloudWatch console. What is the most likely cause of this issue?

  1. A
    Detailed monitoring has not been enabled on the EC2 instances hosting the EKS worker nodes, which prevents CloudWatch from collecting container-level metrics.
  2. B
    The log retention period for the `/aws/containerinsights/<cluster-name>/performance` log group has been configured to expire immediately, preventing CloudWatch from extracting metrics.
  3. The ServiceAccount used by the CloudWatch agent DaemonSet is not bound to the required ClusterRole, preventing the agent from querying the Kubernetes API for container and node metrics.Answer
  4. D
    An Amazon EventBridge rule is missing to route performance log events from the CloudWatch Logs group to the CloudWatch metrics engine.

Answer

The ServiceAccount used by the CloudWatch agent DaemonSet is not bound to the required ClusterRole, preventing the agent from querying the Kubernetes API for container and node metrics.
To retrieve the performance metrics for Container Insights, the CloudWatch agent running in EKS needs read permissions to the Kubernetes API and Kubelet stats endpoint. This is achieved by creating a ServiceAccount for the DaemonSet pods and linking it to a ClusterRole that has get, list, and watch permissions on resources like pods and nodes. The link must be established using a ClusterRoleBinding. If this binding is missing, the agent cannot access the API, preventing it from writing performance events, which results in empty Container Insights metrics.

Step-by-Step Solution

1
Examine the container logs of the CloudWatch agent DaemonSet pods using the kubectl logs command.
Identify permission errors (such as HTTP 403403 Forbidden) when the agent attempts to fetch node and pod resource metrics from the API server or kubelet stats summary endpoint.
This confirms that the pod is failing to authenticate or lacks authorization to gather the required container-level telemetry data.
2
Verify the Role-Based Access Control (RBAC) definitions in the cluster, checking for the existence of the ClusterRole, ServiceAccount, and ClusterRoleBinding.
Confirm that while the ServiceAccount and ClusterRole are defined, the ClusterRoleBinding linking the ServiceAccount to the ClusterRole is either missing or incorrectly configured.
A ServiceAccount on Kubernetes cannot access cluster-level endpoints without a ClusterRoleBinding associating it with the proper ClusterRole.
3
Create and apply the ClusterRoleBinding manifest targeting the cloudwatch-agent ServiceAccount in the amazon-cloudwatch namespace.
The CloudWatch agent successfully queries the Kubernetes API, starts logging performance telemetry to the performance log group, and metrics populate in CloudWatch.
Correctly binding the ServiceAccount grants the containerized agent authorization to collect and emit the Embedded Metric Format logs required for Container Insights.

Key Concept

CloudWatch Container Insights EKS RBAC Requirements
Estimated Time:2m 0s
Rate this question

Topics

CloudWatch Dashboards and Container Insights

More questions from this topic