Question

Difficulty: MediumImproving Operational Excellence via Monitoring and Logging

A biotechnology company has a multi-account AWS environment managed under AWS Organizations. Genomic sequencing applications run on Amazon EC2 instances across multiple member accounts. The application writes log files to a local directory `/var/log/sequencer/`. To manage disk space, the application rotates logs hourly, creating new files with timestamp suffixes (for example, `sequencer_20260716-11.log`). A Solutions Architect needs to design a centralized logging solution to collect these application logs and AWS CloudTrail logs from all member accounts into a single Amazon S3 bucket in a dedicated Security account. The architecture must stream application logs in real time and minimize operational overhead.

Which solution should the Solutions Architect implement to meet these requirements?

  1. A
    Configure a cron job on the EC2 instances to run a script hourly that uses the AWS CLI to upload `/var/log/sequencer/sequencer_*.log` files directly to the Security account S3 bucket. Create individual AWS CloudTrail trails in each member account and configure them to deliver log files to a local S3 bucket in each account. Set up Amazon S3 Cross-Region Replication (CRR) to replicate the CloudTrail logs from the local buckets to the centralized S3 bucket in the Security account.
  2. B
    Configure the CloudWatch agent on the EC2 instances with the static `file_path` parameter set to `/var/log/sequencer/sequencer.log`. Set up a cron job on the EC2 instances that runs hourly to copy the active timestamped log file to `/var/log/sequencer/sequencer.log`. Stream the logs to Amazon Kinesis Data Firehose to write to the Security account S3 bucket. Create an organization trail in AWS CloudTrail that delivers log files directly to the Security account S3 bucket, using a bucket policy that grants `s3:PutObject` permissions to the CloudTrail service principal with a condition checking for the Organization ID.
  3. Configure the CloudWatch agent on the EC2 instances with the `file_path` parameter set to `/var/log/sequencer/sequencer_*.log` to stream logs to Amazon CloudWatch Logs. Use CloudWatch Logs subscription filters to send the log events to a centralized Amazon Kinesis Data Firehose delivery stream that writes to the Security account S3 bucket. Create an organization trail in AWS CloudTrail that delivers log files directly to the Security account S3 bucket, ensuring the bucket policy grants `s3:PutObject` permissions to the CloudTrail service principal with a condition checking for the Organization ID.Answer
  4. D
    Configure the CloudWatch agent on the EC2 instances with the `file_path` parameter set to `/var/log/sequencer/sequencer_*.log` to stream logs to Amazon CloudWatch Logs. Use CloudWatch Logs subscription filters to send the log events to a centralized Amazon Kinesis Data Firehose delivery stream that writes to the Security account S3 bucket. Create an organization trail in AWS CloudTrail that delivers log files directly to the Security account S3 bucket, but configure the S3 bucket policy in the Security account to grant `s3:PutObject` permissions only to the IAM roles of the member accounts rather than the CloudTrail service principal.

Answer

Configure the CloudWatch agent with a wildcard path (`/var/log/sequencer/sequencer_*.log`) to track rotated logs in real time, stream them using CloudWatch subscription filters and Kinesis Data Firehose to the Security account's S3 bucket, and configure an organization trail in CloudTrail with a bucket policy that grants access to the CloudTrail service principal restricted by the Organization ID.
The correct solution uses a wildcard pattern in the CloudWatch agent configuration (`sequencer_*.log`) which natively detects and monitors newly created timestamped log files as they are rotated, ensuring real-time log ingestion without custom scripts. It also uses CloudWatch subscription filters to stream these logs to Kinesis Data Firehose for centralization, and leverages an AWS CloudTrail organization trail with the correct S3 bucket policy allowing the CloudTrail service principal to write to the centralized bucket using the Organization ID condition, minimizing operational overhead.

Step-by-Step Solution

1
Configure the CloudWatch agent using wildcard patterns for rotated logs.
The agent dynamically detects and streams newly generated log files without requiring manual symlinks or file copying.
Ensures operational excellence by avoiding missing log data during rotation.
2
Set up Kinesis Data Firehose with CloudWatch Logs subscription filters.
Log events are streamed in real time from member accounts to the central S3 bucket in the Security account.
Provides an automated, scalable streaming pipeline for cross-account log centralization.
3
Configure an Organization-wide CloudTrail and verify the Security account's S3 bucket policy.
All AWS API logs are delivered to the central S3 bucket with write permissions granted to the `cloudtrail.amazonaws.com` service principal and restricted by Organization ID.
Secures cross-account log delivery and simplifies configuration across the Organization.

Key Concept

Centralized multi-account logging for applications and CloudTrail with optimal CloudWatch agent configuration for rotated log files.
Rate this question