A security monitoring firm is designing an ingestion pipeline for 15,000 smart security ca… — AWS Certified Solutions Architect - Associate

A security monitoring firm is designing an ingestion pipeline for $15,000$ smart security cameras. The cameras generate a combined peak of $6,000$ motion detection events per second, with each event payload being a $2.5\text{ KB}$ JSON object. The company requires that events from each camera be processed in the exact order they are generated to maintain a chronological timeline. The data must be transformed into Apache Parquet format and stored in Amazon S3 for ad-hoc querying via Amazon Athena. The ingestion latency from event generation to storage in Amazon S3 must be less than $5$ minutes, and the solution should minimize operational management. Which architecture meets these requirements?

Ingest the data into an Amazon Kinesis Data Stream configured with On-Demand capacity, using the camera ID as the partition key. Configure Amazon Data Firehose to consume the data from the stream, convert the format to Apache Parquet using the AWS Glue Schema Registry, and write the output to Amazon S3 with a buffer interval of 300 seconds.Answer
B
Ingest the data into an Amazon SQS Standard queue, using the camera ID as the message group ID. Configure an AWS Glue ETL job running every 5 minutes to read messages from the queue, convert them to Apache Parquet format, and write the output to Amazon S3.
C
Ingest the data into an Amazon Kinesis Data Stream provisioned with 8 shards, using the camera ID as the partition key. Configure Amazon Data Firehose to consume from the stream, convert the JSON payloads to Apache Parquet using AWS Glue, and write to Amazon S3 with a buffer interval of 300 seconds.
D
Ingest the data into an Amazon Kinesis Data Stream configured with On-Demand capacity, using the camera ID as the partition key. Deploy a fleet of AWS Lambda functions running continuously to pool the stream, buffer the records in memory for 5 minutes to perform Parquet transformation, and upload the results to Amazon S3.

Answer

Ingest the data into an Amazon Kinesis Data Stream configured with On-Demand capacity, using the camera ID as the partition key, and use Amazon Data Firehose to perform format conversion and delivery to Amazon S3 with a 300-second buffer interval.

The correct architecture uses Amazon Kinesis Data Streams with On-Demand capacity to ingest high-velocity data. Using the camera ID as the partition key ensures that all events for a given camera are mapped to the same shard, which guarantees in-order delivery. Amazon Data Firehose integrates directly with Kinesis Data Streams, scales automatically, performs native JSON-to-Parquet conversion using the AWS Glue Schema Registry, and writes to Amazon S3 within the 5-minute latency requirement, minimizing both operational overhead and infrastructure management.

Step-by-Step Solution

Calculate the peak data throughput and record rate to determine scale requirements.

The total throughput is

6,000 \text{ records/sec} \times 2.5\text{ KB} = 15,000\text{ KB/s}

(or

15\text{ MB/s}

), with a total of

6,000

records per second.

This calculation determines the necessary scaling capacity for the ingestion stream.

Choose the streaming ingestion service that supports scale and strict ordering.

Amazon Kinesis Data Streams with the camera ID as the partition key. Kinesis Data Streams On-Demand automatically scales up to

200\text{ MB/s}

and

200,000

records/sec, and partition keys guarantee ordering per camera.

Ensures that the pipeline does not throttle during peak traffic and preserves the required chronological sequence.

Select a serverless delivery and transformation mechanism.

Amazon Data Firehose with native AWS Glue schema conversion, configured with a 300-second buffer interval writing to Amazon S3.

Allows direct conversion of JSON to Parquet format without provisioning server resources, while adhering to the 5-minute ingestion latency window.

Key Concept

High-performing real-time data ingestion pipelines must dynamically scale to meet peak throughput while satisfying operational simplicity and strict message ordering constraints by using Amazon Kinesis Data Streams with partition keys coupled with Amazon Data Firehose for format transformation.

Question

Answer

Step-by-Step Solution

Key Concept

Common Mistakes

Topics

More questions from this topic