A company is designing an IoT telemetry ingestion pipeline. During peak hours, thousands o… — AWS Certified Solutions Architect - Associate

A company is designing an IoT telemetry ingestion pipeline. During peak hours, thousands of devices send JSON payload messages to AWS, resulting in a data ingestion rate of $15\text{ MB/s}$ and $15,000\text{ records/s}$ (averaging $1\text{ KB}$ per record). The data must be partitioned by region and timestamp, converted into Apache Parquet format, and stored in Amazon S3. The total latency from data generation to S3 delivery must not exceed $2\text{ minutes}$ . The pipeline must scale dynamically to handle traffic spikes without manual administration. Which architecture meets these requirements with the highest performance and least operational overhead?

Configure Amazon Kinesis Data Firehose with Amazon Kinesis Data Streams in on-demand mode as the source. Enable dynamic partitioning in Kinesis Data Firehose using jq expressions to partition by region and timestamp, and enable format conversion to Parquet using the AWS Glue Schema Registry before delivering the data to Amazon S3.Cevap
B
Configure a standard Amazon SQS queue to ingest the payloads. Set up an AWS Lambda function triggered by the queue to partition the data, convert it to Parquet, and write it to Amazon S3. Choose this configuration to guarantee strict first-in, first-out ordering of the incoming telemetry events.
C
Configure Amazon Kinesis Data Streams in provisioned mode with $5$ shards to ingest the data. Attach an Amazon Kinesis Data Firehose delivery stream to the data stream to handle format conversion to Parquet and S3 partitioning. The stream will automatically scale its capacity dynamically to absorb the peak throughput.
D
Configure Amazon Kinesis Data Streams in provisioned mode with a single shard. Use an AWS Lambda function triggered by the stream to convert the records to Parquet and save them to Amazon S3. The Lambda function will partition the records dynamically based on the payload attributes.

Cevap

The architecture using Kinesis Data Streams in on-demand mode combined with Kinesis Data Firehose's dynamic partitioning and format conversion provides the highest performance and lowest operational overhead.

The correct solution utilizes Amazon Kinesis Data Streams in on-demand mode, which instantly scales to accommodate ingestion rates up to

200\text{ MB/s}

200,000\text{ records/s}

. Coupling this stream with Amazon Kinesis Data Firehose allows the pipeline to ingest, dynamically partition (using jq expressions on payload attributes), and convert JSON to Parquet format (using AWS Glue Schema Registry) before storing in S3. This managed solution meets the sub-2-minute latency requirement with zero administrative overhead.

Adım Adım Çözüm

Calculate the ingestion throughput requirements.

The peak throughput is

15\text{ MB/s}

and

15,000\text{ records/s}

. This exceeds the limits of standard low-capacity configurations (such as a single shard or small shard pools).

To ensure no data loss or throttling, the ingestion layer must support at least

15\text{ MB/s}

and

15,000\text{ records/s}

Determine the scaling mechanism for the ingestion tier.

Using Kinesis Data Streams with on-demand capacity mode allows the stream to automatically scale up to

200\text{ MB/s}

and

200,000\text{ records/s}

write throughput without manual shard management.

On-demand capacity mode eliminates the need to manage shard scaling manually or build complex autoscaling scripts, satisfying the requirement to scale dynamically without manual administration.

Select the transformation and partitioning mechanism.

Amazon Kinesis Data Firehose with Glue Schema Registry can automatically convert JSON payloads to Parquet format and write them to S3 using dynamic partitioning based on jq expressions.

Performing format conversion and partitioning natively within Kinesis Data Firehose removes the operational overhead of running custom Lambda functions or Glue ETL jobs, while meeting the 2-minute latency delivery window.

Anahtar Kavram

High-throughput real-time ingestion requires automated scaling (Kinesis Data Streams on-demand mode) paired with managed, serverless delivery and transformation tools (Kinesis Data Firehose dynamic partitioning and Parquet conversion) to minimize operational overhead and latency.

Tahmini Süre:2m 0s

Soru

Cevap

Adım Adım Çözüm

Anahtar Kavram

Sık Yapılan Hatalar

Konular

Bu konudan daha fazla soru