An energy utility company is designing an ingestion pipeline for 5,000,000 smart grid sens… — AWS Certified Solutions Architect - Associate

An energy utility company is designing an ingestion pipeline for $5,000,000$ smart grid sensors. Each sensor transmits consumption data every 20 seconds, resulting in an average ingestion rate of $250,000$ records per second. Each payload averages $2\text{ KB}$ in size. During peak grid events, traffic spikes immediately to $500,000$ records per second ( $1\text{ GB/s}$ ). The pipeline must satisfy the following architectural requirements: 1) Deliver the bulk metrics to Amazon S3 in Apache Parquet format within 5 minutes of ingestion for historical analysis. 2) Filter and route critical grid failure events (less than $0.1\%$ of total traffic) to an operational dashboard with sub-second latency. 3) Ensure that updates from each individual sensor are processed in the order they are received to maintain data integrity. Which architecture meets these requirements with the highest performance and the least risk of data ingestion throttling?

A
Ingest the sensor data into an Amazon SQS Standard queue. Configure an AWS Lambda function to poll the SQS queue, extract critical grid failure events for real-time alerting, and forward the remaining data to Amazon Data Firehose. Configure Amazon Data Firehose to buffer the data for 5 minutes, convert it to Apache Parquet format, and store it in Amazon S3.
B
Configure an Amazon Kinesis Data Stream in On-Demand mode, using the sensor ID as the partition key. Configure Amazon Data Firehose to read from the Kinesis Data Stream, buffer the data for 5 minutes, convert the format to Apache Parquet using AWS Glue Schema Registry, and write to Amazon S3. Simultaneously, configure an AWS Lambda function as a consumer of the Kinesis Data Stream to process and alert on grid failure events.
Configure an Amazon Kinesis Data Stream in Provisioned mode with $1,000$ shards, using the sensor ID as the partition key. Configure Amazon Data Firehose to read from the Kinesis Data Stream, buffer the data for 5 minutes, convert the format to Apache Parquet using AWS Glue Schema Registry, and write to Amazon S3. Simultaneously, configure an AWS Lambda function as a consumer of the Kinesis Data Stream to process and alert on grid failure events.Cevap
D
Configure an Amazon Kinesis Data Stream in Provisioned mode with $1,000$ shards. Write the ingested data directly to an Amazon DynamoDB table with a partition key of the ingestion timestamp rounded to the nearest second. Use DynamoDB Streams to trigger an AWS Lambda function for real-time alerting, and run a daily AWS Glue ETL job to export the table data to Amazon S3 in Apache Parquet format.

Cevap

Configure an Amazon Kinesis Data Stream in Provisioned mode with 1,000 shards, using the sensor ID as the partition key. Configure Amazon Data Firehose to read from the Kinesis Data Stream, buffer the data for 5 minutes, convert the format to Apache Parquet using AWS Glue Schema Registry, and write to Amazon S3. Simultaneously, configure an AWS Lambda function as a consumer of the Kinesis Data Stream to process and alert on grid failure events.

The correct architecture uses Amazon Kinesis Data Streams in Provisioned mode with 1,000 shards because the peak throughput is 1 GB/s (

500,000 \text{ records/second} \times 2 \text{ KB/record}

), which matches the maximum write capacity of 1,000 shards (

1 \text{ MB/s}

per shard). Using the sensor ID as the partition key ensures that updates from individual sensors are processed in chronological order while distributing the workload evenly. The dual-consumer configuration allows AWS Lambda to process critical events with sub-second latency, while Amazon Data Firehose asynchronously buffers, transforms to Parquet via AWS Glue, and stores the bulk data in Amazon S3 within the 5-minute latency requirement.

Adım Adım Çözüm

Calculate average and peak ingestion throughput requirements.

Average write throughput:

250,000 \text{ records/second} \times 2 \text{ KB/record} = 500 \text{ MB/s}

. Peak write throughput:

500,000 \text{ records/second} \times 2 \text{ KB/record} = 1 \text{ GB/s}

(

1,000 \text{ MB/s}

Sizing the ingestion mechanism requires knowing both the record volume and raw throughput.

Determine ingestion stream mode and shard count.

A single shard in Kinesis Data Streams supports up to

1 \text{ MB/s}

1,000 \text{ records/second}

write capacity. Peak throughput (

1 \text{ GB/s}

) requires

1,000

shards (

1,000 \text{ MB/s} / 1 \text{ MB/s}

). Kinesis On-Demand mode has default write limits of

200 \text{ MB/s}

and does not scale instantaneously to handle a 1 GB/s spike, making Provisioned mode with 1,000 shards the correct choice.

On-demand limits and scaling latency can cause ingestion throttling during sudden spikes.

Select a partition key strategy to preserve message order.

Using the sensor ID as the partition key ensures that all messages from a specific sensor map to the same shard. This guarantees strict ordering of updates per sensor while distributing the overall workload evenly across all 1,000 shards.

Preserving chronological update sequences per sensor is required to maintain grid data integrity.

Design dual consumer paths for sub-second alerting and 5-minute storage.

AWS Lambda is configured to poll the Kinesis Data Stream shards directly for real-time anomalous alerts (sub-second latency). Concurrently, Amazon Data Firehose reads from the stream, buffers for 5 minutes, transforms data to Parquet using AWS Glue Schema Registry, and writes to Amazon S3.

Handling mixed-latency destinations requires separate ingestion consumers to isolate low-latency tasks from batch-oriented delivery paths.

Anahtar Kavram

High-throughput data ingestion using Amazon Kinesis Data Streams Provisioned capacity, partition key selection for order preservation, and dual-consumer routing for mixed-latency outputs.

Tahmini Süre:3m 0s

Soru

Cevap

Adım Adım Çözüm

Anahtar Kavram

Sık Yapılan Hatalar

Konular

Bu konudan daha fazla soru