Examkin

Question

Difficulty: HardHigh-Performing Data Ingestion and Transformation Solutions

A smart grid utility company is designing a real-time data ingestion and transformation pipeline to process telemetry from 50,00050,000 smart grid sensors. Each sensor transmits a 5 KB5\text{ KB} JSON payload 1010 times per second. The pipeline must guarantee chronological order of records for each individual sensor. The raw JSON payloads must be converted into Apache Parquet format and stored in Amazon S3, partitioned by sensor ID and hour, with a maximum latency of 55 minutes. Which architecture provides the highest performance and scaling capability with the least operational overhead?

  1. Ingest data into Amazon Kinesis Data Streams provisioned with 2,5002,500 shards, using the sensor ID as the partition key. Configure an Amazon Data Firehose stream with the Kinesis stream as the source, enable dynamic partitioning based on the sensor ID and timestamp, use AWS Glue Schema Registry for JSON-to-Parquet conversion, and write to Amazon S3.Answer
  2. B
    Ingest data into Amazon Kinesis Data Streams configured in On-Demand mode, using the sensor ID as the partition key. Configure an Amazon Data Firehose stream with the Kinesis stream as the source. Use an AWS Lambda function invoked by Firehose to transform the format to Parquet, and write to Amazon S3.
  3. C
    Ingest data into Amazon SQS Standard queues to decouple the sensors. Configure an AWS Glue ETL job to run every 55 minutes to read from the SQS queues, convert the JSON payloads to Parquet format, partition the data by sensor ID and hour, and write the output to Amazon S3.
  4. D
    Ingest data directly into Amazon Data Firehose using the DirectPUT API. Configure Firehose to partition the incoming data dynamically by sensor ID and hour, convert the JSON payloads to Parquet format using the AWS Glue Schema Registry, and write the output to Amazon S3.

Answer

Ingest data into Amazon Kinesis Data Streams provisioned with 2,5002,500 shards, using the sensor ID as the partition key. Configure an Amazon Data Firehose stream with the Kinesis stream as the source, enable dynamic partitioning based on the sensor ID and timestamp, use AWS Glue Schema Registry for JSON-to-Parquet conversion, and write to Amazon S3.
The correct architecture uses Amazon Kinesis Data Streams in provisioned mode with at least 2,5002,500 shards. This satisfies the 2.5 GB/s2.5\text{ GB/s} throughput limit (1 MB/s1\text{ MB/s} per shard write capacity). Using the sensor ID as the partition key ensures that records for each sensor are sent to the same shard, preserving chronological order. Amazon Data Firehose integrates directly with Kinesis Data Streams, converts JSON payloads to Parquet format using the AWS Glue Schema Registry, and utilizes dynamic partitioning to organize the files in Amazon S3 by sensor ID and hour, meeting all requirements with minimal operational overhead.

Step-by-Step Solution

1
Calculate the total ingestion metrics.
Total records/sec = 50,000×10=500,00050,000 \times 10 = 500,000 records/sec. Total throughput = 500,000×5 KB=2,500,000 KB/s≈2.44 GB/s500,000 \times 5\text{ KB} = 2,500,000\text{ KB/s} \approx 2.44\text{ GB/s} (2,500 MB/s2,500\text{ MB/s}).
Sizing the ingestion layer requires calculating both the throughput (MB/s) and record rate (records/s).
2
Select and size the ingestion service.
Amazon Kinesis Data Streams in provisioned mode with at least 2,5002,500 shards is selected. SQS FIFO is ruled out due to throughput limits (30,00030,000 records/sec max). Kinesis On-Demand is ruled out due to its default capacity limit (200 MB/s200\text{ MB/s}).
Each Kinesis shard supports a maximum write capacity of 1 MB/s1\text{ MB/s} or 1,0001,000 records/sec. 2,500 MB/s/1 MB/s per shard=2,5002,500\text{ MB/s} / 1\text{ MB/s per shard} = 2,500 shards. Sizing is bound by throughput rather than record count.
3
Determine ordering and delivery requirements.
Using the sensor ID as the partition key ensures all records from a given sensor route to the same shard, preserving chronological order. Amazon Data Firehose is used to consume from the Kinesis stream.
Firehose DirectPUT does not guarantee event ordering. Processing order can only be guaranteed by partition keys in Kinesis.
4
Configure the transformation and storage destination details.
Enable dynamic partitioning in Firehose using jq queries to extract the sensor ID and timestamp. Enable JSON-to-Parquet conversion using the AWS Glue Schema Registry, writing results directly to Amazon S3.
Using native Firehose transformations and dynamic partitioning minimizes operational overhead and satisfies the 5-minute latency limit through buffering.

Key Concept

High-throughput data ingestion scaling limits (Kinesis shard calculations), ordering requirements, and serverless format conversion.
Rate this question

Topics

High-Performing Data Ingestion and Transformation Solutions

More questions from this topic