Examkin

Soru

Zorluk: ZorHigh-Performing Data Ingestion and Transformation Solutions

A logistics company is designing an ingestion pipeline to track high-frequency GPS telemetry from a fleet of 100,000100,000 delivery vehicles. Each vehicle transmits a 2 KB2\text{ KB} JSON payload every 10 seconds10\text{ seconds} containing status updates that must be processed in strict chronological order per vehicle to calculate accurate transit times. The data must be converted to Apache Parquet format and stored in Amazon S3 within 5 minutes5\text{ minutes} of generation. During peak morning hours, a grid-lock event can cause all vehicles to transmit data within a 2 second2\text{ second} window, resulting in transient spikes of up to 50,00050,000 requests per second. Which architecture meets these requirements most performantly and reliably?

  1. Configure an Amazon Kinesis Data Stream in On-Demand mode, using the vehicle ID as the partition key. Configure an Amazon Data Firehose delivery stream to consume from the Kinesis stream, transform the JSON payloads to Apache Parquet format using an AWS Glue table, and deliver the data to Amazon S3 with a buffer interval of 300 seconds300\text{ seconds}.Cevap
  2. B
    Configure an Amazon Kinesis Data Stream in Provisioned mode with 1010 shards, using the vehicle ID as the partition key. Configure an Amazon Data Firehose delivery stream to consume from the Kinesis stream, convert the format to Apache Parquet using AWS Glue, and deliver the data to Amazon S3.
  3. C
    Configure an Amazon SQS standard queue to receive the telemetry data. Use an AWS Lambda function to poll the SQS queue, batch the records, convert them to Apache Parquet format, and write them to Amazon S3.
  4. D
    Configure an Amazon SQS FIFO queue to ingest the telemetry data using the vehicle ID as the MessageGroupID. Configure an AWS Lambda function to retrieve messages from the queue, transform them to Apache Parquet format, and upload the files to Amazon S3.

Cevap

Configure an Amazon Kinesis Data Stream in On-Demand mode with the vehicle ID as the partition key, and use Amazon Data Firehose to transform the data to Parquet via AWS Glue and deliver it to Amazon S3 with a 5-minute buffer interval.
The correct architecture uses Amazon Kinesis Data Streams in On-Demand mode, which scales dynamically to support up to 200 MB/sec200\text{ MB/sec} of write throughput, easily absorbing the 100 MB/sec100\text{ MB/sec} peak spike. Partitioning by the vehicle ID ensures that all status updates for a single vehicle are directed to the same shard, preserving their chronological sequence. Amazon Data Firehose then consumes from the stream, transforms the JSON payload to Parquet format using metadata defined in AWS Glue, and writes the output files to Amazon S3 within the required 5-minute buffering window (300 seconds300\text{ seconds}). This provides a highly performant, fully managed, and cost-effective ingestion pipeline.

Adım Adım Çözüm

1
Calculate the average and peak throughput requirements for the ingestion layer.
Average throughput is 10,00010,000 records/sec (20 MB/sec20\text{ MB/sec}), and peak throughput is 50,00050,000 records/sec (100 MB/sec100\text{ MB/sec}).
This determines the minimum shard or partition capacity needed to prevent ingestion bottlenecks.
2
Select an ingestion service that supports both strict ordering per device and the calculated throughput scaling.
Kinesis Data Streams On-Demand is selected, using the vehicle ID as the partition key.
On-Demand mode automatically scales to accommodate the 100 MB/sec100\text{ MB/sec} peak, while the partition key guarantees that all updates for a specific vehicle are routed to the same shard and processed sequentially.
3
Design a serverless transformation and delivery mechanism to output Parquet files to Amazon S3.
Amazon Data Firehose consumes from the Kinesis stream, converts JSON to Parquet using AWS Glue Schema Registry, and writes to Amazon S3.
Firehose handles the buffering, format conversion, and S3 delivery serverlessly without needing to manage EC2 instances or custom consumer application scaling.

Anahtar Kavram

High-throughput real-time data ingestion with ordering guarantees and serverless format transformation.
Tahmini Süre:2m 30s
Bu soruyu puanla

Konular

High-Performing Data Ingestion and Transformation Solutions

Bu konudan daha fazla soru