A logistics company is designing a high-performance data ingestion pipeline for real-time … — AWS Certified Solutions Architect - Associate

A logistics company is designing a high-performance data ingestion pipeline for real-time telemetry from $150,000$ active delivery vehicles. The peak ingestion rate is $45\text{ MB/s}$ with an average payload size of $3\text{ KB}$ per message. Telemetry data from each vehicle must be processed in the exact chronological order it was generated to calculate accurate velocity trends. The company wants to convert the telemetry from JSON to Apache Parquet format and store it in an Amazon S3 data lake within $5\text{ minutes}$ of arrival. Which architecture meets these requirements with the highest performance and the lowest operational overhead?

Deploy Amazon Kinesis Data Streams in On-Demand capacity mode, utilizing the vehicle ID as the partition key. Configure Amazon Kinesis Data Firehose to consume from the stream, perform format conversion from JSON to Apache Parquet using AWS Glue Schema Registry, and deliver the output to the Amazon S3 data lake.Cevap
B
Deploy Amazon Kinesis Data Streams in Provisioned capacity mode configured with $10$ shards, utilizing the vehicle ID as the partition key. Configure Amazon Kinesis Data Firehose to consume from the stream, perform format conversion to Apache Parquet, and write the output to the Amazon S3 data lake.
C
Deploy an Amazon SQS standard queue to receive the telemetry data. Configure an AWS Lambda function triggered by the queue to process the messages in batches, perform format conversion to Apache Parquet, and upload the files to the Amazon S3 data lake.
D
Deploy an Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster with three active brokers. Configure an Apache Kafka Connect Amazon S3 sink connector to consume from the topic, convert the data to Apache Parquet, and write the output to the Amazon S3 data lake.

Cevap

Deploy Amazon Kinesis Data Streams in On-Demand capacity mode with the vehicle ID as the partition key, and use Amazon Kinesis Data Firehose with AWS Glue Schema Registry to convert the JSON data to Apache Parquet format before storing it in Amazon S3.

The correct architecture uses Amazon Kinesis Data Streams in On-Demand capacity mode alongside Amazon Kinesis Data Firehose. Kinesis Data Streams guarantees ordering per partition key (vehicle ID), and the On-Demand mode dynamically scales write throughput up to

200\text{ MB/s}

200,000\text{ records/s}

without administrative intervention. Kinesis Data Firehose consumes from the stream, performs serverless format conversion to Apache Parquet using AWS Glue Schema Registry, and delivers the data to Amazon S3 within the required 5-minute buffering interval. This provides a highly performant and scalable solution with the lowest operational overhead.

Adım Adım Çözüm

Analyze ordering requirements

Vehicle telemetry requires strict chronological order per vehicle ID, meaning partition keys based on vehicle ID must be used to keep records on the same partition/shard.

Ensures that sequential messages from the same vehicle are processed in order.

Calculate ingestion throughput and shard capacity limits

Peak throughput is

45\text{ MB/s}

and message rate is

15,000\text{ records/s}

(

45\text{ MB/s} / 3\text{ KB}

). A single Kinesis shard supports

1\text{ MB/s}

1,000\text{ records/s}

. This requires at least

45

shards at peak.

Ensures the stream is sized correctly to prevent ProvisionedThroughputExceededException errors during peak loads.

Select ingestion stream capacity mode

Choose Kinesis Data Streams with On-Demand capacity mode to automatically scale throughput up to

200\text{ MB/s}

and

200,000\text{ records/s}

without manual administrative effort.

Meets the requirement for minimum operational overhead while accommodating fluctuating traffic spikes.

Design the transformation and delivery tier

Use Amazon Kinesis Data Firehose to pull data from Kinesis Data Streams, perform serverless Parquet formatting via integration with AWS Glue Schema Registry, and write to S3 within a

5\text{-minute}

buffering window.

Allows near-real-time transformation and ingestion into S3 without managing custom EC2 or ECS transformation nodes.

Anahtar Kavram

High-performing data ingestion and format transformation using Kinesis Data Streams On-Demand and Kinesis Data Firehose with minimal operational overhead.

Soru

Cevap

Adım Adım Çözüm

Anahtar Kavram

Sık Yapılan Hatalar

Konular

Bu konudan daha fazla soru