Examkin

Question

Difficulty: Very hardHigh-Performing Data Ingestion and Transformation Solutions

An automotive manufacturer is designing a telemetry ingestion and processing platform for its new fleet of connected vehicles. The system must ingest continuous data from 10,00010,000 vehicles, resulting in a peak write throughput of 80 MB/s80\text{ MB/s} and 100,000100,000 records per second. The telemetry data includes a sequence number and must be processed in the exact order it is received per vehicle for real-time anomaly detection. Additionally, the data must be transformed into Apache Parquet format and stored in Amazon S3 for historical analytics with a maximum delivery latency of 55 minutes. Which combination of actions should the solutions architect recommend to design a high-performing and scalable ingestion and transformation pipeline? (Select two.)

  1. Configure an Amazon Kinesis Data Stream with at least 100100 shards, using the vehicle identifier as the partition key.Answer
  2. Configure an Amazon Kinesis Data Firehose delivery stream with the Kinesis Data Stream as the source, enabling native format conversion to Apache Parquet using an AWS Glue table schema, and deliver the data to Amazon S3.Answer
  3. C
    Configure an Amazon Kinesis Data Stream with at least 8080 shards, using the vehicle identifier as the partition key.
  4. D
    Ingest the telemetry data into a standard Amazon SQS queue to decouple the ingestion layer, then use an AWS Glue streaming ETL job to process the queue and write the data to Amazon S3.
  5. E
    Configure an Amazon Kinesis Data Firehose delivery stream to ingest the data directly from the vehicles, using an inline AWS Lambda function to transform the raw payloads to Apache Parquet before writing to Amazon S3.

Answer

Configure an Amazon Kinesis Data Stream with at least 100 shards, using the vehicle identifier as the partition key, and configure an Amazon Kinesis Data Firehose delivery stream with the Kinesis Data Stream as the source, enabling native format conversion to Apache Parquet using an AWS Glue table schema to deliver the data to Amazon S3.
The correct combination configures an Amazon Kinesis Data Stream with at least 100100 shards and uses Kinesis Data Firehose for Parquet conversion. To handle 100,000100,000 records per second, the stream must have at least 100100 shards since each shard is limited to 1,0001,000 write records per second. Using the vehicle identifier as the partition key ensures strict ordering per vehicle. Kinesis Data Firehose can consume from the stream and use its native conversion capabilities with AWS Glue to convert the JSON payload into Apache Parquet before writing to Amazon S3 within the 5-minute requirement.

Step-by-Step Solution

1
Determine the required shard capacity for data ingestion based on stream limitations.
At least 100100 shards are needed.
A Kinesis Data Stream shard supports up to 1 MB/s1\text{ MB/s} or 1,0001,000 records/sec write throughput. The vehicle telemetry produces 80 MB/s80\text{ MB/s} (requiring 8080 shards) and 100,000100,000 records/sec (requiring 100100 shards). The record count is the bottleneck, so the stream must be provisioned with at least 100100 shards.
2
Configure the partition key to preserve message ordering per vehicle.
Partition key is set to the vehicle identifier.
By using the vehicle identifier as the partition key, all telemetry records from a specific vehicle are directed to the same shard, which guarantees strict in-order delivery to the real-time anomaly detection consumer.
3
Select the optimal delivery and transformation mechanism for historical storage.
Use Amazon Kinesis Data Firehose with the Kinesis Data Stream as the source, enabling native Parquet format conversion.
Kinesis Data Firehose can read directly from Kinesis Data Streams and perform native Apache Parquet conversion using a schema defined in the AWS Glue Data Catalog. This handles the transformation without custom code (like AWS Lambda) and natively buffers up to 5 minutes before delivering to Amazon S3.

Key Concept

Calculating Kinesis Data Stream shards based on both throughput and record counts while preserving key-based message ordering, and integrating with Kinesis Data Firehose for native Parquet conversion.
Estimated Time:3m 0s
Rate this question

Topics

High-Performing Data Ingestion and Transformation Solutions

More questions from this topic