Examkin

Soru

Zorluk: OrtaHigh-Performing Data Ingestion and Transformation Solutions

A logistics company is designing an ingestion pipeline for real-time tracking data from a fleet of 50,00050,000 delivery vehicles. Each vehicle transmits a 1.5 KB1.5\text{ KB} JSON location update every 5 seconds5\text{ seconds}. The company has two primary requirements: real-time route optimization calculations with a latency of less than one second, and storing the data in Amazon S3 in Apache Parquet format for long-term historical query analysis using Amazon Athena. Which combination of actions will meet these requirements with the highest performance? (Select two.)

  1. Configure an Amazon Kinesis Data Stream with at least 15 shards to ingest the tracking data, and use a real-time consuming application to process the stream for route optimization.Cevap
  2. Configure an Amazon Kinesis Data Firehose delivery stream with the Kinesis Data Stream as the source to convert the JSON payloads to Apache Parquet using AWS Glue, and deliver the data to Amazon S3.Cevap
  3. C
    Ingest the tracking data using a standard Amazon SQS queue, and configure Amazon Kinesis Data Firehose to poll the queue to deliver data directly to the route optimization application.
  4. D
    Configure an Amazon Kinesis Data Stream with 5 provisioned shards, using the vehicle's unique device ID as the partition key, to stream data to the route optimization application.
  5. E
    Use a standalone Amazon Kinesis Data Firehose delivery stream with a buffer interval of 1 second to write the incoming data directly to Amazon S3, and have the route optimization application read from S3.

Cevap

Configure an Amazon Kinesis Data Stream with at least 15 shards to ingest the tracking data for real-time processing, and configure an Amazon Kinesis Data Firehose delivery stream with the Kinesis Data Stream as the source to convert the data to Apache Parquet and deliver it to Amazon S3.
To process the ingestion of 10,00010,000 records/sec at 15 MB/sec15\text{ MB/sec}, Amazon Kinesis Data Streams is provisioned with at least 15 shards because each shard supports up to 1 MB/sec1\text{ MB/sec} write throughput or 1,0001,000 records/sec. A real-time consumer application processes the stream directly to achieve sub-second latency. To satisfy the storage and query optimization requirements, Kinesis Data Firehose is configured to read from the Kinesis Data Stream, convert the format to Apache Parquet using AWS Glue, and write the records to Amazon S3.

Adım Adım Çözüm

1
Calculate the required ingestion throughput and record rate to properly size the streaming service.
The fleet generates 10,00010,000 records per second (50,00050,000 vehicles / 55 seconds) and a total write throughput of 15 MB/sec15\text{ MB/sec} (10,000×1.5 KB10,000 \times 1.5\text{ KB}).
Correct sizing is necessary to avoid ingestion bottlenecks and scale limits.
2
Select the ingestion service that supports sub-second latency and multiple consumers.
Amazon Kinesis Data Streams is chosen with at least 15 shards, as each shard supports up to 1 MB/sec1\text{ MB/sec} or 1,0001,000 records/sec write throughput.
This meets the sub-second route optimization calculation latency requirement.
3
Select the component for long-term storage format conversion and S3 delivery.
Amazon Kinesis Data Firehose delivery stream is configured to consume from the Kinesis Data Stream, use AWS Glue Data Catalog for format conversion to Parquet, and write to S3.
This provides serverless, performant transformation and delivery without impacting the real-time consumer.

Anahtar Kavram

Designing high-performing real-time data ingestion pipelines with Kinesis Data Streams and Firehose, calculating shard requirements, and performing schema-based conversions.
Tahmini Süre:1m 30s
Bu soruyu puanla

Konular

High-Performing Data Ingestion and Transformation Solutions

Bu konudan daha fazla soru