Examkin

Soru

Zorluk: OrtaHigh-Performing Data Ingestion and Transformation Solutions

A financial analytics platform is designing an ingestion pipeline to capture real-time stock ticker updates from external market feeds. During periods of high volatility, the stream is expected to peak at 12,00012,000 updates per second, with an average update payload size of 1.5 KB1.5\text{ KB}. The solutions architect must design a solution that ingests this data with sub-second latency, preserves the strict chronological order of updates for each individual stock ticker symbol, and writes the processed updates to an Amazon DynamoDB table. Which ingestion architecture should the solutions architect choose to support the peak load?

  1. Configure an Amazon Kinesis data stream with 1818 shards in Provisioned mode, using the stock ticker symbol as the partition key. Use an AWS Lambda function to consume records from the stream and write them to the DynamoDB table.Cevap
  2. B
    Configure an Amazon Kinesis data stream with 1212 shards in Provisioned mode, using the stock ticker symbol as the partition key. Use an AWS Lambda function to consume records from the stream and write them to the DynamoDB table.
  3. C
    Configure a standard Amazon SQS queue to capture the incoming updates. Use an AWS Lambda function to poll the queue, sort the updates by stock ticker symbol, and write them to the DynamoDB table.
  4. D
    Configure the market feeds to write updates directly to the Amazon DynamoDB table using the AWS SDK, utilizing the update timestamp as the partition key.

Cevap

The architecture using an Amazon Kinesis data stream with 1818 shards in Provisioned mode, with the stock ticker symbol as the partition key, and consumed by an AWS Lambda function to write to DynamoDB.
The correct solution calculates the shard count by evaluating both the record write limit (1,0001,000 records/sec per shard) and the throughput write limit (1 MB/sec1\text{ MB/sec} per shard). To support 12,00012,000 records/sec at 1.5 KB1.5\text{ KB} each (18 MB/sec18\text{ MB/sec} total), a minimum of 1818 shards is required. Furthermore, using the stock ticker symbol as the partition key ensures that all updates for a given ticker are routed to the same shard and processed sequentially, maintaining order.

Adım Adım Çözüm

1
Calculate the total ingestion write throughput.
12,000 updates/sec×1.5 KB/update=18,000 KB/sec=18 MB/sec12,000\text{ updates/sec} \times 1.5\text{ KB/update} = 18,000\text{ KB/sec} = 18\text{ MB/sec}.
Understanding total payload volume is necessary to determine the bandwidth requirements of the ingestion layer.
2
Determine the Kinesis Data Streams shard count required based on both throughput and record limits.
Based on the 1,0001,000 records/sec limit: 12,000/1,000=1212,000 / 1,000 = 12 shards. Based on the 1 MB/sec1\text{ MB/sec} write throughput limit: 18 MB/sec/1 MB/sec=1818\text{ MB/sec} / 1\text{ MB/sec} = 18 shards. The minimum required is the maximum of the two calculations, which is 1818 shards.
Kinesis shards have independent limits for write throughput (1 MB/sec1\text{ MB/sec}) and write record count (1,0001,000 records/sec). Failing to scale for both limits causes ingestion throttling.
3
Select a partition key that ensures correct message ordering.
Using the stock ticker symbol as the partition key ensures all updates for a single ticker are mapped to the same shard and processed sequentially.
Kinesis preserves order within a single shard. Routing all records with the same partition key to the same shard guarantees their chronological order is maintained.

Anahtar Kavram

Kinesis Data Streams shard capacity calculations and partition key design for message ordering.
Tahmini Süre:1m 30s
Bu soruyu puanla

Konular

High-Performing Data Ingestion and Transformation Solutions

Bu konudan daha fazla soru