Examkin

Question

Difficulty: MediumStream Processing and Event Routing with Amazon Kinesis and EventBridge

An online auction platform uses an Amazon Kinesis Data Stream to process real-time bidding events. The data stream has 16 shards, and an AWS Lambda function is configured to process the incoming records. Each bidding event payload includes a unique `bidder_id`, an `auction_id` (representing one of thousands of active auctions), a `category` (representing one of 8 major item categories), and a `bid_amount`. During a high-profile auction event, developers observe frequent `ProvisionedThroughputExceededException` errors, and cloud watch metrics show that only a small number of shards are receiving traffic. The current partition key is set to the `category` field. Which partition key design should the developer implement to resolve the throttling and distribute the load evenly across all shards?

  1. Configure the partition key to use the `auction_id` field to ensure a high-entropy key that distributes records uniformly across all shards.Answer
  2. B
    Configure the partition key to use a static string value for all records to guarantee strict chronological order of processing across all shards.
  3. C
    Increase the execution timeout of the consumer Lambda function to allow it to recover from Kinesis stream throttling.
  4. D
    Modify the IAM trust policy of the consumer Lambda function's execution role to allow the Kinesis service principal to assume the role.

Answer

Configure the partition key to use the auction identifier field to ensure a high-entropy key that distributes records uniformly across all shards.
The correct choice is to configure the partition key to use the auction identifier field. Kinesis Data Streams distributes incoming records across shards using an MD5 hash of the partition key. Choosing a high-entropy key with thousands of distinct values ensures a uniform distribution of records, preventing hot shards and resolving the ProvisionedThroughputExceededException errors.

Step-by-Step Solution

1
Diagnose the cause of the ProvisionedThroughputExceededException errors and the uneven shard usage.
The current partition key is category, which only has 8 distinct values. Since there are 16 shards, at least half of the shards will receive no data based on this key, and popular categories will cause hot shards.
Identifying the mismatch between key cardinality and shard count explains the resource bottleneck.
2
Select a high-entropy field from the event payload to serve as the new partition key.
The auction identifier has thousands of unique active values.
A high-entropy key ensures that the MD5 hashing algorithm distributes the data uniformly across all 16 shards.
3
Modify the producer application to utilize the selected high-entropy field.
The workload is balanced across all shards, eliminating the hot shard issue and resolving the throttling errors.
Applying the high-entropy key on the producer side is the standard pattern to resolve partition key hot spots in Kinesis.

Key Concept

Partition Key Entropy in Amazon Kinesis Data Streams
Estimated Time:1m 30s
Rate this question

Topics

Stream Processing and Event Routing with Amazon Kinesis and EventBridge

More questions from this topic