Examkin

Question

Difficulty: MediumStream Processing and Event Routing with Amazon Kinesis and EventBridge

An API gateway service writes usage logs for a multi-tenant software application. Each log entry is a JSON object containing `tenant_id`, `request_id`, `timestamp`, and `response_time`. A developer configures Amazon Kinesis Data Streams to ingest these logs for real-time usage billing calculations. During peak hours, the stream encounters `ProvisionedThroughputExceededException` errors on a single shard, even though the total write throughput is well below the overall provisioned capacity of the stream. The developer discovers that the partition key is set to the tenant's geographic region, which has only three possible values. Which partition key strategy should the developer implement to resolve this issue and distribute the workload evenly across all shards?

  1. Use the request_id from each log entry as the partition key.Answer
  2. B
    Set a static string such as 'billing_stream' as the partition key for all records.
  3. C
    Deploy the consumer Lambda function inside a private VPC subnet without a NAT Gateway or VPC endpoint.
  4. D
    Modify the trust policy of the consumer Lambda function's IAM role to allow the Kinesis Data Stream to assume the role.

Answer

Use the request_id from each log entry as the partition key.
The correct answer is to use the request_id from each log entry as the partition key. Kinesis Data Streams uses the partition key input to an MD5 hash function to determine which shard a record is assigned to. By choosing a high-entropy key like request_id, which is unique for every transaction, records are distributed uniformly across all shards, resolving the hot shard throttling issue.

Step-by-Step Solution

1
Analyze the cause of the ProvisionedThroughputExceededException error on a single shard.
The error occurs because the write capacity of a specific shard is exceeded, which is caused by an uneven distribution of records (hot shard).
Identify why the overall stream capacity is not exceeded, but a single shard is throttled.
2
Evaluate the current partition key strategy.
The current partition key is the geographic region, which has only three possible values. This low-entropy key causes all records for a region to map to the same shard, leading to uneven distribution.
Determine how the partition key affects shard mapping in Kinesis.
3
Select a partition key with high entropy.
The request_id is highly unique for each API call, ensuring that the hash function distributes the records evenly across all available shards.
Ensure a uniform distribution of records to fully utilize the stream's capacity.

Key Concept

Kinesis partition keys and shard distribution
Estimated Time:1m 30s
Rate this question

Topics

Stream Processing and Event Routing with Amazon Kinesis and EventBridge

More questions from this topic