Examkin

Question

Difficulty: Very hardStream Processing and Event Routing with Amazon Kinesis and EventBridge

An enterprise IoT application ingests high-frequency status updates from millions of devices into an Amazon Kinesis Data Stream. The updates are currently partitioned by `DeviceType`. A small number of popular device types account for 90%90\% of the total data volume, leading to frequent `ProvisionedThroughputExceededException` errors.

A developer must design a solution that achieves the following:
1. Resolves the stream throttling issue without unnecessary cost.
2. Archives all raw status updates to an Amazon S3 bucket.
3. Routes only the status updates containing a `severity` of `"FATAL"` to Amazon EventBridge for incident management.

Which approach meets these requirements with the lowest latency and operational overhead?

  1. A
    Modify the stream producers to use a composite partition key such as `DeviceType_DeviceID`. Attach an Amazon Kinesis Data Firehose delivery stream to archive all events to Amazon S3. Configure an AWS Lambda function with an Event Source Mapping to consume from Kinesis. Modify the Lambda execution role's trust policy to trust `kinesis.amazonaws.com` instead of `lambda.amazonaws.com` so that the stream can invoke the Lambda function to forward `"FATAL"` events to EventBridge.
  2. B
    Keep `DeviceType` as the partition key and increase the Kinesis Data Stream shard count to 50 to scale out processing. Write an AWS Lambda function to poll the stream using the AWS SDK, filter for `"FATAL"` events, and publish them to Amazon EventBridge while also writing every incoming event to the Amazon S3 bucket.
  3. Modify the stream producers to use a composite partition key such as `DeviceType_DeviceID`. Attach an Amazon Kinesis Data Firehose delivery stream directly to the Kinesis Data Stream to deliver all events to Amazon S3. Configure an AWS Lambda function with an Event Source Mapping that includes a filter pattern for `{"data": {"severity": ["FATAL"]}}` to publish only those critical events to Amazon EventBridge.Answer
  4. D
    Use a composite partition key of `DeviceType_DeviceID`. Configure Kinesis Data Firehose to archive all events to Amazon S3. Deploy an AWS Lambda function to consume from the stream, but place it inside a private VPC subnet with no NAT Gateway or VPC Endpoint to ensure maximum security, and write a script inside the Lambda function to filter and send `"FATAL"` events to EventBridge.

Answer

The correct option is the one proposing a composite partition key to resolve throttling, Kinesis Data Firehose to archive all events to S3, and a Lambda function with Event Source Mapping filters to forward FATAL events to EventBridge.
The correct answer proposes using a composite partition key (`DeviceType_DeviceID`) to resolve the hot shard issue. It uses Kinesis Data Firehose to efficiently write all events to Amazon S3 without code maintenance. It then routes only `"FATAL"` events to EventBridge by configuring an Event Source Mapping filter on the Lambda function, minimizing Lambda invocations and execution cost.

Step-by-Step Solution

1
Analyze the cause of stream throttling.
The current partition key (DeviceType) has low entropy, causing most traffic to go to a single shard (hot shard). A high-entropy key like DeviceType_DeviceID is needed to distribute the load across all shards.
Kinesis routes records to shards based on the hash of the partition key.
2
Identify the most efficient archiving strategy.
Kinesis Data Firehose can consume directly from the stream and batch-write to S3 with zero code.
This minimizes operational overhead and avoids writing custom Lambda archiving code.
3
Identify the most cost-effective routing strategy to EventBridge.
Use Lambda with Event Source Mapping filters set to severity: ['FATAL'].
Filters prevent Lambda from being invoked for non-matching records, reducing costs and code complexity.

Key Concept

Selecting high-entropy partition keys for Kinesis Data Streams and utilizing Event Source Mapping filters to cost-effectively process stream subsets.
Rate this question

Topics

Stream Processing and Event Routing with Amazon Kinesis and EventBridge

More questions from this topic