Examkin

Question

Difficulty: EasyHigh-Performing Data Ingestion and Transformation Solutions

A company needs to ingest 3 MB/s3\text{ MB/s} of transaction records from an e-commerce website. The records must be processed in the exact order they are received to maintain inventory accuracy, and then saved to Amazon S3. Which AWS service configuration provides the most suitable ingestion pipeline with the least operational overhead?

  1. Amazon Kinesis Data Streams in On-Demand mode to ingest the records using partition keys, and Amazon Data Firehose to deliver the data to Amazon S3.Answer
  2. B
    An Amazon SQS standard queue to ingest the records, and an AWS Lambda function to process and write the data to Amazon S3.
  3. C
    Amazon Kinesis Data Streams with a single provisioned shard to ingest the records, and Amazon Data Firehose to deliver the data to Amazon S3.
  4. D
    Amazon Kinesis Data Streams to ingest the records, and an AWS Lambda function running a continuous loop to poll the stream and write the data to Amazon S3.

Answer

Amazon Kinesis Data Streams in On-Demand mode to ingest the records using partition keys, and Amazon Data Firehose to deliver the data to Amazon S3.
The correct answer combines Amazon Kinesis Data Streams in On-Demand mode with Amazon Data Firehose. Kinesis Data Streams in On-Demand mode automatically scales to accommodate the 3 MB/s3\text{ MB/s} data stream without manual intervention, while using partition keys ensures records are processed chronologically within the shard. Amazon Data Firehose handles data batching and delivery directly to Amazon S3 with minimal operational overhead.

Step-by-Step Solution

1
Determine throughput requirements and choose an appropriate stream capacity configuration.
The ingestion rate is 3 MB/s3\text{ MB/s}. Since a single Kinesis shard supports only 1 MB/s1\text{ MB/s} of write capacity, multiple shards are required. Selecting Kinesis Data Streams in On-Demand mode automatically handles scaling and dynamically allocates the required capacity without manual shard management.
To prevent throttling and eliminate operational overhead associated with manually scaling shards.
2
Select a mechanism to guarantee strict chronological message ordering.
Using a partition key (such as the transaction ID) ensures that records for the same transaction are routed to the same Kinesis shard and processed in the exact order they were received.
To satisfy the business requirement of maintaining inventory accuracy through chronological record processing.
3
Select the most efficient and lowest-overhead service to write the streaming data to Amazon S3.
Amazon Data Firehose is integrated natively with Amazon Kinesis Data Streams and automatically batches, compresses, and delivers data to Amazon S3 with zero server administration.
To minimize operational overhead as required by the business.

Key Concept

Selecting high-performance, serverless data ingestion solutions with strict ordering requirements.
Rate this question

Topics

High-Performing Data Ingestion and Transformation Solutions

More questions from this topic