A financial services company is building a real-time fraud detection and transaction audit… — AWS Certified Solutions Architect - Associate

A financial services company is building a real-time fraud detection and transaction auditing system. The system receives clickstream data from millions of mobile devices. During normal business hours, the data ingestion rate averages $15\text{ MB/s}$ , but it peaks at $45\text{ MB/s}$ . The average size of each clickstream payload is $1\text{ KB}$ . The solutions architect must design a system that satisfies two requirements: first, a fraud-detection application must consume the data with sub-second latency and process user clickstreams in strict sequential order based on the user session ID; second, the raw data must be transformed into Apache Parquet format and stored in Amazon S3 within $5\text{ minutes}$ of ingestion for downstream batch analysis. Which architecture meets these performance and ordering requirements most cost-effectively?

Ingest the data using Amazon Kinesis Data Streams configured with $45$ shards, using the user session ID as the partition key. Configure the fraud-detection application to consume records from the stream in real time. Create an Amazon Data Firehose delivery stream that reads from the same Kinesis data stream, transforms the incoming data into Parquet using an inline AWS Lambda function, and writes the output directly to Amazon S3 Standard.Answer
B
Ingest the data using an Amazon SQS Standard queue. Configure the fraud-detection application as a consumer of the queue to process messages with sub-second latency. Set up an Amazon Data Firehose delivery stream to read from the queue, transform the data into Parquet format using an AWS Lambda function, and write the output to Amazon S3 Standard.
C
Ingest the data using Amazon Kinesis Data Streams configured with $15$ shards, using the user session ID as the partition key. Configure the fraud-detection application to consume records from the stream. Create an Amazon Data Firehose delivery stream that reads from the Kinesis data stream, transforms the data into Parquet using an inline AWS Lambda function, and writes the output to Amazon S3 Standard.
D
Ingest the data using Amazon Kinesis Data Streams configured with $45$ shards, using the user session ID as the partition key. Configure the fraud-detection application to consume records from the stream. Create an Amazon Data Firehose delivery stream that reads from the Kinesis data stream, transforms the data into Parquet using an inline AWS Lambda function, and writes the output to Amazon S3 Standard-Infrequent Access (S3 Standard-IA). Configure an S3 Lifecycle policy to delete the files after $15\text{ days}$ .

Answer

The architecture that uses Amazon Kinesis Data Streams with

45

shards (partitioned by user session ID) for real-time consumption, along with Amazon Data Firehose writing to Amazon S3 Standard with an inline AWS Lambda function for Parquet transformation.

The correct option is correct because the peak write throughput of

45\text{ MB/s}

and

45,000\text{ records/s}

requires at least

45

shards in Amazon Kinesis Data Streams to prevent write throttling. Using the user session ID as the partition key guarantees that all records for a session land in the same shard and are consumed in sequential order, meeting the ordering requirement. Amazon Data Firehose can read from the stream to buffer, transform, and write data to Amazon S3 Standard within the required

5\text{ minutes}

. Using S3 Standard avoids the minimum storage billing duration constraints associated with S3 Standard-IA.

Step-by-Step Solution

Calculate the minimum number of Kinesis Data Streams shards required for peak ingestion throughput.

Peak throughput of

45\text{ MB/s}

and

45,000\text{ records/s}

requires

\max\left(\frac{45\text{ MB/s}}{1\text{ MB/s/shard}}, \frac{45,000\text{ records/s}}{1,000\text{ records/s/shard}}\right) = 45

shards.

Each shard supports up to

1\text{ MB/s}

write throughput or

1,000

write records per second.

Identify the mechanism for maintaining user clickstream ordering.

Using the user session ID as the partition key in Kinesis Data Streams.

This ensures all events for a given session are mapped to the same shard and processed sequentially.

Select the data delivery and transformation path for batch analysis.

Amazon Data Firehose reads from the stream, transforms data to Parquet using AWS Lambda, and stores it in Amazon S3 Standard.

Firehose simplifies the buffering, transformation, and ingestion to S3 within the 5-minute requirement, and S3 Standard avoids the 30-day minimum duration charge of S3 Standard-IA.

Key Concept

High-performing data ingestion and transformation involving Kinesis Data Streams, Data Firehose, and S3 storage tiering.

Question

Answer

Step-by-Step Solution

Key Concept

Common Mistakes

Topics

More questions from this topic