A company is designing a high-performance analytics ingestion pipeline for mobile applicat… — AWS Certified Solutions Architect - Associate

A company is designing a high-performance analytics ingestion pipeline for mobile application clickstream events. During peak hours, the application generates up to $50,000$ events per second, with an average event size of $1.5\text{ KB}$ . The events must be ingested with minimal latency, transformed from JSON to Apache Parquet format, partitioned by date and hour, and stored in Amazon S3 for query analysis using Amazon Athena. The solution must scale automatically to handle peak traffic spikes without manual intervention, ensure no data loss, and maintain an end-to-end delivery latency of under $5$ minutes. Which two steps should the solutions architect recommend to implement this architecture?

Ingest the incoming clickstream events using Amazon Kinesis Data Streams configured in On-Demand mode to automatically scale shard capacity in response to the peak throughput.Answer
Configure Amazon Kinesis Data Firehose with the Kinesis Data Stream as the source, enable format conversion to Apache Parquet using an AWS Glue schema, and set the buffer interval to $300$ seconds before writing to Amazon S3.Answer
C
Ingest the incoming clickstream events using an Amazon SQS Standard queue, and configure Amazon Kinesis Data Firehose to pull events from the queue to write them directly to Amazon S3.
D
Deploy a provisioned Amazon Kinesis Data Stream with $15$ shards, and enable partition keys based on device ID to distribute the traffic across the shards.
E
Ingest the incoming clickstream events into an Amazon SQS FIFO queue with high-throughput mode enabled, and trigger an AWS Lambda function to batch-convert the records to Parquet before writing to Amazon S3.

Answer

To build this high-performance ingestion and transformation architecture, the solutions architect should ingest the clickstream events using Amazon Kinesis Data Streams in On-Demand mode and configure Amazon Kinesis Data Firehose with the stream as the source to convert JSON to Parquet using AWS Glue with a

300

-second buffer interval.

The correct solution uses Amazon Kinesis Data Streams in On-Demand mode to handle the ingestion. Since peak throughput is

50,000

events per second (

75\text{ MB/s}

), On-Demand mode is perfect as it automatically scales up to

200\text{ MB/s}

and

200,000

records per second. Amazon Kinesis Data Firehose can then be configured with the Kinesis Data Stream as its source, allowing it to natively convert the JSON payloads into Apache Parquet format using a schema from the AWS Glue Data Catalog. Setting the buffer interval to

300

seconds guarantees that data is written to S3 within the

5

-minute latency requirement.

Step-by-Step Solution

Calculate the peak throughput requirement.

Peak traffic of

50,000

events per second at

1.5\text{ KB}

per event results in a write throughput of

75\text{ MB/s}

Sizing the ingestion mechanism requires determining both the event rate and the aggregate bandwidth.

Evaluate Kinesis Data Streams capacity modes.

On-Demand mode is selected because it automatically scales up to

200\text{ MB/s}

and

200,000

records per second, handling the

50,000

records/sec peak without manual shard provisioning.

Provisioned mode with static shard allocations (like

15

shards) would throttle the stream, as

15

shards support only

15,000

records/sec.

Determine the transformation and destination mechanism.

Amazon Kinesis Data Firehose is configured to read directly from Kinesis Data Streams, transforming JSON payloads into Apache Parquet format using a schema from the AWS Glue Data Catalog before writing to Amazon S3.

Firehose provides serverless, inline format conversion, which eliminates the need to run custom compute tasks for data format transformation.

Configure the buffering configurations for the destination.

The buffer interval is configured to

300

seconds.

This configuration ensures that data is written to S3 within the required

5

-minute latency window while maximizing compression ratio.

Key Concept

High-throughput data ingestion using Kinesis Data Streams in On-Demand mode combined with serverless format transformation via Kinesis Data Firehose and AWS Glue.

Estimated Time:3m 0s

Question

Answer

Step-by-Step Solution

Key Concept

Common Mistakes

Topics

More questions from this topic