A media streaming company is designing a high-performing data ingestion and transformation… — AWS Certified Solutions Architect - Associate

A media streaming company is designing a high-performing data ingestion and transformation pipeline to process user clickstream events from their mobile application. During live events, clickstream traffic experiences sudden, unpredictable spikes up to $8,000$ events per second, with an average event size of $1.5\text{ KB}$ . The processed data must be stored in Amazon S3 in Apache Parquet format for downstream analytics. The company wants to minimize administrative overhead and ensure that the pipeline scales automatically without manual intervention.

Which combination of actions should a solutions architect recommend to meet these requirements? (Select two.)

Use Amazon Kinesis Data Streams in On-Demand capacity mode to ingest the clickstream events from the mobile application.Answer
Configure Amazon Data Firehose with the Kinesis Data Stream as the source, and enable format conversion to Apache Parquet using an AWS Glue schema before delivering the data to Amazon S3.Answer
C
Use Amazon Kinesis Data Streams in Provisioned capacity mode with 5 shards to ingest the incoming clickstream data.
D
Ingest the clickstream events using an Amazon SQS standard queue and deploy a fleet of Amazon EC2 instances to consume, order, and transform the events.
E
Configure Amazon Data Firehose to ingest data directly from the application and use a custom AWS Lambda function to perform the conversion to Apache Parquet.

Answer

The solutions architect should recommend ingesting the clickstream events using Amazon Kinesis Data Streams in On-Demand capacity mode, and then using Amazon Data Firehose with AWS Glue schema integration to convert the incoming JSON data to Apache Parquet format before delivering it to Amazon S3.

The correct solution uses Amazon Kinesis Data Streams in On-Demand capacity mode to ingest clickstream traffic, which automatically scales up to

200\text{ MB/sec}

to accommodate the calculated

12\text{ MB/sec}

peak traffic without manual shard calculations. It then uses Amazon Data Firehose with AWS Glue schema integration to natively transform the incoming JSON clickstream data into Apache Parquet format before writing it to Amazon S3. This architecture meets the requirements of auto-scaling, low administrative overhead, and high-performance ingestion and transformation.

Step-by-Step Solution

Calculate the peak throughput requirement for the ingestion layer.

Multiply

8,000\text{ events/sec}

1.5\text{ KB/event}

to get

12,000\text{ KB/sec}

, which is equivalent to

12\text{ MB/sec}

of write throughput.

This establishes the minimum ingestion capacity that the solution must handle during peak traffic spikes.

Compare scaling options for the Kinesis ingestion layer.

Kinesis Data Streams in On-Demand mode automatically scales to handle up to

200\text{ MB/sec}

write throughput. A provisioned capacity stream with 5 shards only handles up to

5\text{ MB/sec}

Choosing On-Demand capacity mode ensures that the system handles unpredictable traffic spikes automatically without administrative overhead or throttling.

Identify the low-overhead conversion mechanism from JSON to Parquet.

Amazon Data Firehose can pull from the Kinesis Data Stream and convert JSON data to Parquet using an AWS Glue schema before writing to Amazon S3.

This avoids writing custom application code (such as in Lambda or on EC2) and minimizes administrative overhead.

Key Concept

High-performing data ingestion and serverless transformation using Kinesis Data Streams On-Demand capacity mode combined with Amazon Data Firehose and AWS Glue schema conversion.

Question

Answer

Step-by-Step Solution

Key Concept

Common Mistakes

Topics

More questions from this topic