A media streaming platform needs to design a high-performing ingestion and transformation … — AWS Certified Solutions Architect - Associate

A media streaming platform needs to design a high-performing ingestion and transformation pipeline to process clickstream data generated by its web application. The platform receives user interaction events (play, pause, seek) at an average rate of $5\text{ MB/s}$ , peaking at $25\text{ MB/s}$ during major content releases.

The pipeline must satisfy two key requirements:
1. Deliver clickstream data with sub-second latency to a real-time personalization engine.
2. Store the clickstream data in Amazon S3 in Apache Parquet format, partitioned by date, with a maximum latency of 5 minutes for historical analytics.

Which combination of actions should a Solutions Architect recommend to build the most performant and scalable solution? (Select two.)

Configure an Amazon Kinesis Data Stream in on-demand capacity mode to ingest the clickstream events, and configure the real-time personalization engine as a consumer of the stream.Cevap
Configure an Amazon Kinesis Data Firehose delivery stream to read from the Kinesis Data Stream, buffer the incoming data for $300\text{ seconds}$ , use an AWS Glue schema to convert the JSON events to Apache Parquet, and write the files to Amazon S3.Cevap
C
Ingest the clickstream events using an Amazon SQS standard queue, and configure both the personalization engine and Amazon Kinesis Data Firehose to pull messages simultaneously from the queue.
D
Configure an Amazon Kinesis Data Stream in provisioned capacity mode with $2\text{ shards}$ to ingest the clickstream events, and use an AWS Glue streaming ETL job to write data to both the personalization engine and Amazon S3.
E
Configure an Amazon Kinesis Data Firehose delivery stream with a buffering interval of $0\text{ seconds}$ to ingest the clickstream, and write the data directly to both the personalization engine and Amazon S3.

Cevap

To build a performant and scalable solution, configure an Amazon Kinesis Data Stream in on-demand capacity mode to ingest the clickstream events and serve the personalization engine, and configure an Amazon Kinesis Data Firehose delivery stream to read from the stream, buffer the data for

300\text{ seconds}

, convert it to Apache Parquet using AWS Glue, and deliver it to Amazon S3.

The correct solution involves ingesting the clickstream data into an Amazon Kinesis Data Stream in on-demand capacity mode and utilizing Amazon Kinesis Data Firehose to transform and load the data into Amazon S3. Kinesis Data Streams supports multiple consumers and provides sub-second latency for direct consumers like the personalization engine. Configuring Kinesis Data Streams in on-demand capacity mode allows it to scale automatically up to the peak write throughput of

25\text{ MB/s}

without manual provisioning. For downstream storage, Kinesis Data Firehose can consume from the same Kinesis Data Stream, buffer the records for

300\text{ seconds}

to meet the latency requirement, convert the JSON payloads into Apache Parquet format using a schema from AWS Glue, and write the partitioned files to Amazon S3.

Adım Adım Çözüm

Analyze ingestion capacity and latency requirements.

Identify that a peak ingestion rate of

25\text{ MB/s}

requires dynamic scaling, and the sub-second personalization engine requires direct stream access, while the historical analytics can accept a latency of up to 5 minutes (

300\text{ seconds}

Establishing clear bounds for throughput and latency guides the selection of Kinesis Data Streams for sub-second ingestion and Kinesis Data Firehose for batched S3 delivery.

Select the ingestion mechanism for sub-second consumption and scaling.

Deploy Amazon Kinesis Data Streams in on-demand capacity mode.

On-demand capacity mode automatically scales to handle up to

200\text{ MB/s}

(well above the

25\text{ MB/s}

peak), avoiding ProvisionedThroughputExceededException errors and allowing the personalization engine to read data immediately (sub-second latency).

Select the downstream delivery and format transformation mechanism.

Deploy Amazon Kinesis Data Firehose pointing to the Kinesis Data Stream as a source, with format conversion enabled via AWS Glue and a

300\text{ seconds}

buffer interval, writing to Amazon S3.

Kinesis Data Firehose natively integrates with Kinesis Data Streams to pull data, buffers it to meet the 5-minute requirement, transforms JSON to Parquet using AWS Glue schemas, and writes the output directly to S3 without custom processing code.

Anahtar Kavram

Leveraging Kinesis Data Streams for real-time, multi-consumer data ingestion alongside Kinesis Data Firehose for managed ETL and buffering into Amazon S3.

Soru

Cevap

Adım Adım Çözüm

Anahtar Kavram

Sık Yapılan Hatalar

Konular

Bu konudan daha fazla soru