A media streaming platform needs to design a high-performing ingestion pipeline to process… — AWS Certified Solutions Architect - Associate

A media streaming platform needs to design a high-performing ingestion pipeline to process video playback quality telemetry from $10 \text{ million}$ active players. During peak hours, the system receives $150{}000$ events per second, with an average event payload size of $2 \text{ KB}$ . The platform requires that buffering events be analyzed within $5 \text{ seconds}$ to trigger adaptive bitrate changes, while all raw telemetry must be transformed into Apache Parquet format and archived in Amazon S3 within $5 \text{ minutes}$ for daily analytical processing. Which architecture meets these requirements with the highest performance and the lowest operational overhead?

A
Configure Amazon Data Firehose as the single ingestion endpoint. Use an AWS Lambda function for inline transformation to analyze the buffering events and write alert records to Amazon DynamoDB, while buffering and transforming the raw telemetry to Apache Parquet before delivering it to Amazon S3.
Configure Amazon Kinesis Data Streams in Provisioned mode with $300$ shards. Deploy an Amazon Managed Service for Apache Flink application to perform real-time analysis of buffering events from the stream. Configure Amazon Data Firehose to consume from the same Kinesis stream to transform the telemetry to Apache Parquet format and write it to Amazon S3.Cevap
C
Configure Amazon Kinesis Data Streams in On-Demand mode. Deploy an Amazon Managed Service for Apache Flink application to perform real-time analysis of buffering events from the stream. Configure Amazon Data Firehose to consume from the stream to transform the telemetry to Apache Parquet format and write it to Amazon S3.
D
Configure an Amazon SQS Standard queue to ingest the telemetry events. Use an AWS Lambda function to poll the queue and perform real-time analysis within $5 \text{ seconds}$ . Configure Amazon Data Firehose to poll the same SQS queue to transform the telemetry to Apache Parquet format and write it to Amazon S3.

Cevap

Configure Amazon Kinesis Data Streams in Provisioned mode with

300

shards, deploy an Amazon Managed Service for Apache Flink application for real-time analysis, and use Amazon Data Firehose to consume from the same stream, transform the data to Parquet, and write it to Amazon S3.

Designing the ingestion layer with Amazon Kinesis Data Streams in Provisioned mode with

300

shards satisfies the throughput requirement. At

150{}000

events/second and

2 \text{ KB}

per event, the peak ingest throughput is

300 \text{ MB/s}

. Since a single shard supports up to

1 \text{ MB/s}

write throughput,

300

shards are needed. Using Amazon Managed Service for Apache Flink enables real-time stream analysis within the

5

-second latency requirement. Amazon Data Firehose can consume from the stream asynchronously to buffer, transform to Parquet, and write to Amazon S3, meeting the

5

-minute archival requirement.

Adım Adım Çözüm

Calculate the peak write throughput requirement.

Peak write throughput is

300 \text{ MB/s}

(derived from

150{}000 \text{ events/sec} \times 2 \text{ KB/event} = 300{}000 \text{ KB/s} = 300 \text{ MB/s}

Knowing the peak throughput is necessary to select the correct ingestion service and capacity configuration.

Determine the stream capacity mode and scaling requirements.

A Kinesis stream with

300

provisioned shards is required because Kinesis Data Streams On-Demand has a default capacity limit of

200 \text{ MB/s}

write throughput. A single shard supports up to

1 \text{ MB/s}

write throughput, meaning

300

shards are needed to support

300 \text{ MB/s}

This ensures the stream can handle peak traffic without throttling or experiencing ProvisionedThroughputExceededException errors.

Evaluate the real-time processing and delivery components.

Amazon Managed Service for Apache Flink provides sub-

5

-second processing, and Amazon Data Firehose handles Parquet transformation and S3 delivery.

This meets the dual latency requirements of real-time analysis (under

5 \text{ seconds}

) and batch archival (under

5 \text{ minutes}

Anahtar Kavram

High-performing real-time data ingestion scaling limits and streaming analytics architectures

Soru

Cevap

Adım Adım Çözüm

Anahtar Kavram

Sık Yapılan Hatalar

Konular

Bu konudan daha fazla soru