Examkin

Question

Difficulty: HardHigh-Performing Data Ingestion and Transformation Solutions

A media streaming platform needs to ingest user interaction telemetry during live broadcasts. The platform expects a baseline ingestion rate of 15 MB/s15\text{ MB/s}, which spikes rapidly to 120 MB/s120\text{ MB/s} during popular live events. The telemetry payloads are sent in JSON format and must be converted to Apache Parquet format and stored in Amazon S3 for long-term analytics. The solution must scale dynamically to handle the peaks without manual intervention or latency degradation, while minimizing operational overhead. Which architecture meets these requirements?

  1. Configure an Amazon Kinesis Data Stream in On-Demand mode to ingest the incoming telemetry data. Set up an Amazon Data Firehose delivery stream that references the Kinesis Data Stream as the source, uses AWS Glue to convert the JSON data to Apache Parquet, and delivers the formatted data to Amazon S3.Answer
  2. B
    Configure an Amazon Kinesis Data Stream in Provisioned mode with a fixed capacity of 2020 shards. Set up an AWS Lambda function triggered by Amazon CloudWatch alarms to programmatically call the UpdateShardCount API when the write throughput exceeds 80%80\% utilization, then deliver the data to Amazon S3 using Amazon Data Firehose.
  3. C
    Configure an Amazon SQS Standard queue to ingest the raw JSON payloads, relying on SQS to automatically scale to the peak rate while guaranteeing strict first-in, first-out sequence. Use an AWS Lambda function triggered by the queue to buffer, transform the payloads to Apache Parquet, and write the objects to Amazon S3.
  4. D
    Configure Amazon Data Firehose to ingest the data directly, using a custom AWS Lambda function to perform the JSON-to-Parquet conversion and write the output files to Amazon S3. Configure the Firehose buffer interval to 6060 seconds to minimize S3 put-object requests.

Answer

Configure an Amazon Kinesis Data Stream in On-Demand capacity mode linked to an Amazon Data Firehose delivery stream that utilizes AWS Glue for serverless JSON-to-Parquet conversion before delivering the files to Amazon S3.
The correct answer combines Amazon Kinesis Data Streams in On-Demand capacity mode with Amazon Data Firehose. On-Demand mode automatically scales shard provisioning up to twice the previous peak rate to accommodate sudden bursts, avoiding ingestion throttling. Firehose then consumes from the stream and uses AWS Glue to convert the incoming JSON telemetry records into Apache Parquet format before saving them to S3, providing a serverless, low-overhead solution.

Step-by-Step Solution

1
Select the correct ingestion ingestion layer for dynamic scaling.
Amazon Kinesis Data Streams in On-Demand mode is chosen because it dynamically manages shard count allocation to handle rapid traffic spikes up to 200 MB/s200\text{ MB/s} without manual administrative overhead or scaling lag.
To avoid ProvisionedThroughputExceededExceptions during sudden live broadcast traffic peaks, capacity must scale instantly.
2
Integrate with a serverless delivery and format transformation pipeline.
Connect Kinesis Data Streams to Amazon Data Firehose. Firehose natively integrates with AWS Glue Schema Registry to perform JSON-to-Parquet conversion inline before writing to S3.
This avoids the need to write and maintain custom AWS Lambda transformation code, minimizing both latency and operational overhead.
3
Configure the storage destination.
Amazon S3 serves as the durable destination for the query-ready Parquet data.
This establishes a high-performance, cost-effective data lake storage solution.

Key Concept

Serverless high-throughput ingestion and transformation architectures using Kinesis Data Streams On-Demand and Amazon Data Firehose.
Estimated Time:2m 30s
Rate this question

Topics

High-Performing Data Ingestion and Transformation Solutions

More questions from this topic