Examkin

Soru

Zorluk: OrtaHigh-Performing Data Ingestion and Transformation Solutions

A retail company expects a massive surge in website traffic during an upcoming marketing campaign. The company needs to design a high-performing data ingestion solution to collect clickstream data in JSON format, transform it into Apache Parquet format, and store it in Amazon S3 for near-real-time analytics. The volume of incoming data is highly unpredictable and fluctuates rapidly. The solution must scale automatically to handle peak throughput without any manual intervention or administrative overhead. Which solution meets these requirements with the highest performance and the least operational effort?

  1. Deploy Amazon Kinesis Data Firehose to ingest the clickstream data, configure it to invoke an AWS Lambda function for inline data transformation, and set the destination to Amazon S3.Cevap
  2. B
    Create an Amazon Kinesis Data Stream with a fixed pool of provisioned shards, and deploy a custom consumer script running on Amazon EC2 instances to read, transform, and write the data to Amazon S3.
  3. C
    Publish the incoming clickstream events to a standard Amazon SQS queue, and trigger an AWS Lambda function to process the messages, transform the payload, and write them sequentially to Amazon S3.
  4. D
    Configure the web application to write raw log files to Amazon S3, and trigger an AWS Lambda function running continuously on a scheduled basis to poll, transform, and rewrite the files to Parquet format.

Cevap

Deploy Amazon Kinesis Data Firehose to ingest the clickstream data, configure it to invoke an AWS Lambda function for inline data transformation, and set the destination to Amazon S3.
The correct option is to use Amazon Kinesis Data Firehose with inline AWS Lambda transformation. Kinesis Data Firehose is a fully managed streaming service that scales automatically to accommodate unpredictable traffic spikes without manual intervention. It offers native integration with AWS Lambda to perform serverless, real-time data transformations (such as converting JSON to Parquet) before delivering the data directly to Amazon S3, maximizing ingestion performance while minimizing operational overhead.

Adım Adım Çözüm

1
Analyze the requirements for data ingestion scale and management overhead.
The ingestion system must handle highly unpredictable peak volume automatically with the least operational effort.
This rules out solutions requiring manual shard provisioning or server management.
2
Evaluate the transformation and destination requirements.
Data must be transformed from JSON to Parquet and delivered to Amazon S3 in near-real-time.
Kinesis Data Firehose provides native integration with AWS Lambda for inline transformation and outputs directly to Amazon S3.
3
Compare scaling and integration features of Kinesis Data Streams, SQS, and Firehose.
Kinesis Data Firehose scales automatically without shard provisioning, maintains streaming ingestion, and integrates with Lambda to output Parquet, making it the most performant and low-overhead choice.
It avoids Kinesis Data Streams shard limit throttling, SQS ordering issues, and Lambda continuous running inefficiencies.

Anahtar Kavram

High-performing streaming data ingestion with automatic scaling and inline serverless transformation using Kinesis Data Firehose and AWS Lambda.
Tahmini Süre:1m 30s
Bu soruyu puanla

Konular

High-Performing Data Ingestion and Transformation Solutions

Bu konudan daha fazla soru