Examkin

Question

Difficulty: MediumHigh-Performing Data Ingestion and Transformation Solutions

A fintech company is designing a high-performing data ingestion and transformation pipeline for a mobile payment application to process real-time transaction logs. The pipeline must ingest a steady volume of 2 MB/s2\text{ MB/s} of JSON data, which can spike up to 10 MB/s10\text{ MB/s} during promotional events. The logs must be converted into Apache Parquet format and stored in Amazon S3 for near-real-time business intelligence queries. The architecture must minimize operational management and scale automatically to handle traffic spikes. Which ingestion and transformation solution should the company implement?

  1. A
    Send the transaction logs to an Amazon SQS Standard queue, configure an AWS Lambda function to poll the queue, perform the Parquet conversion, and write the files to Amazon S3.
  2. B
    Ingest the logs using Amazon Kinesis Data Streams with a provisioned capacity of 55 shards, trigger an AWS Lambda function to convert the records to Parquet, and write the results to Amazon S3.
  3. Use Amazon Data Firehose to receive the transaction logs, configure native data format conversion using an AWS Glue schema to output Apache Parquet, and deliver the transformed logs directly to Amazon S3.Answer
  4. D
    Ingest the logs into Amazon Kinesis Data Streams, and configure an AWS Lambda function to continuously poll the stream, convert the payloads to Parquet, and write to Amazon S3.

Answer

Use Amazon Data Firehose to receive the transaction logs, configure native data format conversion using an AWS Glue schema to output Apache Parquet, and deliver the transformed logs directly to Amazon S3.
The correct solution uses Amazon Data Firehose to automatically scale up to 10 MB/s10\text{ MB/s} to handle promotional traffic spikes. It leverages Firehose's native integration with AWS Glue to convert JSON records into Apache Parquet format directly before writing them to Amazon S3. This eliminates the operational complexity of managing shards or writing custom ETL code in AWS Lambda.

Step-by-Step Solution

1
Analyze scale and ingestion requirements.
The pipeline must handle steady ingestion at 2 MB/s2\text{ MB/s} and spikes up to 10 MB/s10\text{ MB/s} with minimal operational management.
This establishes that the solution must scale dynamically without manual intervention.
2
Evaluate data transformation overhead.
Converting JSON data to Apache Parquet can be done natively by Amazon Data Firehose using an AWS Glue schema, eliminating the need for custom transformation code in AWS Lambda.
Choosing native features minimizes custom code maintenance, CPU usage charges, and operational overhead.
3
Select the optimal architecture.
Amazon Data Firehose combined with AWS Glue for schema definition and Amazon S3 for destination is the most performant, self-scaling, and operationally simple design.
It addresses ingestion, transformation, and storage without provisioning shards or maintaining polling code.

Key Concept

Serverless real-time data ingestion and native schema-based format transformation using Amazon Data Firehose and AWS Glue.
Estimated Time:1m 30s
Rate this question

Topics

High-Performing Data Ingestion and Transformation Solutions

More questions from this topic