Examkin

Question

Difficulty: MediumHigh-Performing Data Ingestion and Transformation Solutions

A manufacturing facility has 20,00020,000 IoT sensors monitoring machine vibrations. Every 1010 seconds, each sensor generates a 5 KB5 \text{ KB} JSON payload containing health telemetry. A solutions architect needs to design a high-performing ingestion pipeline that buffers this data, converts the JSON format to Apache Parquet, and writes the transformed data to Amazon S3. The solution must handle spikes up to three times the average ingestion rate and minimize operational overhead. Which solution meets these requirements with the lowest operational overhead?

  1. A
    Configure Amazon Kinesis Data Streams in Provisioned mode with 10 shards to ingest the sensor data. Configure Amazon Data Firehose to consume data from the stream, convert the format to Apache Parquet, and write it to the Amazon S3 bucket.
  2. B
    Send the telemetry data to an Amazon SQS standard queue. Configure an AWS Lambda function to poll the queue, convert the JSON payloads to Apache Parquet, and write the output files to Amazon S3.
  3. Configure Amazon Data Firehose to receive the telemetry data directly, configure it to convert the format to Apache Parquet using an AWS Glue schema table, and specify an Amazon S3 bucket as the destination.Answer
  4. D
    Write the JSON telemetry files directly to an Amazon S3 bucket. Configure an AWS Lambda function triggered by S3 ObjectCreated events to read each file, transform the JSON format to Apache Parquet, and write the Parquet file to a destination S3 bucket.

Answer

Configure Amazon Data Firehose to receive the telemetry data directly, configure it to convert the format to Apache Parquet using an AWS Glue schema table, and specify an Amazon S3 bucket as the destination.
The correct solution uses Amazon Data Firehose to directly ingest the data and convert the JSON payloads to Apache Parquet format using an AWS Glue schema table. Amazon Data Firehose is a fully managed, serverless stream delivery service that automatically scales to handle high-throughput workloads and spikes (up to 30 MB/s30 \text{ MB/s} in this scenario) without manual shard management. Its built-in format conversion capability eliminates the need to develop, scale, and maintain custom transformation code in AWS Lambda or Amazon ECS.

Step-by-Step Solution

1
Calculate the average and peak ingestion throughput.
The average ingestion rate is 20,000 sensors×5 KB/10 seconds=10 MB/s20,000 \text{ sensors} \times 5 \text{ KB} / 10 \text{ seconds} = 10 \text{ MB/s} (at 2,0002,000 records/second). The peak ingestion rate is 3×10 MB/s=30 MB/s3 \times 10 \text{ MB/s} = 30 \text{ MB/s} (at 6,0006,000 records/second).
Determining throughput requirements is necessary to evaluate service scaling limits.
2
Evaluate the scaling capacity of the ingestion services.
A provisioned Kinesis Data Stream with 10 shards can only ingest up to 10 MB/s10 \text{ MB/s} (1 MB/s per shard), which will fail during peak spikes. Amazon Data Firehose automatically scales up to the required 30 MB/s30 \text{ MB/s} without manual shard configuration.
The ingestion tier must scale dynamically to handle spike workloads without throwing errors.
3
Identify the solution with the lowest operational overhead that performs transformation.
Amazon Data Firehose natively integrates with AWS Glue to perform serverless JSON-to-Parquet conversion, outputting directly to Amazon S3 without requiring custom EC2, ECS, or Lambda compute infrastructure.
Minimizing operational overhead involves leveraging native, managed service features instead of writing custom batching and transformation logic.

Key Concept

Serverless high-throughput data ingestion, buffering, and format transformation using Amazon Data Firehose and AWS Glue.
Rate this question

Topics

High-Performing Data Ingestion and Transformation Solutions

More questions from this topic