A meteorological research organization is designing a high-performing data ingestion and t… — AWS Certified Solutions Architect - Associate

A meteorological research organization is designing a high-performing data ingestion and transformation pipeline to process real-time atmospheric readings from thousands of weather balloons. During peak hours, the sensor data stream reaches a throughput of $15\text{ MiB/s}$ , with an average payload size of $2\text{ KiB}$ per reading. The organization requires that the data be ingested without loss, preserved in chronological order per balloon for accurate modeling, transformed from JSON to Apache Parquet format to optimize SQL queries, and stored in Amazon S3 within $5$ minutes of generation. Which two solutions should a solutions architect recommend to meet these requirements with the lowest operational overhead? (Select two.)

Ingest the sensor readings using an Amazon Kinesis Data Stream configured in On-Demand capacity mode.Answer
Create an Amazon Data Firehose delivery stream with the Kinesis Data Stream as the source, and enable record format conversion to Apache Parquet using AWS Glue before writing to Amazon S3.Answer
C
Ingest the sensor readings using an Amazon SQS standard queue to decouple the ingestion tier from downstream consumers.
D
Ingest the sensor readings using an Amazon Kinesis Data Stream configured in Provisioned capacity mode with $5$ shards.
E
Configure an Amazon SQS standard queue to trigger an AWS Lambda function that converts individual JSON payloads to Apache Parquet and writes them directly to Amazon S3.

Answer

Ingesting the sensor readings using an Amazon Kinesis Data Stream in On-Demand capacity mode, and using Amazon Data Firehose with the stream as a source to convert record formats to Apache Parquet using AWS Glue before writing to Amazon S3.

Ingesting the sensor data with an Amazon Kinesis Data Stream in On-Demand capacity mode ensures that the stream automatically scales to handle the peak write throughput of

15\text{ MiB/s}

without manual administrative intervention. Connecting Amazon Data Firehose to the Kinesis Data Stream allows for automatic, near-real-time delivery to Amazon S3. Enabling record format conversion in the Firehose delivery stream using AWS Glue allows the JSON data to be converted to Apache Parquet format on the fly with low operational overhead and minimal latency, meeting the

5

-minute storage requirement.

Step-by-Step Solution

Analyze ingestion scaling requirements and calculate the required capacity.

The peak ingestion rate is

15\text{ MiB/s}

. In Kinesis Data Streams, each shard supports

1\text{ MiB/s}

of ingress. An On-Demand stream automatically scales to handle this volume, whereas a provisioned stream with fewer than

15

shards would throttle.

To prevent data loss and support ingestion throughput demands.

Evaluate ordering requirements for processing.

The readings must preserve the chronological order per weather balloon. Kinesis Data Streams naturally preserves ordering per partition key (e.g., balloon ID), whereas Amazon SQS standard queues do not guarantee ordering.

Ensuring chronological data alignment is critical for accurate weather modeling.

Select the format conversion and storage mechanism.

Amazon Data Firehose can consume from the Kinesis Data Stream and perform native JSON-to-Parquet conversion using schema definition in the AWS Glue Data Catalog, then deliver the files directly to Amazon S3 within the

5

-minute time window.

This serverless integration provides the lowest operational overhead compared to custom processing configurations.

Key Concept

Leveraging Kinesis Data Streams On-Demand capacity for high-throughput scaling, combined with Amazon Data Firehose for zero-infrastructure data transformation and S3 delivery.

Question

Answer

Step-by-Step Solution

Key Concept

Common Mistakes

Topics

More questions from this topic