A healthcare provider is designing a data ingestion pipeline to collect real-time health m… — AWS Certified Solutions Architect - Associate

A healthcare provider is designing a data ingestion pipeline to collect real-time health metrics from $20,000$ patient monitoring devices. Each device transmits a $4\text{ KB}$ JSON payload every $10\text{ seconds}$ to a public endpoint. The provider needs to ingest these payloads, transform them into Apache Parquet format, and save them to Amazon S3. The data must be queryable in Amazon Athena within $3\text{ minutes}$ of generation. Additionally, the chronological order of records sent by each device must be maintained during ingestion to ensure clinical analysis accuracy. The architecture must scale dynamically to handle peak traffic with minimal management overhead. Which architecture meets these requirements?

A
Use Amazon Kinesis Data Streams (KDS) configured with $4$ provisioned shards to ingest the data, and set up Amazon Kinesis Data Firehose as the destination to convert the format to Apache Parquet before writing to Amazon S3.
Use Amazon Kinesis Data Firehose (Amazon Data Firehose) to ingest the stream. Configure Firehose to perform inline format conversion to Apache Parquet using an AWS Glue Data Catalog schema, and deliver the files to Amazon S3 with a buffer interval of $120\text{ seconds}$ .Answer
C
Use a standard Amazon SQS queue to ingest the payloads, and trigger an AWS Lambda function to batch and transform the payloads to Apache Parquet format before writing them to Amazon S3.
D
Use Amazon Kinesis Data Streams (KDS) configured with $2$ provisioned shards to ingest the data, and use an AWS Glue streaming ETL job to transform and write the data to Amazon S3 in Apache Parquet format.

Answer

The correct architecture uses Amazon Kinesis Data Firehose to ingest the data, performs format conversion to Apache Parquet using AWS Glue, and buffers the data for 120 seconds before writing to Amazon S3.

The ingestion architecture using Amazon Kinesis Data Firehose (Amazon Data Firehose) with AWS Glue inline format conversion is the most operationally efficient. Amazon Data Firehose is a fully managed, serverless service that automatically scales to handle incoming throughput without shard provisioning. It natively integrates with the AWS Glue Data Catalog to perform real-time format conversion from JSON to Apache Parquet before writing to Amazon S3. A buffer interval of

120\text{ seconds}

ensures that the files are delivered to S3 within the

3\text{ minutes}

requirement. Furthermore, Firehose preserves the ingestion order of incoming records.

Step-by-Step Solution

Calculate the required ingestion throughput of the pipeline.

20,000\text{ devices} \times (1\text{ payload} / 10\text{ seconds}) = 2,000\text{ records/second}

. Total throughput is

2,000\text{ records/second} \times 4\text{ KB/record} = 8,000\text{ KB/second} = 8\text{ MB/s}

Determining the aggregate message rate and bandwidth is essential to select and size the ingestion service appropriately.

Evaluate the capability of Kinesis Data Streams shard configurations against the calculated throughput.

A single Kinesis Data Streams shard supports a maximum write capacity of

1\text{ MB/s}

1,000\text{ records/sec}

. Standard configurations with

2

4

shards only support

2\text{ MB/s}

and

4\text{ MB/s}

respectively, resulting in throttling under the

8\text{ MB/s}

load.

To ensure data ingestion reliability, the service capacity must meet or exceed the peak ingestion throughput.

Assess the ordering and operational overhead constraints of the ingestion options.

Standard SQS queues do not guarantee message ordering, violating the chronological constraint. Using Amazon Data Firehose with inline AWS Glue format conversion provides automatic scaling up to the required throughput, preserves record order, and transforms JSON to Parquet natively without custom code or infrastructure management.

Selecting the service that satisfies both structural constraints (ordering) and operational principles (minimal overhead) leads to the optimal AWS architecture.

Key Concept

Serverless streaming ingestion and inline format conversion using Amazon Kinesis Data Firehose and AWS Glue.

Estimated Time:2m 0s

Question

Answer

Step-by-Step Solution

Key Concept

Common Mistakes

Topics

More questions from this topic