A ride-sharing platform needs to design an ingestion pipeline to process GPS location upda… — AWS Certified Solutions Architect - Associate

A ride-sharing platform needs to design an ingestion pipeline to process GPS location updates from its fleet of vehicles. During peak hours, the system receives approximately $8,000$ write requests per second, with an average payload size of $1.5 \text{ KB}$ per update. The GPS data must be processed in real-time with sub-second latency for immediate routing and ETA calculations. Additionally, the data must be transformed into Parquet format and stored in Amazon S3 for historical analysis. Which architecture meets these requirements with the highest performance and scalability?

Ingest the data using Amazon Kinesis Data Streams configured in On-Demand capacity mode. Process the stream in real-time with a custom consumer application for ETA calculations. Configure Amazon Data Firehose to consume from the same Kinesis stream, transform the data to Parquet format using an AWS Lambda function, and deliver it to an Amazon S3 Standard bucket.Answer
B
Ingest the data using Amazon Kinesis Data Streams configured in Provisioned capacity mode with $5$ shards. Process the stream in real-time with a custom consumer application for ETA calculations. Configure Amazon Data Firehose to consume from the same Kinesis stream, transform the data to Parquet format using an AWS Lambda function, and deliver it to an Amazon S3 Standard bucket.
C
Ingest the data using a standard Amazon SQS queue to decouple the ingestion layer. Configure the real-time application to poll the queue to compute ETAs, and configure a fleet of EC2 instances to retrieve the messages, transform them to Parquet format, and write them to an Amazon S3 Standard bucket.
D
Ingest the data using Amazon Kinesis Data Streams configured in On-Demand capacity mode. Process the stream in real-time with a custom consumer application for ETA calculations. Configure Amazon Data Firehose to consume from the same Kinesis stream, transform the data to Parquet format using an AWS Lambda function, and deliver the output to an Amazon S3 Standard-IA bucket with an S3 Lifecycle policy to delete the files after $7$ days.

Answer

Ingest the data using Amazon Kinesis Data Streams configured in On-Demand capacity mode. Process the stream in real-time with a custom consumer application for ETA calculations. Configure Amazon Data Firehose to consume from the same Kinesis stream, transform the data to Parquet format using an AWS Lambda function, and deliver it to an Amazon S3 Standard bucket.

The correct answer provides an ingestion layer (Amazon Kinesis Data Streams in On-Demand capacity mode) that dynamically scales to handle peak volumes of

8,000

records per second (

12 \text{ MB/s}

). It supports multi-consumer routing: a custom real-time application can query Kinesis directly to keep latency below one second for route modeling, while Amazon Data Firehose consumes the same stream asynchronously to convert payloads to Parquet using AWS Lambda and delivers them to Amazon S3 Standard for long-term historical query optimization.

Step-by-Step Solution

Calculate the peak throughput requirements of the system.

The peak ingestion rate is

8,000

records per second with a payload of

1.5 \text{ KB}

per record, which equates to

12 \text{ MB/s}

of write throughput (

8,000 \times 1.5 \text{ KB} = 12,000 \text{ KB/s}

Understanding the numerical throughput bounds helps in selecting the appropriate stream capacity model.

Choose the optimal stream capacity configuration.

Select Kinesis Data Streams in On-Demand capacity mode, which dynamically scales to accommodate the required

12 \text{ MB/s}

and

8,000

write requests per second.

Using a fixed number of provisioned shards (such as

5

shards, which only support up to

5 \text{ MB/s}

and

5,000

records/sec) would lead to write throttling under peak conditions.

Select the consumer patterns for real-time processing and storage delivery.

Connect a real-time consumer application directly to Kinesis Data Streams to handle sub-second latency routing calculations. Simultaneously, attach Amazon Data Firehose to the same stream to run format conversion to Parquet via AWS Lambda, saving the outputs in Amazon S3 Standard.

Kinesis Data Streams allows multiple consumers to process the same stream in parallel, accommodating both real-time analytical calculations and near-real-time batch transformation workflows.

Key Concept

Selecting and sizing ingestion services (Kinesis Data Streams vs. SQS vs. Firehose) to meet strict throughput, ordering, and format transformation performance requirements.

Estimated Time:1m 30s

Question

Answer

Step-by-Step Solution

Key Concept

Common Mistakes

Topics

More questions from this topic