Examkin

Question

Difficulty: MediumHigh-Performing Data Ingestion and Transformation Solutions

A financial trading firm is designing an ingestion and transformation pipeline to process high-frequency market data feeds from multiple external stock exchanges. The system must ingest up to 120 MB/s120\text{ MB/s} of streaming data during peak trading hours, preserving strict ordering per stock ticker symbol. Real-time analytics dashboards require sub-second processing latency. Additionally, a backup of all raw feeds must be stored in Apache Parquet format in Amazon S3, partitioned by date, with a maximum ingestion-to-storage latency of 55 minutes. The archived data must also be available for query within minutes if an ad-hoc audit is requested. Which combination of actions should the solutions architect take to meet these performance and storage requirements? (Select TWO.)

  1. Provision an Amazon Kinesis Data Stream to ingest the market data, using the stock ticker symbol as the partition key.Answer
  2. Create an Amazon Kinesis Data Firehose delivery stream with the Kinesis Data Stream as the source, enabling data format conversion to Apache Parquet before writing to Amazon S3 Standard.Answer
  3. C
    Ingest the data feeds directly into Amazon SQS Standard queues, utilizing message group IDs to guarantee order processing.
  4. D
    Configure Amazon Kinesis Data Firehose to write the transformed data directly to Amazon S3 Glacier Flexible Retrieval, using standard retrieval to access the files for ad-hoc audits.
  5. E
    Deploy an Amazon Kinesis Data Stream with a single shard and configure the Kinesis Producer Library (KPL) to scale the throughput automatically during peak hours.

Answer

The correct configuration is to provision an Amazon Kinesis Data Stream to ingest the market data using the stock ticker symbol as the partition key, and create an Amazon Kinesis Data Firehose delivery stream with the Kinesis Data Stream as the source, enabling data format conversion to Apache Parquet before writing to Amazon S3 Standard.
Provisioning an Amazon Kinesis Data Stream using the stock ticker symbol as the partition key ensures that all transactions for a specific stock are routed to the same shard, which guarantees chronological order preservation and sub-second ingestion latency. Using Amazon Kinesis Data Firehose with the stream as its source allows the streaming data to be asynchronously buffered, converted into Apache Parquet format, and delivered to Amazon S3 Standard within the 55-minute limit, without impacting the latency of the real-time processing path.

Step-by-Step Solution

1
Analyze the real-time ingestion and ordering requirements.
Real-time processing requires sub-second latency and strict ordering per stock symbol. Kinesis Data Streams supports sub-second latency and partition-key-based ordering.
Using the stock symbol as the partition key ensures all updates for a specific stock are routed to the same shard and processed in order.
2
Analyze the transformation and backup requirements.
Data must be converted to Apache Parquet and stored in S3 within 55 minutes. Amazon Kinesis Data Firehose natively supports format conversion to Parquet and delivery to S3.
By sourcing from the Kinesis Data Stream, Firehose can batch and transform the stream asynchronously without adding latency to the real-time path.
3
Evaluate the storage tier requirement for audit access speed.
The audited data must be accessible within minutes. S3 Standard allows immediate access, whereas S3 Glacier Flexible Retrieval with standard retrieval takes 33 to 55 hours.
S3 Standard meets both the immediate query requirement and the 55-minute ingestion latency target.

Key Concept

Architecting high-throughput, low-latency streaming ingestion using Kinesis Data Streams for sub-second ordering, paired with Kinesis Data Firehose for near-real-time data transformation and delivery to S3 Standard.
Estimated Time:2m 0s
Rate this question

Topics

High-Performing Data Ingestion and Transformation Solutions

More questions from this topic