A company is designing a high-performing data ingestion and transformation pipeline to process streaming clickstream data from its web application. The pipeline must scale automatically to handle traffic spikes, transform the incoming data into Apache Parquet format, and deliver the final output to Amazon S3 with the least operational complexity. Which two AWS configurations should the solutions architect combine to meet these requirements? (Select two.)
- Configure an Amazon Kinesis Data Stream in on-demand capacity mode to ingest the streaming clickstream data.Answer
- Use Amazon Data Firehose to consume data from the stream, convert the format to Apache Parquet using AWS Glue, and deliver it to Amazon S3.Answer
- CUse an Amazon SQS standard queue to ingest the stream and guarantee that the clickstream events are processed in strict chronological order.
- DDeploy a provisioned Amazon Kinesis Data Stream with a fixed capacity of shard to ingest peak clickstream traffic of .
- EDeploy an AWS Lambda function configured to run continuously in a loop to poll the ingestion queue and transform the data.
Answer
Configure an Amazon Kinesis Data Stream in on-demand capacity mode, and use Amazon Data Firehose to consume the data, convert the format to Apache Parquet using AWS Glue, and deliver it to Amazon S3.
To build a serverless, auto-scaling data ingestion and transformation pipeline, the solutions architect should use Kinesis Data Streams configured in on-demand capacity mode to handle variable streaming throughput automatically. Then, Amazon Data Firehose can consume the stream and convert JSON data to Apache Parquet format using the AWS Glue Data Catalog, writing the results directly to Amazon S3. This combination scales automatically and requires minimal management.
Step-by-Step Solution
Key Concept
Serverless and auto-scaling real-time data ingestion and format transformation using Kinesis Data Streams, Amazon Data Firehose, and AWS Glue.