A retail company wants to design a serverless ingestion pipeline to collect real-time JSON clickstream data from its web application at a rate of . The data must be converted into Apache Parquet format and saved in Amazon S3 for subsequent analysis. The solution should require minimal administration and operational overhead.
Which two AWS services or features should be used together to meet these requirements? (Select two.)
- Amazon Kinesis Data Firehose to ingest the clickstream data and deliver it directly to Amazon S3.Answer
- AWS Glue integration within Amazon Kinesis Data Firehose to perform format conversion from JSON to Apache Parquet.Answer
- CAmazon SQS standard queues to ingest and buffer the raw clickstream data while maintaining strict chronological message ordering.
- DAmazon Kinesis Data Streams configured with a single provisioned shard to automatically handle throughput peaks up to .
- EAWS Lambda functions running continuously on a 24/7 schedule to poll a web server and handle the stream ingestion.
Answer
To ingest and transform JSON clickstream data with minimal operational overhead, use Amazon Kinesis Data Firehose to receive and write the stream directly to Amazon S3, and configure AWS Glue integration within the Firehose stream to handle the inline transformation to Apache Parquet.
Using Amazon Kinesis Data Firehose simplifies the ingestion pipeline by automatically writing stream data to Amazon S3. Its native integration with AWS Glue allows for serverless, zero-code format conversion from JSON to Apache Parquet before S3 storage, keeping operational overhead to a minimum.
Step-by-Step Solution
Key Concept
Serverless data ingestion and inline transformation using Kinesis Data Firehose and AWS Glue.