A solutions architect is designing a solution to ingest real-time streaming data from web application logs and store the data in Amazon S3. The logs must be converted from JSON to Apache Parquet format before storage to optimize query performance in Amazon Athena. The architect wants to implement a fully managed, serverless solution with minimal operational overhead.
Which AWS service should the solutions architect use to meet these requirements?
- AAmazon Kinesis Data Streams with a custom consumer application running on Amazon EC2 to format and write the log data.
- Amazon Data Firehose configured to deliver log data directly to Amazon S3 with data format conversion enabled.Answer
- CAmazon SQS standard queue buffering the incoming log data, with an AWS Lambda function processing the messages and writing to Amazon S3.
- DAWS Lambda configured with an application integration to continuously pull logs and write them directly to Amazon S3.
Answer
Amazon Data Firehose configured to deliver log data directly to Amazon S3 with data format conversion enabled.
The correct choice is the option that proposes Amazon Data Firehose with data format conversion. Amazon Data Firehose is a serverless, fully managed service designed to ingest streaming data and load it directly into Amazon S3. It natively supports converting incoming JSON data to Apache Parquet format using a schema defined in AWS Glue, without requiring any custom consumer applications or manual infrastructure scaling.
Step-by-Step Solution
Key Concept
Amazon Data Firehose provides serverless ingestion, transformation, and delivery of streaming data directly to destinations like Amazon S3, with built-in format conversion capabilities.