A smart utility company is designing a real-time data ingestion pipeline to monitor smart electricity meters. The meters transmit power consumption metrics every seconds, resulting in a total write throughput of and approximately records per second. The company has two requirements:
1. An anomaly detection application must process the incoming metrics and identify power spikes. To prevent false positives, the application must process the metrics from each individual meter in the exact order they were generated.
2. The raw data must be converted to Apache Parquet format and archived in Amazon S3 within minutes of generation for historical analysis.
Which architecture should a solutions architect recommend to meet these requirements with the highest performance and the lowest operational overhead?
- AIngest the data into an Amazon SQS standard queue. Configure the anomaly detection application to process messages from the queue. Create an Amazon Data Firehose delivery stream to poll the SQS queue, convert the messages to Parquet, and write them to Amazon S3.
- BIngest the data into Amazon Kinesis Data Streams, using a single static partition key for all smart meter payloads. Configure the anomaly detection application to consume from the stream. Configure an Amazon Data Firehose delivery stream to read from the Kinesis Data Stream, convert the data, and deliver it to Amazon S3.
- Ingest the data into Amazon Kinesis Data Streams, using the smart meter ID as the partition key. Build the anomaly detection application to consume from the stream. Configure an Amazon Data Firehose delivery stream to read from the Kinesis Data Stream, use AWS Glue to convert the data format, and deliver the files to Amazon S3.Answer
- DIngest the data into Amazon Kinesis Data Streams. Configure a continuously running AWS Lambda function that polls the stream, performs anomaly detection, and immediately writes each record individually to Amazon S3 after converting it to Parquet.