A telemetry organization ingests vehicle diagnostic logs into Cloud Storage. Once a week, an intensive batch processing job analyzes the aggregated files. The processing run requires substantial compute capacity for approximately 6 hours, can tolerate individual node interruptions, and must complete within a 12-hour SLA window. The business objective is to minimize total compute costs while avoiding unnecessary baseline infrastructure expenditure. Which architecture should you recommend?
- Deploy a Compute Engine Managed Instance Group (MIG) using Spot VMs with custom instance templates and queue-based autoscaling.Answer
- BProvision a dedicated Google Kubernetes Engine (GKE) cluster with continuously running standard On-Demand nodes to guarantee zero disruption.
- CPurchase 3-year Committed Use Discounts (CUDs) for a dedicated pool of standard Compute Engine instances sized for peak weekly processing requirements.
- DStore the telemetry data in Cloud Spanner and execute analytical processing directly against the database cluster using SQL queries.
Answer
Deploying a Compute Engine Managed Instance Group (MIG) using Spot VMs with queue-based autoscaling is the optimal choice to minimize compute costs for fault-tolerant, short-duration weekly batch processing.
Using a Compute Engine Managed Instance Group (MIG) configured with Spot VMs allows the system to scale compute resources dynamically for the 6-hour processing run at up to an 80% discount compared to standard rates. Because the batch job can tolerate node interruptions and has a 12-hour completion window, Spot VMs fulfill all business and SLA constraints at minimal cost.
Step-by-Step Solution
Key Concept
Cost optimization for fault-tolerant batch processing using Spot VMs and dynamic scaling.