A financial services organization operates a daily batch processing pipeline on a Compute Engine Managed Instance Group (MIG) consuming messages from Cloud Pub/Sub. Every morning at market open, incoming message rates surge by within 30 seconds. Because instance initialization and software warmup take approximately 4 minutes, dynamic auto-scaling policies based on Pub/Sub queue depth or CPU load experience severe queue backlogs and SLA violations during the first 10 minutes of the spike. Which architecture strategy should a Cloud Architect recommend to eliminate initial latency spikes while maintaining cost efficiency during off-peak hours?
- Implement scheduled auto-scaling rules on the Managed Instance Group to scale out baseline capacity prior to market open, combined with dynamic target tracking scaling based on Cloud Pub/Sub unacknowledged message depth.Answer
- BConfigure the Managed Instance Group auto-scaler to use a very low CPU utilization target threshold () so scaling actions trigger immediately when traffic begins to rise.
- CMigrate the processing workload to a Google Kubernetes Engine (GKE) cluster with Cluster Autoscaler and Horizontal Pod Autoscaler (HPA) configured on default node pools.
- DAutomate daily Cloud Quota API requests to temporarily increase regional vCPU limits 15 minutes before market open and release them afterward.
Answer
Implement scheduled auto-scaling rules on the Managed Instance Group to scale out baseline capacity prior to market open, combined with dynamic target tracking scaling based on Cloud Pub/Sub unacknowledged message depth.
Combining scheduled auto-scaling with queue-based dynamic auto-scaling solves both the predictable instant spike and off-peak elasticity needs. Scheduling ensures instances are provisioned and warm before market open, avoiding the 4-minute boot delay, while Cloud Pub/Sub queue depth scaling dynamically adjusts capacity for unexpected workload variations throughout the day.
Step-by-Step Solution
Key Concept
Combining scheduled auto-scaling for predictable spikes with custom metric queue-depth scaling for asynchronous workloads.
Estimated Time:2m 0s