A data engineering team is deploying a high-throughput, fault-tolerant batch processing pipeline to Google Kubernetes Engine (GKE). The batch worker pods are stateless and can tolerate immediate termination when compute capacity is reclaimed. However, the cluster must also host a stateful database service that requires uninterrupted availability and persistent disk storage to prevent data corruption. To optimize operational costs while guaranteeing stability for the critical component, which cluster deployment strategy should be implemented?
- Provision a GKE Standard cluster with separate node pools: a Spot VM node pool with node selectors and tolerations for batch workers, and a standard node pool for the stateful database.Answer
- BDeploy both the batch workers and the stateful database onto a single node pool provisioned entirely with Spot VMs to maximize overall cost savings.
- CConfigure the Horizontal Pod Autoscaler (HPA) on the cluster to dynamically convert standard nodes into Spot VM nodes whenever batch processing workload demands surge.
- DDeploy a GKE Autopilot cluster and run `gcloud config set compute/zone` to enforce that node provisioning excludes preemptible instances for all deployed workloads.
Answer
Provision a GKE Standard cluster with separate node pools: a Spot VM node pool with node selectors and tolerations for batch workers, and a standard node pool for the stateful database.
GKE Standard clusters support multiple heterogeneous node pools. Placing fault-tolerant batch workloads on a Spot VM node pool drastically reduces compute costs, while placing the stateful database on a standard node pool protects it from sudden preemption.
Step-by-Step Solution
Key Concept
Workload Isolation with GKE Node Pools and Spot VMs