Question

Difficulty: MediumManaging Google Kubernetes Engine Resources

A DevOps engineer needs to safely migrate running workloads from an existing GKE node pool named `old-pool` to a newly created node pool named `new-pool` in a GKE Standard cluster without incurring unexpected downtime. Arrange the operational steps in the correct order to safely migrate the workloads and decommission the old node pool.

  1. 1Run `kubectl cordon` on the nodes in `old-pool` to mark them as unschedulable.
  2. 2Run `kubectl drain` with `--ignore-daemonsets` on the nodes in `old-pool` to gracefully evict running workloads.
  3. 3Verify that all workload pod replicas are running healthily on `new-pool` and serving traffic.
  4. 4Delete `old-pool` using `gcloud container node-pools delete old-pool --cluster=[CLUSTER_NAME]`.

Answer

The correct order of operations is: first cordon the nodes in `old-pool` to prevent new pod assignments, then drain `old-pool` to evict existing pods, next verify workload readiness on `new-pool`, and finally delete `old-pool` using gcloud.
To perform a zero-downtime workload migration between GKE node pools, the existing nodes must first be cordoned (`kubectl cordon`) so Kubernetes marks them unschedulable. Next, nodes are drained (`kubectl drain`) to evict existing pods, forcing them to reschedule onto the new node pool. After verifying that the pods are healthy and running on the new node pool, the old node pool can be safely deleted using `gcloud container node-pools delete`.

Step-by-Step Solution

1
Mark nodes in `old-pool` as unschedulable using `kubectl cordon`.
Prevents new pod replicas from being scheduled on `old-pool` nodes.
Cordoning first ensures that evicted pods are directed to `new-pool` instead of being rescheduled back onto `old-pool`.
2
Gracefully evict workloads using `kubectl drain <node-name> --ignore-daemonsets` on nodes in `old-pool`.
Safely terminates pods on `old-pool` so Kubernetes recreates them on `new-pool`.
Draining ensures pods are evicted cleanly while ignoring system DaemonSets.
3
Inspect workload pod status and service health on `new-pool`.
Confirms all pods are in the `Running` state and ready to handle user requests.
Verification prevents deleting infrastructure while workloads might be failing to schedule or start.
4
Execute `gcloud container node-pools delete old-pool` to delete the old node pool.
Decommissions the compute instances associated with `old-pool` in GCP.
Deletes obsolete GCP compute resources after successful workload migration.

Key Concept

Safely migrating workloads between GKE node pools using cordon, drain, health verification, and node pool deletion commands.
Rate this question