An enterprise web application deployed across primary region `us-central1` and secondary recovery region `us-east4` experiences a total regional failure in `us-central1`. The application uses a Pilot Light DR pattern comprising a Cloud SQL PostgreSQL cross-region read replica and a minimal regional Managed Instance Group (MIG) in `us-east4`. To achieve recovery without split-brain data corruption or premature exposure of unvalidated endpoints, in what exact sequence should the SRE team execute the disaster recovery failover runbook tasks?
- 1Revoke IAM database write permissions and isolate compute workloads in `us-central1` to prevent dual-writing.
- 2Promote the Cloud SQL cross-region read replica in `us-east4` to a standalone writeable primary instance.
- 3Scale up the pre-provisioned Compute Engine Managed Instance Group in `us-east4` to target production capacity.
- 4Execute automated synthetic transactions and database health checks against the internal load balancer endpoint in `us-east4`.
- 5Update the Global External Application Load Balancer backend service to route client ingress traffic to the `us-east4` MIG.
Answer
The correct operational sequence for regional DR failover is: 1) Isolate `us-central1` compute and database write paths to eliminate split-brain risk, 2) Promote the `us-east4` Cloud SQL cross-region read replica to standalone primary, 3) Scale up the `us-east4` Managed Instance Group to full capacity, 4) Execute synthetic health and data integrity checks against the internal `us-east4` endpoint, and 5) Update Global Load Balancer backends to direct ingress traffic to `us-east4`.
In GCP enterprise DR execution, ensuring data consistency and split-brain prevention is the highest priority. The primary degraded region must first be fenced off by revoking write access. Next, the secondary database replica is promoted to primary so that compute nodes have a valid write target. Then, the pilot-light MIG is scaled to full capacity to accommodate production traffic. Next, internal synthetic validation tests verify system readiness. Finally, the Global Load Balancer backend configuration is updated to route public client traffic to the secondary region.
Step-by-Step Solution
Key Concept
Disaster Recovery Failover Execution & Split-Brain Prevention