Question

Difficulty: HardDisaster Recovery and Business Continuity Execution

A global supply chain enterprise operates a critical order management platform with a warm standby disaster recovery topology across Google Cloud regions `us-central1` (primary) and `europe-west3` (secondary). The application relies on Cloud SQL for PostgreSQL with asynchronous cross-region read replication. Following an unrecoverable regional disaster in `us-central1`, the operational engineering team must execute the disaster recovery runbook to restore service while ensuring data consistency and preventing split-brain states. In what sequential order should the operational team execute the disaster recovery failover tasks?

  1. 1Disable client traffic ingress at the primary load balancer and revoke database write permissions to isolate the degraded primary instance.
  2. 2Promote the cross-region Cloud SQL read replica in `europe-west3` to a standalone primary database instance.
  3. 3Scale out the secondary Compute Engine managed instance groups to full production capacity.
  4. 4Update Cloud DNS routing policies to point production domain names to the secondary region endpoint.
  5. 5Configure automated Cloud Storage backup schedules and initialize a new cross-region read replica from the newly promoted database instance.

Answer

The correct operational sequence begins by stopping ingress traffic and revoking write permissions on the primary region to prevent split-brain updates. Next, promote the cross-region Cloud SQL read replica in the secondary region to a standalone read-write database instance. Third, scale up the secondary Compute Engine managed instance groups to handle full application load. Fourth, update Cloud DNS routing policies to direct client traffic to the secondary region endpoints. Finally, re-establish disaster recovery redundancy by enabling backup schedules and creating a new cross-region read replica from the newly promoted primary instance.
The correct operational order minimizes data loss and prevents split-brain scenarios. First, isolating the degraded primary stops incoming transactions and lets asynchronous replication finish applying in-flight logs. Second, promoting the database replica enables write capabilities in the secondary region. Third, scaling up compute capacity ensures the secondary application tier can handle the incoming user load. Fourth, modifying DNS records steers user traffic to the active secondary region. Finally, restoring backup schedules and creating a new cross-region replica re-establishes disaster recovery protection.

Step-by-Step Solution

1
Isolate the primary region workload
Prevents split-brain conditions and ensures all in-flight asynchronous database replication logs settle.
Promoting a replica while writes are still hitting the primary can lead to data divergence or corrupted application state.
2
Promote the secondary database replica
The Cloud SQL instance in `europe-west3` becomes a standalone primary database supporting read and write operations.
Application services cannot function properly without a writable persistent storage layer.
3
Provision and scale compute capacity in the secondary region
Compute Engine managed instance groups scale out to 100% target production size.
Ensures the application tier has sufficient resources to process production traffic without failure or severe latency degradation.
4
Redirect traffic via Cloud DNS
Client application traffic flows to the secondary region load balancer.
Traffic must only be routed once the underlying storage and compute layers are fully operational and ready to serve requests.
5
Re-establish disaster recovery redundancy
Automated backups are configured and a new secondary replica is provisioned in a third region or restored environment.
Leaves the enterprise with continuous DR protection following the primary region loss.

Key Concept

Disaster Recovery Failover Sequence and Split-Brain Prevention
Rate this question