Examkin

Question

Difficulty: HardElastic Load Balancing Health Checks and Troubleshooting

A company hosts a high-traffic web application on a fleet of Amazon EC2 instances managed by an Auto Scaling group. The instances are registered with a target group of an Application Load Balancer (ALB). The target group's health check is configured to request the path `/health` on port 80. The script at `/health` performs a deep health check by querying an Amazon RDS PostgreSQL database to verify connection status.

During a marketing event, a massive traffic spike causes the RDS database CPU utilization to reach 100%, resulting in query queuing. Consequently, the ALB health checks on the EC2 instances begin timing out, and the ALB marks all instances as unhealthy. The Auto Scaling group immediately begins terminating the EC2 instances and launching new ones, which also fail health checks upon initialization, leading to a complete application outage.

Which action should a SysOps administrator take to resolve this issue and prevent future cascading failures?

  1. Change the target group health check path to a shallow endpoint, such as a static page or a simple script that only verifies the web server status without querying the database.Answer
  2. B
    Configure the target group health check protocol to HTTPS and change the port to match the database port to directly monitor database connectivity.
  3. C
    Update the subnet's outbound Network Access Control List (NACL) rules to restrict outbound traffic solely to port 80 and the database port, removing access to the ephemeral port range to prioritize health checks.
  4. D
    Configure an Amazon Route 53 active-passive failover routing policy to route client traffic to a secondary static error page when the primary ALB's health checks fail.

Answer

Change the target group health check path to a shallow endpoint, such as a static page or a simple script that only verifies the web server status without querying the database.
Changing the target group health check path to a shallow endpoint is the correct resolution. A shallow health check only verifies that the web server process (e.g., Apache, Nginx, or a Node.js process) is running and responsive on the designated port. It does not perform deep database queries. This prevents database overload or connection exhaustion from causing a cascading failure, where the Auto Scaling group terminates healthy EC2 instances that are simply waiting on database responses.

Step-by-Step Solution

1
Analyze the cause of the health check failures and the cascading outage.
The database CPU utilization spiked to 100%, causing the deep health check script `/health` (which queries the database) to time out. The ALB marked all instances unhealthy, prompting the Auto Scaling group to terminate and replace them, which worsened the outage.
Understanding the interaction between deep health checks, database load, and Auto Scaling group termination behavior is critical to identifying the root cause.
2
Decouple the application instance health from the backend database health.
Replace the deep health check path with a shallow health check path (e.g., `/ping` or `/health-shallow`) that only validates the web server is running and listening on port 80.
A shallow health check prevents transient database latency or overload from triggering mass instance termination, localizing the database issue without destroying the compute capacity.
3
Configure separate database monitoring and alerting.
Use Amazon CloudWatch alarms on the RDS database's CPU utilization and DB connections to trigger alerts or Auto Scaling actions on the database tier (like read replicas or scaling up).
Database health should be monitored and managed independently of individual web application instance health to prevent cascading failures.

Key Concept

Deep vs. Shallow Health Checks in Load Balancing
Rate this question

Topics

Elastic Load Balancing Health Checks and Troubleshooting

More questions from this topic