Question

Difficulty: HardAuto Scaling and Elastic Load Balancing (ELB)

A logistics tracking application ingests real-time GPS telemetry data from fleet vehicles. The backend processing service runs on Amazon EC2 instances in an Auto Scaling group (ASG) located in private subnets behind an Application Load Balancer (ALB) in public subnets. The backend service listens on TCP port 9000. The ALB uses an HTTPS listener on port 443 and forwards requests to a target group configured to use port 9000.

To satisfy a new security policy, a network administrator modified the Network Access Control List (NACL) of the private subnets to allow inbound traffic from the public subnets only on port 9000. Shortly after this modification, the ALB marks all EC2 instances as unhealthy, and the ASG begins repeatedly terminating and replacing them. Local system logs show that the application is running and listening on port 9000 successfully on all instances.

Which configuration change will resolve this issue and restore high availability?

  1. A
    Configure the target group's health check settings to use port 443 instead of port 9000 to match the Application Load Balancer's public-facing listener port.
  2. B
    Add a rule to the security group of the EC2 instances to allow outbound traffic to the public subnets on ports 1024-65535, because security groups require explicit outbound rules for return traffic.
  3. Modify the network ACL of the private subnet to allow outbound traffic to the public subnets on ephemeral ports (1024-65535) to permit return traffic for the health checks.Answer
  4. D
    Change the Amazon Route 53 DNS routing policy to Latency routing with target health evaluation enabled, directing client traffic to bypass the Application Load Balancer.

Answer

Modify the network ACL of the private subnet to allow outbound traffic to the public subnets on ephemeral ports (1024-65535) to permit return traffic for the health checks.
The correct configuration change is to modify the network ACL of the private subnet to allow outbound traffic to the ephemeral ports. Network Access Control Lists (NACLs) are stateless, which means that return traffic must be explicitly allowed by an outbound rule. When the Application Load Balancer performs health checks or forwards traffic to the EC2 instances on port 9000, the instances attempt to respond using a destination port from the ephemeral range (1024-65535). Since the outbound NACL is blocking this return traffic, the ALB receives no response and marks the targets as unhealthy, causing the Auto Scaling group to replace them.

Step-by-Step Solution

1
Analyze the stateful vs. stateless nature of the security controls applied to the EC2 instances.
Recognize that security groups are stateful, meaning they track connection states and automatically allow return traffic. Network Access Control Lists (NACLs) are stateless, requiring explicit rules for both inbound and outbound directions.
Correctly identifying which security boundary is blocking the response traffic is essential for implementing the correct fix.
2
Determine the path of the Load Balancer health checks and how return traffic is routed.
The Application Load Balancer sends health check requests to the instances on port 9000. The EC2 instances attempt to return the response to the load balancer using a dynamic port within the ephemeral port range (1024-65535).
Understanding the source and destination ports of the return packets reveals why they are being dropped by the stateless network barrier.
3
Identify the missing rule in the subnet-level configuration.
The private subnet NACL only allows inbound traffic on port 9000 but lacks an outbound rule to allow return traffic back to the load balancer on the ephemeral ports.
This explains why the EC2 instances show the service is running locally, but the load balancer marks them as unhealthy.
4
Apply the outbound rule correction to the NACL.
Add an outbound rule to the private subnet NACL that allows traffic destined for the public subnets over ports 1024-65535.
This allows the return packets of the health check probes to pass through the NACL to the ALB, resolving the health check failures.

Key Concept

Stateless vs. Stateful Network Filtering in AWS VPC and ELB Health Checks
Rate this question