Question

Difficulty: HardElastic Load Balancing Health Checks and Troubleshooting

A company runs a REST API on a fleet of Amazon EC2 instances located in a private subnet. These instances are registered to a target group for an Application Load Balancer (ALB) situated in the public subnets of the same VPC. The target group is configured to send HTTP health checks on port 80808080 with the path `/healthz`.

A SysOps administrator observes that the ALB is reporting all registered targets as unhealthy with the error code `Target.FailedHealthChecks`. Consequently, clients requesting the application receive HTTP 502502 (Bad Gateway) errors.

The administrator verifies the following configuration details:
- Connecting to the EC2 instances directly via AWS Systems Manager Session Manager and running `curl -I http://localhost:8080/healthz` returns HTTP 200200 OK.
- The Security Group for the EC2 instances allows inbound TCP traffic on port 80808080 from the ALB security group, and allows all outbound traffic.
- The Security Group for the ALB allows inbound TCP traffic on ports 8080 and 443443 from `0.0.0.0/0`, and allows all outbound traffic.
- The stateless Network Access Control List (NACL) for the private subnet has the following rules:
- Inbound: Rule 100100 allows TCP traffic on port 80808080 from the public subnets' CIDR block. Rule 3276732767 denies all other traffic.
- Outbound: Rule 100100 allows TCP traffic on port 80808080 to the public subnets' CIDR block. Rule 3276732767 denies all other traffic.

Which of the following is the most likely cause of the failing health checks?

  1. The outbound Network Access Control List (NACL) for the private subnet blocks response traffic back to the ALB because it does not allow outbound TCP traffic to the public subnets on ephemeral ports (10241024-6553565535).Answer
  2. B
    The target group is configured to perform health checks on port 8080 by default, creating a port mismatch because the instances are only listening on port 80808080.
  3. C
    The Route 53 active-passive failover routing policy is configured without an associated health check, which causes the load balancer health status to be ignored.
  4. D
    The target group's health check is pointing to a path that redirected the ALB to a secure HTTPS URL, leading the target group to mark the instances as unhealthy due to an unsupported redirect.

Answer

The outbound Network Access Control List (NACL) for the private subnet blocks response traffic back to the ALB because it does not allow outbound TCP traffic to the public subnets on ephemeral ports (10241024-6553565535).
The correct answer is correct because Network Access Control Lists (NACLs) in AWS are stateless, meaning that return traffic must be explicitly allowed by rules. When an Application Load Balancer (ALB) performs a health check, it sends a request from a random port in the ephemeral range (10241024-6553565535) to the target group port (80808080). The instance attempts to respond back to the ALB's ephemeral port. Under the current configuration, the private subnet's outbound NACL rule only allows outbound traffic destined for port 80808080. Because the response traffic is destined for an ephemeral port (10241024-6553565535), it is blocked by the catch-all deny rule.

Step-by-Step Solution

1
Analyze the stateless nature of Network Access Control Lists (NACLs).
Identify that stateless firewalls require rules to explicitly allow both inbound and outbound traffic flows, unlike stateful security groups.
Since security groups are stateful and allow response traffic automatically, the root cause must reside in the stateless NACL configuration.
2
Trace the health check network request from the Application Load Balancer (ALB) to the EC2 instances.
The ALB sends a request from a random ephemeral source port (10241024-6553565535) to the EC2 instance's destination port 80808080. The inbound NACL rule 100100 permits this traffic.
This verifies that the request packet successfully reaches the application on the EC2 instances.
3
Trace the health check response traffic from the EC2 instances back to the ALB.
The EC2 instance sends a response packet from source port 80808080 to the ALB's destination port (which is the ephemeral port used to open the connection). The outbound NACL only allows traffic where the destination port is 80808080.
Since the destination port of the response is an ephemeral port (10241024-6553565535) and not 80808080, the outbound response packet is dropped by the default NACL deny rule.
4
Identify the required modification to resolve the block.
Add an outbound rule to the private subnet's NACL that allows outbound TCP traffic to the public subnets' CIDR block on ports 10241024-6553565535.
This allows the response packet to traverse the private subnet boundary and reach the ALB, completing the three-way handshake and validating the health check.

Key Concept

Stateless Network Access Control Lists (NACLs) and Ephemeral Ports
Estimated Time:2m 30s
Rate this question