Access Proxy: getting 502 error since the EC2 instance in Auto Scaling group failed to pass the health check

[JoeShi]

Describe the bug

In some situation, the EC2 instance launch by the Auto Scaling group will fail to pass the health check. This is because there is a mismatch between the health check grace period and the instance startup time.

The EC2 instance will install and configure Nginx to proxy the request to the OpenSearch cluster, which may take up to 4~5 minutes. The current health check internal is 30s, and will instance will turn into unhealthy after 2 failed health check. The root cause is that the grace period is currently set to 0 which mean the ELB will check the instance immediately after the instance become running (Nginx installation in progress).

To address this issue, we can adjust the grace period to 300s. more details can be found https://docs.aws.amazon.com/autoscaling/ec2/userguide/ts-as-healthchecks.html#ts-failed-status-checks

Expected Behavior

N/A

Current Behavior

N/A

Reproduction Steps

N/A

Possible Solution

No response

Additional Information/Context

No response

Solution Version

v2.0.1

AWS Region. e.g., us-east-1

No response

Other information

No response

[JoeShi]

You have chance to enter this state. If you entered this state. You can use the following workaround to address this.

1. Go to EC2 console, and select Auto Scaling Groups.
2. Find the Auto Scaling group, and enter the details page
3. Find the Health checks section, you can find the grace period is 0.
4. click Edit, and update to 300.