Unable to use STS regional endpoint

[bodgit]

I'm trying to spin the NTH up using queue mode in an EKS cluster that doesn't have direct internet access but does have VPC endpoints created for (hopefully) every required service.

This isn't the first controller that I've spun up; every previous one I'm using requires patching in the environment variable AWS_STS_REGIONAL_ENDPOINTS=regional in order to make them use sts.<region>.amazonaws.com instead of the global sts.amazonaws.com endpoint. The former endpoint then resolves to the VPC endpoint I've created and authentication is successful.

I'm getting the following logged when running the NTH:

2021/04/26 17:09:25 WRN There was a problem monitoring for events error="WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \"https://sts.amazonaws.com/\": dial tcp 52.94.241.129:443: i/o timeout" event_type=SQS_TERMINATE
2021/04/26 17:11:25 WRN There was a problem monitoring for events error="WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \"https://sts.amazonaws.com/\": dial tcp 52.46.134.192:443: i/o timeout" event_type=SQS_TERMINATE

I've set the above environment variable, and also tried removing the AWS_ENDPOINT and AWS_REGION variables just in case they were confusing things but the Deployment/Pod doesn't appear to be respecting it.

[haugenj]

We use a session for the service calls when running in queue processor mode, so my guess is we'll need to set the sts regional endpoint on the session, likely around here

[bodgit]

I was looking at the code that set up the AWS session and trying to comparing it to one of the controllers I've deployed that works. One such one is the external DNS controller. Here's its initialisation:

https://github.com/kubernetes-sigs/external-dns/blob/5806e3474f2e13254498bd2af34302a4e283ae39/provider/aws/aws.go#L178-L201

I can't see that much different, maybe it's just the session.Config.WithCredentials() call at the bottom of that section?

[bodgit]

I think I've figured it out:

aws-node-termination-handler/pkg/config/config.go

Lines 190 to 207 in 7a29872

	if config.EnableSQSTerminationDraining {
	sess := session.Must(session.NewSessionWithOptions(session.Options{
	SharedConfigState: session.SharedConfigEnable,
	}))
	if config.AWSRegion != "" {
	sess.Config.Region = &config.AWSRegion
	} else if *sess.Config.Region == "" && config.QueueURL != "" {
	config.AWSRegion = getRegionFromQueueURL(config.QueueURL)
	log.Debug().Str("Retrieved AWS region from queue-url: \"%s\"", config.AWSRegion)
	sess.Config.Region = &config.AWSRegion
	} else {
	config.AWSRegion = *sess.Config.Region
	}
	config.AWSSession = sess
	if config.AWSEndpoint != "" {
	config.AWSSession.Config.Endpoint = &config.AWSEndpoint
	}
	}

You need to work out the AWS region before you instantiate the session. Currently you're creating the session, then changing the region which means the STS endpoint is still the global one. If you set the region when you create the session, with the AWS_STS_REGIONAL_ENDPOINTS environment variable set then you get the regional endpoint. I proved it out with the following hack:

diff --git a/pkg/config/config.go b/pkg/config/config.go
index 3fa7aef..f47113e 100644
--- a/pkg/config/config.go
+++ b/pkg/config/config.go
@@ -20,6 +20,7 @@ import (
        "strconv"
        "strings"

+       "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/endpoints"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/rs/zerolog/log"
@@ -189,6 +190,10 @@ func ParseCliArgs() (config Config, err error) {

        if config.EnableSQSTerminationDraining {
                sess := session.Must(session.NewSessionWithOptions(session.Options{
+                       Config: aws.Config{
+                               Region: aws.String("eu-west-2"),
+                       },
                        SharedConfigState: session.SharedConfigEnable,
                }))
                if config.AWSRegion != "" {

With this, I don't get any errors but obviously I've hardcoded the region, the whole if-block needs restructuring so that the session instantiation is the final call rather than the first.

[lawliet89]

@bwagner5 Could we get this released please? I cannot use ASG instance refresh in a EKS cluster that has no access to the internet until this is released, along with a fix to the Helm chart to allow setting the AWS_STS_REGIONAL_ENDPOINTS environment variable (aws/eks-charts#531).

[bwagner5]

Yes, apologies for the delay in the release! We need to make sure the pod list is working on the webhook and then I can cut a release. Also, the AWS_STS_REGIONAL_ENDPOINTS env var will not be needed once the latest NTH is released since it now always uses the regional sts endpoints when instantiating the aws session (#421).

[lawliet89]

It's been released in 1.3.1. Tried it and it works fine. Thanks!