InvalidSignatureException when calling the GetSamplingTargets

[IrinaAlexandra]

Hello,

We are currently using Aws Xray in production on our django applications. The application is running on ECS using EC2. The xray-daemon is installed on django container itself. We have given the daemon permissions by adding the AWSXRayDaemonWriteAccess to the task's role. Occasionally we are seeing this error:

An error occurred (InvalidSignatureException) when calling the GetSamplingTargets operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

I guess the important thing to note here is we are only seeing this occasionally. As you see below, most of requests to this endpoint are good:

From some debugging it looks like the failing request is using us-west-2 as the region, whilst the region the daemon and our services are running are on eu-west-1

{
    body: b'{"SamplingStatisticsDocuments": [{"RuleName": "Default", "ClientID": "xxxxxxxxxxxxx", "RequestCount": 
         1, "BorrowCount": 0, "SampledCount": 0, "Timestamp": 1617955918}]}', 
    context: {
         auth_type: [Filtered], 
         client_config: <botocore.config.Config object at 0x7f6c7220a110>, 
         client_region: 'us-west-2', 
         has_streaming_input: False
     }, 
     headers: {
          User-Agent: 'Botocore/1.20.47 Python/3.7.10 Linux/4.14.225-169.362.amzn2.x86_64 exec-env/AWS_ECS_EC2'
     }, 
     method: 'POST', 
    query_string: {}, 
    url: 'http://127.0.0.1:2000/SamplingTargets', 
    url_path: '/SamplingTargets'
}

[stacknowledge]

Hey, thanks for raising this issue, i'm having the exact same problem, any findings so far?

[srprash]

Hi @IrinaAlexandra
The InvalidSignatureException is thrown by clients calling a remote AWS service API (XRay GetSamplingTargets in this case). However, the SDK doesn't really call the remote service endpoint, rather it calls the X-Ray Daemon endpoint which then signs the request and proxies it to X-Ray backend.
The boto client here is hard-coded to use us-west-2 and empty credentials but this shouldn't matter since the actual signing of the request is being done by the Daemon and response is sent back to the SDK. So I think the issue might be on the Daemon process.
Taking a shot in the dark here, but can you verify if there are any credentials being rotated for the daemon? Or is the daemon being periodically restarted?
Looking at the AWS doc for troubleshooting this error, the response should include the expected signature by the service. Do you see such response being recorded?
Also, how often do you see this error and is it only for the /SamplingTargets endpoint?

Thanks.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in next 7 days. Thank you for your contributions.