Can't provide AWSCredentials

[emustafa]

Hi,

I am having a weird issue which I am pretty sure I am doing something wrong because it is such a simple thing.

I am simply trying to download a file from s3 using AWSTransferManager. The file is protected so credentials has to be used.

I am creating the S3Client in the following way:

Aws::Client::ClientConfiguration config;
config.connectTimeoutMs = 15000;
config.requestTimeoutMs = 15000;
s3Client = new Aws::S3::S3Client(Aws::Auth::AWSCredentials("XXXXXXXXXXXXXXXXX", "YYYYYYYYYYYYYYYYYYYYY"), config);

Then I just create my TransferManager and all that and try to download file by giving its bucket name and key name. Then in the errorCallback that I set in Aws::Transfer::TransferManagerConfiguration I get the following message:

s3Erros.GetMessage() << " - " << s3Erros.GetErrorType()

prints out

No response body. Response code: 400 - 100

When I make the file public and provide empty credentials then the download works fine. But I can't seem to be able pass in credentials in the right way. I went over all the issues and documentation and I think I am using the api in the right way but it does not work for me. Any help is appreciated.

[leftytennis]

Here's what I use which works just fine:

ClientConfiguration s3_cfg;
AWSCredentials credentials("foo", "bar");
m_s3 = MakeShared("s3bkup", credentials, s3_cfg);

[emustafa]

Yeah I am doing similar thing and it does not work for me. Here is what I do. Could you please review the code below and see if it has any issues?

    Aws::Client::ClientConfiguration config;
    config.connectTimeoutMs = 15000;
    config.requestTimeoutMs = 15000;
    auto s3Client = Aws::MakeShared<Aws::S3::S3Client>("AwsHelper", Aws::Auth::AWSCredentials("foo", "bar"), config);

    Aws::Transfer::TransferManagerConfiguration conf;
    conf.s3Client = s3Client;
    conf.maxParallelTransfers = QThread::idealThreadCount();
    conf.transferStatusUpdatedCallback = [&](const Aws::Transfer::TransferManager* transferManager_, const Aws::Transfer::TransferHandle& transferHandle_) {
        transferStatusUpdatedCallback(transferManager_, transferHandle_);
    };
    conf.errorCallback = [&](const Aws::Transfer::TransferManager* transferManager_, const Aws::Transfer::TransferHandle& transferHandle_, const Aws::Client::AWSError<Aws::S3::S3Errors>& s3Erros) {
        errorCallback(transferManager_, transferHandle_, s3Erros);
    };

    transferManager = Aws::Transfer::TransferManager::Create(conf);

[marcomagdy]

Your code looks fine. Make sure you're using the right access/secret keys and that the IAM role that those credentials belong to have permissions to access your bucket/file.

[emustafa]

I am pretty sure the keys and the role are fine. Because when I use boto3 it works fine. Actually, I am porting my code from python to cpp so there is no issue there.

The only IAM role the keys have is GetObject. So even GetObjectList won't work. Does TransferManager::DownloadFile try to use any other role other than GetObject? Because I remember when I was doing boto3, it was using GetObjectList to get the list of keys in the bucket. I had to pass in a flag to disable that.

[marcomagdy]

It might be helpful to set your logging verbosity to Trace and paste the output here.
Look at the unit tests for an example on how to do that.

[emustafa]

So I set the logging to Trace and here is the logs

[INFO] 2017-08-22 17:08:13 Aws_Init_Cleanup [6496] Initiate AWS SDK for C++ with Version:1.1.30
[INFO] 2017-08-22 17:08:13 WinHttpSyncHttpClient [6496] Creating http client with user agent aws-sdk-cpp/1.1.30 Windows/10.0.15063.0 AMD64 with max connections 25 request timeout 15000,and connect timeout 15000
[DEBUG] 2017-08-22 17:08:13 WinHttpSyncHttpClient [6496] API handle 000001AB5DC2C970
[INFO] 2017-08-22 17:08:13 ConnectionPoolMgr [6496] Creating connection pool mgr with handle 000001AB5DC2C970, and max connections per host 25, request timeout 15000 ms, and connect timeout in 15000 ms.
[TRACE] 2017-08-22 17:08:13 AWSClient [8944] No content body, content-length headers
[DEBUG] 2017-08-22 17:08:13 AWSAuthV4Signer [8944] Using cached empty string sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 because payload is empty.
[DEBUG] 2017-08-22 17:08:13 AWSAuthV4Signer [8944] Canonical Header String: content-type:application/xml
host:somebucketxxxx.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20170822T210813Z

[DEBUG] 2017-08-22 17:08:13 AWSAuthV4Signer [8944] Signed Headers value:content-type;host;x-amz-content-sha256;x-amz-date
[DEBUG] 2017-08-22 17:08:13 AWSAuthV4Signer [8944] Canonical Request String: HEAD
/somekeyxxxx

content-type:application/xml
host:somebucketxxxx.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20170822T210813Z

content-type;host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
[DEBUG] 2017-08-22 17:08:13 AWSAuthV4Signer [8944] Final String to sign: AWS4-HMAC-SHA256
20170822T210813Z
20170822/us-east-1/s3/aws4_request
0454b8730e4c6cd464d20e5e9d3ce4c8a1ad40cfca95dc25a5303178c96a1416
[DEBUG] 2017-08-22 17:08:13 AWSAuthV4Signer [8944] Final computed signing hash: a624481d0ae2510bc03a323f53a545528c7bee52c1b55d0c977779c231850717
[DEBUG] 2017-08-22 17:08:13 AWSAuthV4Signer [8944] Signing request with: AWS4-HMAC-SHA256 Credential=AKIAJY2LINRAF25KYDAQ/20170822/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=a624481d0ae2510bc03a323f53a545528c7bee52c1b55d0c977779c231850717
[DEBUG] 2017-08-22 17:08:13 AWSClient [8944] Request Successfully signed
[TRACE] 2017-08-22 17:08:13 WinHttpSyncHttpClient [8944] Making HEAD request to uri https://somebucketxxxx.s3.amazonaws.com/somekeyxxxx
[INFO] 2017-08-22 17:08:13 WinHttpConnectionPoolMgr [8944] Attempting to aquire connection for somebucketxxxx.s3.amazonaws.com:443
[DEBUG] 2017-08-22 17:08:13 WinHttpConnectionPoolMgr [8944] Pool doesn't exist for endpoint, creating...
[DEBUG] 2017-08-22 17:08:13 WinHttpConnectionPoolMgr [8944] Pool has no available existing connections for endpoint, attempting to grow pool.
[INFO] 2017-08-22 17:08:13 WinHttpConnectionPoolMgr [8944] Connection now available, continuing.
[DEBUG] 2017-08-22 17:08:13 WinHttpConnectionPoolMgr [8944] Returning connection handle 000001ABA4C538A0
[DEBUG] 2017-08-22 17:08:13 WinHttpSyncHttpClient [8944] Acquired connection 000001ABA4C538A0
[DEBUG] 2017-08-22 17:08:13 WinHttpSyncHttpClient [8944] AllocateWindowsHttpRequest returned handle 000001ABA4C53DC0
[DEBUG] 2017-08-22 17:08:13 WinHttpSyncHttpClient [8944] with headers:
[DEBUG] 2017-08-22 17:08:13 WinHttpSyncHttpClient [8944] authorization: AWS4-HMAC-SHA256 Credential=AKIAJY2LINRAF25KYDAQ/20170822/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=a624481d0ae2510bc03a323f53a545528c7bee52c1b55d0c977779c231850717

content-type: application/xml

host: somebucketxxxx.s3.amazonaws.com

user-agent: aws-sdk-cpp/1.1.30 Windows/10.0.15063.0 AMD64

x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

x-amz-date: 20170822T210813Z


[DEBUG] 2017-08-22 17:08:13 WinHttpSyncHttpClient [8944] Received response code 400
[DEBUG] 2017-08-22 17:08:13 WinHttpSyncHttpClient [8944] Received content type application/xml
[DEBUG] 2017-08-22 17:08:13 WinHttpSyncHttpClient [8944] Received headers:
[DEBUG] 2017-08-22 17:08:13 WinHttpSyncHttpClient [8944] HTTP/1.1 400 Bad Request

Connection: close

Date: Tue, 22 Aug 2017 21:08:11 GMT

Transfer-Encoding: chunked

Content-Type: application/xml

Server: AmazonS3

x-amz-request-id: CDCF8D415C026C67

x-amz-id-2: kVZNLpLEXsJ6Bic9/oLkIVP6XhByconz/VC3rhaNmEA7mQz9V+PDtZ1O3FoB3scueBAW0STppX0=




[DEBUG] 2017-08-22 17:08:13 WinHttpSyncHttpClient [8944] Closing http request handle 000001ABA4C53DC0
[DEBUG] 2017-08-22 17:08:13 WinHttpSyncHttpClient [8944] Releasing connection handle 000001ABA4C538A0
[DEBUG] 2017-08-22 17:08:13 WinHttpConnectionPoolMgr [8944] Releasing connection to endpoint somebucketxxxx.s3.amazonaws.com:443
[DEBUG] 2017-08-22 17:08:13 AWSClient [8944] Request returned error. Attempting to generate appropriate error codes from response
[ERROR] 2017-08-22 17:08:13 AWSClient [8944] No response body. Response code: 400
[WARN] 2017-08-22 17:08:13 AWSClient [8944] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[DEBUG] 2017-08-22 17:08:13 AWSClient [8944] Server time is Tue, 22 Aug 2017 21:08:11 GMT, while client time is Tue, 22 Aug 2017 21:08:13 GMT

[marcomagdy]

The only IAM role the keys have is GetObject.

Sorry I missed that the first time.
TransferManager requires s3:GetObjectVersion permission as well to determine if it can use multi-part downloads. GetObjectVersion allows TransferManager to read the metadata of the object to determine its size.

Hope this helps.

[emustafa]

I changed my permission file and now it is like this

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "*"
        }
    ]
}

However, it still does not work.

[emustafa]

I checked the code and it looks it is making HeadObject request first. And in the logs, I think that is the request that is failing.

I checked Head Object docs here. It looks like s3:GetObject should be enough.

Yet it fails.

[marcomagdy]

Just to rule out permissions, can you set your permissions to:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*"
            ],
            "Resource": "*"
        }
    ]
}

Also is it still the Head request that is failing after you set those permissions?

[emustafa]

I tried that and still the same thing.

I am reading HEAD Object request docs and there is note

Encryption request headers, like x-amz-server-side-encryption, should not be sent for GET requests if your object uses server-side encryption with AWS KMS–managed encryption keys (SSE-KMS) or server-side encryption with Amazon S3–managed encryption keys (SSE-S3). If your object does use these types of keys, you’ll get an HTTP 400 BadRequest error.

This is the only place where it says it will return 400. And the file, I am trying to download, has Encryption setting set to AES-256 (Use Amazon S3 server-side encryption to encrypt your data.). Would this have anything to do with it? But also note that I tried to set encryption setting to None and it still had the same issue.

Btw, if I make the file public and specify empty credentials then I can download the file fine. So this is definitely either permission issue or something related to file's attributes.

EDIT:
Another observation I have is this. If I provide the credentials and make the file public then download still fails with the same error.

EDIT2:
This is getting weird. I created a new user and gave all the permissions

"Action": [
                "*"
            ],

And I am trying to download a file that does not have encryption, uses standard storage and is public. And still the same thing happens.

[marcomagdy]

I was able to reproduce this when I accessed a bucket created in a different region other than the default us-east-1.

config.region = <region where your bucket is created>

Give that a try, if that still doesn't work then try to get the error by using S3 client directly.

Aws::S3::Model::GetObjectRequest req;
req.WithBucket("somebucket").WithKey("somekey");
auto result = s3client->GetObject(req);
if(!result.IsSuccess())
{
  printf("Failing with error: %s - %s", result.GetError().GetMessage().c_str(), result.GetError().GetExceptionName().c_str());
}

[emustafa]

Hi,

Yes that was the issue. For me the region is us-west-2. I set the region to us-west-2 and now it downloads fine. So that is great.

However, I might have to download from any region. So how do I handle that? Do I create different s3Client and transfer manager for each region? Or is there an easier way of doing that?

[marcomagdy]

Awesome.

Do I create different s3Client and transfer manager for each region?

Each S3Client is tied to a single region. So yes, you need a separate TransferManager instance for every region you are hitting.
You can find out the region dynamically by calling the GetBucketLocation API http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETlocation.html

Please reopen this issue if you have further questions.