Bucket Names/Endpoints/Permissions/CoreDump

[hnaik]

I am trying out the aws-sdk-s3 examples and here are some of the issues I am seeing:
(My examples are built with GCC 6.3.0)

I setup my credentials and it looks fine.

On my command line, I ran:

$ aws s3 ls

I get the following:

YYYY-mm-DD HH:MM:SS bucket_1
YYYY-mm-DD HH:MM:SS bucket_2
YYYY-mm-DD HH:MM:SS bucket_3

Then I ran the example:
$ ./list_buckets

and I got:

* bucket_1
* bucket_2
* bucket_3

So far, everything looks good and as expected. This also tells me I have the correct credentials setup.

---

Now I run:

$ aws s3 ls bucket_1

I get:

                                  PRE data/  
YYYY-mm-DD HH:MM:SS xxxxx data_1_1
YYYY-mm-DD HH:MM:SS xxxxx data_1_2

looks good.

---

Now when I run:

$ ./list_objects bucket_1

I see the following:

Objects in S3 bucket: bucket_1
ListObjects error: PermanentRedirect Unable to parse ExceptionName: PermanentRedirect Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

---

I took suggestions from: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro and ran again:

$ ./list_objects http://s3.amazonaws.com/bucket_1

and see this:
ListObjects error: NoSuchBucket The specified bucket does not exist

---

Then I ran:
$ ./list_objects http://bucket_1.s3.amazonaws.com

and see this:
ListObjects error: NoSuchBucket The specified bucket does not exist

---

Then I run:
$ ./list_objects s3.amazonaws.com/bucket_1

and see this:
ListObjects error: AccessDenied Access Denied

---

Also, when I enable the line:

Aws::ShutdownAPI(options);
I get a core dump and here is the gdb backtrace:

---

#0  __GI___pthread_mutex_lock (mutex=0x280) at ../nptl/pthread_mutex_lock.c:67
#1  0xabcdef9876543210 in __gthread_mutex_lock (__mutex=0x280) at /home/username/local/gcc-6.3.0/include/c++/6.3.0/x86_64-linux-gnu/bits/gthr-default.h:748
#2  0xabcdef9876543210 in std::mutex::lock (this=0x280) at /home/username/local/gcc-6.3.0/include/c++/6.3.0/bits/std_mutex.h:103
#3  0xabcdef9876543210 in Aws::Utils::Crypto::OpenSSL::locking_fn (mode=9, n=16) at /home/username/git/aws-sdk-cpp/aws-cpp-sdk-core/source/utils/crypto/openssl/CryptoImpl.cpp:83
#4  0xabcdef9876543210 in CRYPTO_add_lock () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#5  0xabcdef9876543210 in SSL_free () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#6  0xabcdef9876543210 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#7  0xabcdef9876543210 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#8  0xabcdef9876543210 in curl_multi_cleanup () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#9  0xabcdef9876543210 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#10 0xabcdef9876543210 in curl_easy_cleanup () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#11 0xabcdef9876543210 in Aws::Http::CurlHandleContainer::~CurlHandleContainer (this=0x14a1ca8, __in_chrg=<optimized out>) at /home/username/git/aws-sdk-cpp/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:42
#12 0xabcdef9876543210 in Aws::Http::CurlHttpClient::~CurlHttpClient (this=0x14a1c40, __in_chrg=<optimized out>) at /home/username/git/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHttpClient.h:33
#13 0xabcdef9876543210 in __gnu_cxx::new_allocator<Aws::Http::CurlHttpClient>::destroy<Aws::Http::CurlHttpClient> (this=0x14a1c40, __p=0x14a1c40) at /home/username/local/gcc-6.3.0/include/c++/6.3.0/ext/new_allocator.h:124
#14 0xabcdef9876543210 in std::allocator_traits<Aws::Allocator<Aws::Http::CurlHttpClient> >::_S_destroy<Aws::Allocator<Aws::Http::CurlHttpClient>, Aws::Http::CurlHttpClient> (__a=..., __p=0x14a1c40) at /home/username/local/gcc-6.3.0/include/c++/6.3.0/bits/alloc_traits.h:236
#15 0xabcdef9876543210 in std::allocator_traits<Aws::Allocator<Aws::Http::CurlHttpClient> >::destroy<Aws::Http::CurlHttpClient> (__a=..., __p=0x14a1c40) at /home/username/local/gcc-6.3.0/include/c++/6.3.0/bits/alloc_traits.h:336
#16 0xabcdef9876543210 in std::_Sp_counted_ptr_inplace<Aws::Http::CurlHttpClient, Aws::Allocator<Aws::Http::CurlHttpClient>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x14a1c30) at /home/username/local/gcc-6.3.0/include/c++/6.3.0/bits/shared_ptr_base.h:529
#17 0xabcdef9876543210 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#18 0xabcdef9876543210 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() ()
#19 0xabcdef9876543210 in std::__shared_ptr<Aws::Http::HttpClient, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7ffee2de1408, __in_chrg=<optimized out>) at /home/username/local/gcc-6.3.0/include/c++/6.3.0/bits/shared_ptr_base.h:928
#20 0xabcdef9876543210 in std::shared_ptr<Aws::Http::HttpClient>::~shared_ptr (this=0x7ffee2de1408, __in_chrg=<optimized out>) at /home/username/local/gcc-6.3.0/include/c++/6.3.0/bits/shared_ptr.h:93
#21 0xabcdef9876543210 in Aws::Client::AWSClient::~AWSClient (this=0x7ffee2de1400, __in_chrg=<optimized out>) at /home/username/git/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:110
#22 0xabcdef9876543210 in Aws::Client::AWSXMLClient::~AWSXMLClient (this=0x7ffee2de1400, __in_chrg=<optimized out>) at /home/username/git/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/client/AWSClient.h:259
#23 0xabcdef9876543210 in Aws::S3::S3Client::~S3Client (this=0x7ffee2de1400, __in_chrg=<optimized out>) at /home/username/git/aws-sdk-cpp/aws-cpp-sdk-s3/source/S3Client.cpp:143
#24 0xabcdef9876543210 in main ()

---

What am I missing? This is likely more of a question than a issue with the SDK, I realize I might be missing something fundamental, because it looks like everyone else has gotten these examples to work.

[bretambrose]

Can you point us to these examples? If they're the ones here: https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/cpp/example_code/s3
then those were developed by the documentation team. I can take a look at them though and try and see what's up.

[hnaik]

Thanks for looking into this!

Yes, those examples exactly.
In the meantime, I also tried building both library and examples with GCC 6.2.0 and Ubuntu 16.04's default GCC 5.4.0. Same results.

If the SDK and examples work fine for everyone else, the only thing I can think of is my local libcurl installation. But that was installed from the Ubuntu repo.

[bretambrose]

Can you tell me how exactly you are building the SDK and how exactly (compile and link lines) you are building the sample?

[bretambrose]

Beyond that, we will need you to provide a trace-level log of the actual web request. To turn this on, add the following line after the declaration:

options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;

Then paste the generated log file from a single run of the program in a reply here.

[hnaik]

So I have 'aws-sdk-cpp' master checked out at: $HOME/git/aws-sdk-cpp and examples at: $HOME/git/aws-doc-sdk-examples. (Here, I just chose to use ccmake for setting paths instead of setting cmake variables on the command line for no particular reason)

Say I have a setup like this

---

export SDK_SRC=$HOME/git/aws-sdk-cpp
export EXAMPLES_SRC=$HOME/git/aws-doc-sdk-examples/cpp/example_code/s3
export SDK_BUILD=$HOME/build/aws-sdk-cpp
export EXAMPLES_BUILD=$HOME/build/aws-doc-sdk-examples

---

Step 1:

$ cd $SDK_BUILD
$ ccmake $SDK_SRC

--> hit 'c'
--> Changed CMAKE_BUILD_TYPE=Debug
--> hit 'c'
--> hit 'g'
$ make -j8

---

Step 2:

$ cd $EXAMPLES_BUILD
$ ccmake $EXAMPLES_SRC

--> hit 'c' (this will display error that aws-sdk-cpp_DIR is not set)
--> hit 'e'
--> set aws-sdk-cpp_DIR=$SDK_BUILD
--> hit 'g'
$ make -j8

---

Noticed one more thing this morning. There are 99 buckets that get listed at the top level. It seems like I don't get permission errors on all of them. So for example a 'bucket_z' for example.

$ ./list_objects bucket_z
* dir_x/dir_y/dir_z/something.tar.gz
* dir_x/dir_y/dir_z/something_else.png

(So it basically shows full path, like the output of 'find' command)

$ aws s3 ls bucket_z

yyyy-mm-dd HH:MM:SS   12345 x_dir

(just shows the contents of the directory in question)

Also, I still get errors on other buckets I listed on Friday (and they are accessible using the aws command line tool)

---

Here is the redirection error (output of ./list_objects bucket_1)

[INFO] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] Initializing Curl library
[TRACE] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Checking HOME for the home directory.
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Environment value for variable HOME is /home/<username>
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Home directory is missing the final / appending one to normalize
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Final Home Directory is /home/<username>/
[INFO] 2017-01-24 08:04:31 Aws::Config::AWSConfigFileProfileConfigLoader [139644598806336] Initializing config loader against fileName /home/<username>//.aws/config and using profilePrefix = 1
[TRACE] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Checking HOME for the home directory.
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Environment value for variable HOME is /home/<username>
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Home directory is missing the final / appending one to normalize
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Final Home Directory is /home/<username>/
[INFO] 2017-01-24 08:04:31 Aws::Config::AWSConfigFileProfileConfigLoader [139644598806336] Initializing config loader against fileName /home/<username>//.aws/credentials and using profilePrefix = 0
[TRACE] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Checking HOME for the home directory.
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Environment value for variable HOME is /home/<username>
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Home directory is missing the final / appending one to normalize
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Final Home Directory is /home/<username>/
[TRACE] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Checking HOME for the home directory.
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Environment value for variable HOME is /home/<username>
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Home directory is missing the final / appending one to normalize
[DEBUG] 2017-01-24 08:04:31 FileSystemUtils [139644598806336] Final Home Directory is /home/<username>/
[INFO] 2017-01-24 08:04:31 ProfileConfigFileAWSCredentialsProvider [139644598806336] Setting provider to read credentials from /home/<username>//.aws/credentials for credentials file and /home/<username>//.aws/config for the config file , for use with profile default
[INFO] 2017-01-24 08:04:31 EC2MetadataClient [139644598806336] Creating HttpClient with max connections 2 and scheme http
[INFO] 2017-01-24 08:04:31 CurlHandleContainer [139644598806336] Initializing CurlHandleContainer with size 2
[INFO] 2017-01-24 08:04:31 InstanceProfileCredentialsProvider [139644598806336] Creating Instance with default EC2MetadataClient and refresh rate 900000
[DEBUG] 2017-01-24 08:04:31 Aws::Config::ConfigFileProfileFSM [139644598806336] found profile default
[DEBUG] 2017-01-24 08:04:31 Aws::Config::ConfigFileProfileFSM [139644598806336] found access key <access-key>
[INFO] 2017-01-24 08:04:31 Aws::Config::AWSProfileConfigLoader [139644598806336] Successfully reloaded configuration.
[TRACE] 2017-01-24 08:04:31 Aws::Config::AWSProfileConfigLoader [139644598806336] reloaded config at 2017-01-24T14:04:31Z
[INFO] 2017-01-24 08:04:31 CurlHandleContainer [139644598806336] Initializing CurlHandleContainer with size 25
[TRACE] 2017-01-24 08:04:31 AWSClient [139644598806336] No content body, content-length headers
[DEBUG] 2017-01-24 08:04:31 AWSAuthV4Signer [139644598806336] Note: Http payloads are not being signed. signPayloads=0 http scheme=https
[DEBUG] 2017-01-24 08:04:31 AWSAuthV4Signer [139644598806336] Canonical Header String: content-type:application/xml
host:s3.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20170124T140431Z

[DEBUG] 2017-01-24 08:04:31 AWSAuthV4Signer [139644598806336] Signed Headers value:content-type;host;x-amz-content-sha256;x-amz-date
[DEBUG] 2017-01-24 08:04:31 AWSAuthV4Signer [139644598806336] Canonical Request String: GET
/bucket_1

content-type:application/xml
host:s3.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20170124T140431Z

content-type;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
[DEBUG] 2017-01-24 08:04:31 AWSAuthV4Signer [139644598806336] Final String to sign: AWS4-HMAC-SHA256
20170124T140431Z
20170124/us-east-1/s3/aws4_request
a645c996905870aa6e4d7b18d01b492024bfb51e69d7b306f7738f08997c425f
[DEBUG] 2017-01-24 08:04:31 AWSAuthV4Signer [139644598806336] Final computed signing hash: 27d4806d1c213a3a95bde245e9016df3b2e36703d850562c5e1446b64e0f9af7
[DEBUG] 2017-01-24 08:04:31 AWSAuthV4Signer [139644598806336] Signing request with: AWS4-HMAC-SHA256 Credential=<access-key>/20170124/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=27d4806d1c213a3a95bde245e9016df3b2e36703d850562c5e1446b64e0f9af7
[DEBUG] 2017-01-24 08:04:31 AWSClient [139644598806336] Request Successfully signed
[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] Making request to https://s3.amazonaws.com/bucket_1
[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] Including headers:
[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] authorization: AWS4-HMAC-SHA256 Credential=<access-key>/20170124/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=27d4806d1c213a3a95bde245e9016df3b2e36703d850562c5e1446b64e0f9af7
[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] content-type: application/xml
[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] host: s3.amazonaws.com
[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] user-agent: aws-sdk-cpp/1.0.56 Linux/4.4.0-59-generic x86_64
[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] x-amz-content-sha256: UNSIGNED-PAYLOAD
[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] x-amz-date: 20170124T140431Z
[DEBUG] 2017-01-24 08:04:31 CurlHandleContainer [139644598806336] Attempting to acquire curl connection.
[DEBUG] 2017-01-24 08:04:31 CurlHandleContainer [139644598806336] No current connections available in pool. Attempting to create new connections.
[DEBUG] 2017-01-24 08:04:31 CurlHandleContainer [139644598806336] attempting to grow pool size by 2
[INFO] 2017-01-24 08:04:31 CurlHandleContainer [139644598806336] Pool successfully grown by 2
[INFO] 2017-01-24 08:04:31 CurlHandleContainer [139644598806336] Connection has been released. Continuing.
[DEBUG] 2017-01-24 08:04:31 CurlHandleContainer [139644598806336] Returning connection handle 0x1fa3320
[DEBUG] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] Obtained connection handle 0x1fa3320
[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] HTTP/1.1 301 Moved Permanently

[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] x-amz-bucket-region: us-west-2

[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] x-amz-request-id: 8DFAA2C98F476FED

[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] x-amz-id-2: 7+Ydw3ZF9Qb+XIf69Br7GnhyGOOFmcSVHuUBfJFRbV+QElRNC/mwNKYy7LcMvLnqPS33vTcm8/I=

[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] Content-Type: application/xml

[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] Transfer-Encoding: chunked

[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] Date: Tue, 24 Jan 2017 14:04:31 GMT

[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] Server: AmazonS3

[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] 

[TRACE] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] 439 bytes written to response.
[DEBUG] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] Returned http response code 301
[DEBUG] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] Returned content type application/xml
[DEBUG] 2017-01-24 08:04:31 CurlHttpClient [139644598806336] Releasing curl handle 0x1fa3320
[DEBUG] 2017-01-24 08:04:31 CurlHandleContainer [139644598806336] Releasing curl handle 0x1fa3320
[DEBUG] 2017-01-24 08:04:31 CurlHandleContainer [139644598806336] Notified waiting threads.
[DEBUG] 2017-01-24 08:04:31 AWSClient [139644598806336] Request returned error. Attempting to generate appropriate error codes from response
[TRACE] 2017-01-24 08:04:31 AWSClient [139644598806336] Error response is <?xml version="1.0"?>
<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>PermanentRedirect</Code>
    <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
    <Bucket>bucket_1</Bucket>
    <Endpoint>bucket_1.s3.amazonaws.com</Endpoint>
    <RequestId>8DFAA2C98F476FED</RequestId>
    <HostId>7+Ydw3ZF9Qb+XIf69Br7GnhyGOOFmcSVHuUBfJFRbV+QElRNC/mwNKYy7LcMvLnqPS33vTcm8/I=</HostId>
</Error>

[WARN] 2017-01-24 08:04:31 AWSErrorMarshaller [139644598806336] Encountered Unknown AWSError
PermanentRedirect
The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.:
[TRACE] 2017-01-24 08:04:31 AWSClient [139644598806336] Request was either successful, or we are now out of retries.

Here is the AccessDenied error (output of ./list_objects s3.amazonaws.com/bucket_1. I did notice in the log that it appends this to https://s3.amazonaws.com which makes the resultant URL: https://s3.amazonaws.com/s3.amazonaws.com/bucket_1 which seems wrong.)

[INFO] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] Initializing Curl library
[TRACE] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Checking HOME for the home directory.
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Environment value for variable HOME is /home/<username>
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Home directory is missing the final / appending one to normalize
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Final Home Directory is /home/<username>/
[INFO] 2017-01-24 08:08:53 Aws::Config::AWSConfigFileProfileConfigLoader [139681549588288] Initializing config loader against fileName /home/<username>//.aws/config and using profilePrefix = 1
[TRACE] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Checking HOME for the home directory.
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Environment value for variable HOME is /home/<username>
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Home directory is missing the final / appending one to normalize
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Final Home Directory is /home/<username>/
[INFO] 2017-01-24 08:08:53 Aws::Config::AWSConfigFileProfileConfigLoader [139681549588288] Initializing config loader against fileName /home/<username>//.aws/credentials and using profilePrefix = 0
[TRACE] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Checking HOME for the home directory.
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Environment value for variable HOME is /home/<username>
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Home directory is missing the final / appending one to normalize
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Final Home Directory is /home/<username>/
[TRACE] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Checking HOME for the home directory.
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Environment value for variable HOME is /home/<username>
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Home directory is missing the final / appending one to normalize
[DEBUG] 2017-01-24 08:08:53 FileSystemUtils [139681549588288] Final Home Directory is /home/<username>/
[INFO] 2017-01-24 08:08:53 ProfileConfigFileAWSCredentialsProvider [139681549588288] Setting provider to read credentials from /home/<username>//.aws/credentials for credentials file and /home/<username>//.aws/config for the config file , for use with profile default
[INFO] 2017-01-24 08:08:53 EC2MetadataClient [139681549588288] Creating HttpClient with max connections 2 and scheme http
[INFO] 2017-01-24 08:08:53 CurlHandleContainer [139681549588288] Initializing CurlHandleContainer with size 2
[INFO] 2017-01-24 08:08:53 InstanceProfileCredentialsProvider [139681549588288] Creating Instance with default EC2MetadataClient and refresh rate 900000
[DEBUG] 2017-01-24 08:08:53 Aws::Config::ConfigFileProfileFSM [139681549588288] found profile default
[DEBUG] 2017-01-24 08:08:53 Aws::Config::ConfigFileProfileFSM [139681549588288] found access key <access-key>
[INFO] 2017-01-24 08:08:53 Aws::Config::AWSProfileConfigLoader [139681549588288] Successfully reloaded configuration.
[TRACE] 2017-01-24 08:08:53 Aws::Config::AWSProfileConfigLoader [139681549588288] reloaded config at 2017-01-24T14:08:53Z
[INFO] 2017-01-24 08:08:53 CurlHandleContainer [139681549588288] Initializing CurlHandleContainer with size 25
[TRACE] 2017-01-24 08:08:53 AWSClient [139681549588288] No content body, content-length headers
[DEBUG] 2017-01-24 08:08:53 AWSAuthV4Signer [139681549588288] Note: Http payloads are not being signed. signPayloads=0 http scheme=https
[DEBUG] 2017-01-24 08:08:53 AWSAuthV4Signer [139681549588288] Canonical Header String: content-type:application/xml
host:s3.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20170124T140853Z

[DEBUG] 2017-01-24 08:08:53 AWSAuthV4Signer [139681549588288] Signed Headers value:content-type;host;x-amz-content-sha256;x-amz-date
[DEBUG] 2017-01-24 08:08:53 AWSAuthV4Signer [139681549588288] Canonical Request String: GET
/s3.amazonaws.com/bucket_1

content-type:application/xml
host:s3.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20170124T140853Z

content-type;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
[DEBUG] 2017-01-24 08:08:53 AWSAuthV4Signer [139681549588288] Final String to sign: AWS4-HMAC-SHA256
20170124T140853Z
20170124/us-east-1/s3/aws4_request
a1635f18f3911739515cd8f56cfff7585e52cc9e9b54f3f2d0174a911b4871de
[DEBUG] 2017-01-24 08:08:53 AWSAuthV4Signer [139681549588288] Final computed signing hash: 13932cda49e6600610daf48a0953ce1ed8a10ecb2303bb5c0b0816b45bea89cf
[DEBUG] 2017-01-24 08:08:53 AWSAuthV4Signer [139681549588288] Signing request with: AWS4-HMAC-SHA256 Credential=<access-key>/20170124/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=13932cda49e6600610daf48a0953ce1ed8a10ecb2303bb5c0b0816b45bea89cf
[DEBUG] 2017-01-24 08:08:53 AWSClient [139681549588288] Request Successfully signed
[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] Making request to https://s3.amazonaws.com/s3.amazonaws.com/bucket_1
[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] Including headers:
[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] authorization: AWS4-HMAC-SHA256 Credential=<access-key>/20170124/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=13932cda49e6600610daf48a0953ce1ed8a10ecb2303bb5c0b0816b45bea89cf
[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] content-type: application/xml
[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] host: s3.amazonaws.com
[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] user-agent: aws-sdk-cpp/1.0.56 Linux/4.4.0-59-generic x86_64
[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] x-amz-content-sha256: UNSIGNED-PAYLOAD
[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] x-amz-date: 20170124T140853Z
[DEBUG] 2017-01-24 08:08:53 CurlHandleContainer [139681549588288] Attempting to acquire curl connection.
[DEBUG] 2017-01-24 08:08:53 CurlHandleContainer [139681549588288] No current connections available in pool. Attempting to create new connections.
[DEBUG] 2017-01-24 08:08:53 CurlHandleContainer [139681549588288] attempting to grow pool size by 2
[INFO] 2017-01-24 08:08:53 CurlHandleContainer [139681549588288] Pool successfully grown by 2
[INFO] 2017-01-24 08:08:53 CurlHandleContainer [139681549588288] Connection has been released. Continuing.
[DEBUG] 2017-01-24 08:08:53 CurlHandleContainer [139681549588288] Returning connection handle 0x1ca9320
[DEBUG] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] Obtained connection handle 0x1ca9320
[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] HTTP/1.1 403 Forbidden

[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] x-amz-request-id: 6407DEF2B33B807F

[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] x-amz-id-2: OL2sXbh2V/Mkv+H5/czFJdCSN+/glnkDlEPzDhEB1HHXhhlim23hiOaC4pWrEE2CIkVLYPFO4nc=

[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] Content-Type: application/xml

[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] Transfer-Encoding: chunked

[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] Date: Tue, 24 Jan 2017 14:08:52 GMT

[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] Server: AmazonS3

[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] 

[TRACE] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] 243 bytes written to response.
[DEBUG] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] Returned http response code 403
[DEBUG] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] Returned content type application/xml
[DEBUG] 2017-01-24 08:08:53 CurlHttpClient [139681549588288] Releasing curl handle 0x1ca9320
[DEBUG] 2017-01-24 08:08:53 CurlHandleContainer [139681549588288] Releasing curl handle 0x1ca9320
[DEBUG] 2017-01-24 08:08:53 CurlHandleContainer [139681549588288] Notified waiting threads.
[DEBUG] 2017-01-24 08:08:53 AWSClient [139681549588288] Request returned error. Attempting to generate appropriate error codes from response
[TRACE] 2017-01-24 08:08:53 AWSClient [139681549588288] Error response is <?xml version="1.0"?>
<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>AccessDenied</Code>
    <Message>Access Denied</Message>
    <RequestId>6407DEF2B33B807F</RequestId>
    <HostId>OL2sXbh2V/Mkv+H5/czFJdCSN+/glnkDlEPzDhEB1HHXhhlim23hiOaC4pWrEE2CIkVLYPFO4nc=</HostId>
</Error>

[WARN] 2017-01-24 08:08:53 AWSErrorMarshaller [139681549588288] Encountered AWSError
AccessDenied
Access Denied:
[TRACE] 2017-01-24 08:08:53 AWSClient [139681549588288] Request was either successful, or we are now out of retries.

[bretambrose]

Have you tried doing a search on the error message? For example:

http://stackoverflow.com/questions/25027462/aws-s3-the-bucket-you-are-attempting-to-access-must-be-addressed-using-the-spec

There are subtle differences between the SDKs in how they handle certain cases with global and region endpoints (some do discovery and some do not), which is probably what you're seeing here when comparing to the cli. Ultimately this is probably not so much an SDK issue (although we need to get that error parsed properly) but instead a property of how the bucket was created.

[hnaik]

Yes, I have tried with region information before. That didn't work either.

[hnaik]

Hi all, thanks for all the help. It turned out to be a permission issue, although I cannot say why I was able to view all the object using the aws tool and not the examples.

I still have the libcurl crash issue when I call Aws::ShutdownAPI(options).
This seems to be the place of crash: aws-cpp-sdk-core/source/utils/crypto/openssl/CryptoImpl.cpp:83

[JonathanHenson]

you likely need scoping brackets. All of your clients need to be deleted before you call ShutdownAPI()

[hnaik]

So is there something missing in this example https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/cpp/example_code/s3/list_buckets.cpp ? That is all I was trying to run.

[JonathanHenson]

you are running it verbatim?

…

Sent from my iPhone

On Feb 2, 2017, at 8:10 AM, Harish Naik ***@***.***> wrote: So is there something missing in this example https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/cpp/example_code/s3/list_buckets.cpp ? That is all I was trying to run. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[hnaik]

Yes: git clone, build, run.

[JonathanHenson]

those were wrong until a few days ago. How long since you pulled?

…

Sent from my iPhone

On Feb 2, 2017, at 11:56 AM, Harish Naik ***@***.***> wrote: Yes: git clone, build, run. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[hnaik]

A little over a week perhaps. Didn't realize they had been updated.