Hang in ListObjectsV2 - pthread_join()

[jmsundar]

Using SDK v1.3.1 on Ubuntu 14.04 compiled using gcc 4.94.
Here's the backtrace from gdb during the hang.

Vector<S3::Model::Object> S3Manager::list_objects()
{
    //std::cout << "Objects in S3 bucket: " << m_bucket << std::endl;
    S3::Model::ListObjectsV2Request objects_request;
    objects_request.WithBucket(m_bucket);

    auto list_objects_outcome = s3_client->ListObjectsV2(objects_request); ###this is where the hang occurs.

    if (list_objects_outcome.IsSuccess())
    {
        return list_objects_outcome.GetResult().GetContents();
    }
    else
    {
        ios_log(ERROR, "listObjects error: %s %s", list_objects_outcome.GetError().GetExceptionName().c_str(), list_objects_outcome.GetError().GetMessage().c_str());
        return Vector<S3::Model::Object>();
    }
}

We are calling this code on a number of buckets in the init() of our library, and it completes successfully for the first two buckets which have local s3 endpoints.

The program this is being called from is single-threaded. Should I set the executor for s3_client to be single-threaded as well?

[singku]

Dose this hang happen all the time?
What Curl version are you using?
What's the corresponding log?

The executor for s3_client are mainly used for asynchronous call. You don't need to touch it.

[jmsundar]

This hang happens whenever ListObjects(v1 and v2 both) is called on an AWS S3 bucket from our shared library. Running the same code in a standalone program does not cause the hang. We're using curl 7.35.0. I don't think it matters, but for our S3Client we have set PayloadSigningPolicy to Always and useVirtualAdressing to false.

Here is the associated log.

[jmsundar]

After a little bit of research, I found this thread on a related hang, but there's no solution posted.

[jp-ryussi]

Gdb Log for the same issue:

Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1
0x00007fe70c65565b in pthread_join (threadid=140630159267584, thread_return=thread_return@entry=0x0)
at pthread_join.c:92
92 pthread_join.c: No such file or directory.
(gdb) bt
#0 0x00007fe70c65565b in pthread_join (threadid=140630159267584, thread_return=thread_return@entry=0x0)
at pthread_join.c:92
#1 0x00007fe706f33c06 in Curl_thread_join (hnd=hnd@entry=0x7fe7080a62e0) at curl_threads.c:95
#2 0x00007fe706f362e2 in destroy_async_data (async=0x7fe7080d8970, async=0x7fe7080d8970) at asyn-thread.c:326
#3 0x00007fe706f36373 in Curl_resolver_cancel (conn=conn@entry=0x7fe7080d8400) at asyn-thread.c:146
#4 0x00007fe706f11fb7 in Curl_done (connp=connp@entry=0x7fe708165010, status=CURLE_OPERATION_TIMEDOUT,
premature=premature@entry=false) at url.c:5734
#5 0x00007fe706f210a4 in curl_multi_remove_handle (multi_handle=0x7fe70802ff00,
curl_handle=curl_handle@entry=0x7fe708165000) at multi.c:547
#6 0x00007fe706f199e6 in easy_perform (events=false, data=0x7fe708165000) at easy.c:788
#7 curl_easy_perform (easy=0x7fe708165000) at easy.c:803
#8 0x00007fe707d1517c in Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const ()
from /usr/local/lib/libaws-cpp-sdk-core.so
#9 0x00007fe707cdc71e in Aws::Client::AWSClient::AttemptOneRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*) const () from /usr/local/lib/libaws-cpp-sdk-core.so
#10 0x00007fe707cdb40d in Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*) const () from /usr/local/lib/libaws-cpp-sdk-core.so
#11 0x00007fe707cdf076 in Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*) const () from /usr/local/lib/libaws-cpp-sdk-core.so
#12 0x00007fe70755e4cb in Aws::S3::S3Client::ListObjectsV2(Aws::S3::Model::ListObjectsV2Request const&) const ()
from /usr/local/lib/libaws-cpp-sdk-s3.so
#13 0x00007fe70888f2ff in S3Manager::list_objects (this=0x7fe7080a04b0) at source/S3Manager.cpp:120

(ListObjects issue)
This is working in case of local object store setup but not working with AWS S3

[jmsundar]

Solved by upgrading curl to 7.57.0