Segmentation fault at Aws::Utils::Threading::ReaderWriterLock::LockReader ()

[kaushikd153]

Confirm by changing [ ] to [x] below:

• I've gone though Developer Guide and API reference
• I've searched for previous similar issues and didn't find any solution

Platform/OS/Hardware/Device

RHEL 8.0 ,GCC 8.4

Describe the question

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff76f0555 in std::__atomic_base::operator++ (this=this@entry=0x178) at /usr/include/c++/8/bits/atomic_base.h:295
295 operator++() noexcept
Got below backtrace using GDB.

(gdb) bt
#0 0x00007ffff76f0555 in std::__atomic_base::operator++ (this=this@entry=0x178) at /usr/include/c++/8/bits/atomic_base.h:295
#1 Aws::Utils::Threading::ReaderWriterLock::LockReader (this=this@entry=0x178)
at /code_v8u1/aws-sdk/aws-sdk-1.8/aws-cpp-sdk-core/source/utils/threading/ReaderWriterLock.cpp:35
#2 0x00007ffff7700add in Aws::Utils::Threading::ReaderLockGuard::ReaderLockGuard (rwl=..., this=)
at /code_v8u1/aws-sdk/aws-sdk-1.8/aws-cpp-sdk-core/include/aws/core/utils/threading/ReaderWriterLock.h:77
#3 Aws::Config::ConfigAndCredentialsCacheManager::GetConfig (this=this@entry=0x0, profileName="default", key="max_attempts")
at /code_v8u1/aws-sdk/aws-sdk-1.8/aws-cpp-sdk-core/source/config/AWSProfileConfigLoader.cpp:420
#4 0x00007ffff770f567 in Aws::Config::GetCachedConfigValue (key="max_attempts") at /usr/include/c++/8/bits/unique_ptr.h:345
#5 0x00007ffff77108ef in Aws::Client::ClientConfiguration::ClientConfiguration (this=0x5555557e6ca8) at /usr/include/c++/8/ext/new_allocator.h:79
#6 0x0000555555562b57 in OFS_Client_S3_Impl::OFS_Client_S3_Impl (this=0x5555557e6ad0) at ../drivers/s3/Client_S3_Impl.h:20
#7 0x000055555556171f in OFS_Client::createInstance (type="S3") at OFS_Client.cpp:36
#8 0x000055555556ba94 in test_s3_credential (src=0x7fffffffd655 "s3://ap-south-1/dev-s3") at mount.cpp:268
#9 0x000055555556e32e in mount_ofs (argc=5, argv=0x7fffffffd2e8) at mount.cpp:591
#10 0x000055555556e56b in main (argc=5, argv=0x7fffffffd2e8) at main.cpp:22

The issue is with one binary ( for the proprietary product of the company, I am working for ) which is getting compiled using aws-sdk-cpp generated shared libs. The code used for generating the binary is doing fine with shared libs from aws-sdk-cpp version 1.7.324 and earlier. But I am getting issue when I compiled the binary using shared libs from aws-sdk-cpp version 1.7.325 or later. I find the issue persisting when using shared libs from aws-sdk-cpp version 1.8 and 1.9 too.

Usage: mount.s3 S3_URL MOUNT_POINT -o OPTIONS

mount.s3 s3://ap-south-1/dev-s3 /tmp/ws_5c318bb4-61c0-486f-baa0-f9fab21f3af2 -o username=******,test_s3_credential

Enter the KEYDATA : data input

Logs/output
output starts***********
[07-19 17:13:28][20773 - mount.cpp:531]src=s3://ap-south-1/dev-s3, tgt= /tmp/ws_f6750a40-6ae2-412e-9a11-7e7bbd673ecc, opt=username=AKIAVFEEXDW4RFIWRHPC,test_s3_credential
[07-19 17:13:28][20773 - mount.cpp:235]region=ap-south-1, bucket=dev-s3, endpoint=(null)
[07-19 17:13:28][20773 - mount.cpp:249]Please input key data:
data input
[07-19 17:13:40][20773 - mount.cpp:254]S3_KEY_DATA:*******************
[07-19 17:13:40][20773 - mount.cpp:265] region=ap-south-1, bucket=dev-s3, endpoint=(null), keyId=AKIAVFEEXDW4RFIWRHPC, keyData= data input
output ends***********

I will not be able to paste the exact code of the product but trying to provide reference/snippet to contact point to AWS api calls as much as possible.

int main(int argc, char *argv[])
{
************************
if(strcmp(p, "mount.s3")==0)
{
return mount_ofs(argc, argv);
}
***************************
}

int mount_ofs(int argc, char *argv[])
{
**********************
if(get_option("test_s3_credential")!=NULL) {
return test_s3_credential(src);
}
**********************
}

static int test_s3_credential(char *src)
{
***********************
LOG0("kas region=%s, bucket=%s, endpoint=%s, keyId=%s, keyData=%s", region, bucket, endpoint,keyId,keyData);
OFS_Client *s3 = OFS_Client::createInstance("S3");
***********************
return 0;
}

OFS_Client* OFS_Client::createInstance(const std::string& type)
{
if(type=="S3")
{
return new OFS_Client_S3_Impl();
}
return NULL;
}

Then the flow is going to Client_S3_Impl.h

Client_S3_Impl.h

#ifndef _CLIENT_S3_IMPL_H
#define _CLIENT_S3_IMPL_H
#include
#include
#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/core/auth/AWSCredentialsProviderChain.h>
#include <aws/core/utils/ratelimiter/DefaultRateLimiter.h>
#include "OFS_Client.h"
using namespace Aws::Http;
using namespace Aws::Client;
using namespace Aws::S3;
using namespace Aws::S3::Model;
using namespace Aws::Utils;

class OFS_Client_S3_Impl : public OFS_Client
{
private:

---

    Aws::SDKOptions m_options;
    ClientConfiguration m_config;
    ClientConfiguration m_configForFile;
    AWSCredentials m_cred;

---

public:

---

    std::shared_ptr<S3Client> getClient();
    virtual int mount(OFS_Vol** vol, const std::string& dev, const std::string& options=NULL);

---

};

I believe from here due to instance of class ClientConfiguration, the flow is going to Aws::Client::ClientConfiguration::ClientConfiguration
and subsequently segmentation fault occurred atlast(or after) Aws::Utils::Threading::ReaderWriterLock::LockReader()

[KaibaLopez]

Hi @kaushikd153 ,
So this doesn't really tell me much, the best way for me to help you is if you can provide a running code snippet that reproduces this, it doesn't have to be the actual product code but if you think that Aws::Utils::Threading::ReaderWriterLock::LockReader() is the root cause then a simple code that calls it and shows the error would be best.
I checked the 1.7.325 release and didn't see anything that would directly cause this so yea, going to need more info.

[kaushikd153]

I will try to write a standalone program.

[maxjiang153]

I also encountered this problem because Aws::Client::ClientConfiguration was created before Aws::InitAPI(options). Modifying the initialization sequence solved the problem.

[kaushikd153]

@wmz7year
Please let me know which aws-sdk-cpp version you have used.
Also if you can provide some example/code snippet to understand the initialization sequence part, that would be great!

[maxjiang153]

@kaushikd153

It's very simple with this example:

Aws::SDKOptions options;
Aws::Auth::InstanceProfileCredentialsProvider provider;  // created before call  Aws::InitAPI(options);

Aws::InitAPI(options);

provider.GetAWSCredentials ();  // this will crash 

but if

Aws::SDKOptions options;
Aws::InitAPI(options);

Aws::Auth::InstanceProfileCredentialsProvider provider;  // created after call  Aws::InitAPI(options);

provider.GetAWSCredentials ();  // it works.

aws-sdk-cpp version is 1.9.68.

[KaibaLopez]

@kaushikd153 ,
@wmz7year is right that the order of the initialization sequence is important, very specifically InitAPI has to go before anything regarding the clientconfiguraiton.
Did his comment solve your problem? Or did you finish the sample code?

[kaushikd153]

Hi @KaibaLopez

yes, the comment from @wmz7year solved issue.
But I am wondering how the reverse order was working until version aws-sdk-cpp 1.7.324
i.e. clientconfiguration object created before InitAPI

basically just for testing I tried different versions of aws-sdk-cpp from 1.7.X to 1.9.X.
But with the same piece of code ( in order clientconfiguration->InitAPI ) with aws-sdk-cpp version 1.6.53( or earlier )
we didn't face issue ( several release of Product happened also).

[kaushikd153]

@wmz7year Thanks for pointing out !

[KaibaLopez]

hard to say, the versions are quite behind now, I wasn't even part of the team on version 1.6...But as far as I understand the intention has always been that anything related to the SDK usage happens between the InitAPI and the ShutdownAPI calls.
It could be that you were hitting undefined behavior which just happened to work, we do more thorough checks for memory on the newer versions and have to keep up with newer compiler versions which are more pedantic on their warnings too.
So yea could be a lot of reasons... sorry I don't have a concrete answer.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.