crash with the SDK library

[msinha31]

I am sending the stack trace also code is attached
build options

-Wall -std=c++0x -g -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -mtune=core2 -l aws_dynamo -pedantic-errors -Wall -Wno-long-long -O1 -L/usr/lib -lstdc++ -lm -laws-cpp-sdk-dynamodb -laws-cpp-sdk-core -laws-cpp-sdk-sdb -DAWS_CUSTOM_MEMORY_MANAGEMENT=1"

Stack trace:

Program terminated with signal 11, Segmentation fault.
#0 std::basic_string<char, std::char_traits, Aws::Allocator >::_Rep::_S_create (__capacity=__capacity@entry=5,

__old_capacity=__old_capacity@entry=0, __alloc=...) at /usr/include/c++/4.8.3/bits/basic_string.tcc:609

609 __p->_M_capacity = __capacity;
Missing separate debuginfos, use: debuginfo-install glibc-2.17-55.145.amzn1.x86_64 gperftools-libs-2.0-11.5.amzn1.x86_64 keyutils-libs-1.5.8-3.12.amzn1.x86_64 krb5-libs-1.13.2-10.39.amzn1.x86_64 libcom_err-1.42.12-4.40.amzn1.x86_64 libevent-2.0.18-1.11.amzn1.x86_64 libselinux-2.1.10-3.22.amzn1.x86_64 openssl-1.0.1k-13.88.amzn1.x86_64 zlib-1.2.8-7.18.amzn1.x86_64
(gdb) where
#0 std::basic_string<char, std::char_traits, Aws::Allocator >::_Rep::_S_create (__capacity=__capacity@entry=5,

__old_capacity=__old_capacity@entry=0, __alloc=...) at /usr/include/c++/4.8.3/bits/basic_string.tcc:609

#1 0x0000000000823f91 in _S_construct<char const*> (__a=..., __end=,

__beg=0x7f4ccb11a5e8 <vtable for std::_Sp_counted_ptr_inplace<Aws::Client::DynamoDBErrorMarshaller, Aws::Allocator<Aws::Client::DynamoDBErrorMarshaller>, (__gnu_cxx::_Lock_policy)2>+40> "Linux")
at /usr/include/c++/4.8.3/bits/basic_string.tcc:138

#2 _S_construct_aux<char const*> (__a=..., __end=,

__beg=0x7f4ccb11a5e8 <vtable for std::_Sp_counted_ptr_inplace<Aws::Client::DynamoDBErrorMarshaller, Aws::Allocator<Aws::Client::DynamoDBErrorMarshaller>, (__gnu_cxx::_Lock_policy)2>+40> "Linux") at /usr/include/c++/4.8.3/bits/basic_string.h:1725

#3 _S_construct<char const*> (__a=..., __end=,

__beg=0x7f4ccb11a5e8 <vtable for std::_Sp_counted_ptr_inplace<Aws::Client::DynamoDBErrorMarshaller, Aws::Allocator<Aws::Client::DynamoDBErrorMarshaller>, (__gnu_cxx::_Lock_policy)2>+40> "Linux") at /usr/include/c++/4.8.3/bits/basic_string.h:1746

#4 std::basic_string<char, std::char_traits, Aws::Allocator >::basic_string (this=0x7f4cbee59b60,

__s=<optimized out>, __a=...) at /usr/include/c++/4.8.3/bits/basic_string.tcc:215

#5 0x00007f4ccac06d2b in Aws::Utils::StringUtils::LTrim(char const*) ()

from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#6 0x00007f4ccac06f5c in Aws::Utils::StringUtils::Trim(char const*) ()

from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#7 0x00007f4ccac05415 in getSysCommandOutput(char const*) () from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#8 0x00007f4ccac054ae in Aws::Utils::OSVersionInfo::ComputeOSVersionString() const ()

from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#9 0x00007f4ccabf1b5e in Aws::Client::ComputeUserAgentString() ()

from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#10 0x00007f4ccabf1eec in Aws::Client::ClientConfiguration::ClientConfiguration() ()

from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#11 0x00000000008201ae in boss::Dig_CP::getScores::retrieveScore (this=this@entry=0x7f4cbee5a770, tableName="good_wtp",

wtp="e2e40c9a-95a8-47c3-a746-e82cbecd2e42")
at /home/boss/boss_deployment/CP_source_new_v2/source_updated/C++/Algorithms/Dig_CP/src/Dig_CP_getScores.cpp:81

#12 0x0000000000821e7a in boss::Dig_CP::getScores::getScoresWRTWTP (this=this@entry=0x7f4cbee5a770,

wtp="e2e40c9a-95a8-47c3-a746-e82cbecd2e42")

Dig_CP_getScores.txt

[JonathanHenson]

-fno-builtin-malloc is the likely problem. The default memory handler if you are using custom memory management does malloc under the hood. You can either build the library with custom memory management turned off, or provide a memory allocation strategy that goes through your allocators (what I'm guessing you'll want to do, otherwise you wouldn't have turned malloc off).

Info is in the readme. Also, I do a live coding demo on this in the video posted in the readme.

[msinha31]

I am unable to fix this
step I did

1. cloned github
2. cmake CMakeList.txt
3. make
4. sudo make install

then

CMakelist.txt has
set(CMAKE_LIBRARY_PATH /usr/local/lib/linux/intel64/Release
)

set(CMAKE_CXX_FLAGS "-Wall -std=c++0x -g -mtune=core2 -l aws_dynamo -pedantic-errors -Wall -Wno-long-long -O1 -L/usr/lib -lstdc++ -lm -laws-cpp-sdk-dynamodb -laws-cpp-sdk-core -laws-cpp-sdk-sdb -DAWS_CUSTOM_MEMORY_MANAGEMENT=1")

Core was generated by `./../../build/bin/Dig_CP_1 conf/Dig_CP_MainConf.xml'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f5408f693db in void std::vectorstd::shared_ptr<Aws::Auth::AWSCredentialsProvider, Aws::Allocatorstd::shared_ptr<Aws::Auth::AWSCredentialsProvider > >::_M_emplace_back_auxstd::shared_ptr<Aws::Auth::AWSCredentialsProvider const&>(std::shared_ptrAws::Auth::AWSCredentialsProvider const&) ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
Missing separate debuginfos, use: debuginfo-install glibc-2.17-55.145.amzn1.x86_64 gperftools-libs-2.0-11.5.amzn1.x86_64 keyutils-libs-1.5.8-3.12.amzn1.x86_64 krb5-libs-1.13.2-10.39.amzn1.x86_64 libcom_err-1.42.12-4.40.amzn1.x86_64 libevent-2.0.18-1.11.amzn1.x86_64 libselinux-2.1.10-3.22.amzn1.x86_64 openssl-1.0.1k-13.88.amzn1.x86_64 zlib-1.2.8-7.18.amzn1.x86_64
(gdb) where
#0 0x00007f5408f693db in void std::vectorstd::shared_ptr<Aws::Auth::AWSCredentialsProvider, Aws::Allocatorstd::shared_ptr<Aws::Auth::AWSCredentialsProvider > >::_M_emplace_back_auxstd::shared_ptr<Aws::Auth::AWSCredentialsProvider const&>(std::shared_ptrAws::Auth::AWSCredentialsProvider const&) ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#1 0x00007f5408f68cb3 in Aws::Auth::DefaultAWSCredentialsProviderChain::DefaultAWSCredentialsProviderChain() ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#2 0x00007f5409241ae3 in Aws::DynamoDB::DynamoDBClient::DynamoDBClient(Aws::Client::ClientConfiguration const&) ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-dynamodb.so
#3 0x0000000000820183 in boss::Dig_CP::getScores::retrieveScore (
this=this@entry=0x7f53ff9e5770, tableName="good_wtp",
wtp="cd27e640-6a82-474f-addb-7747c0d88986")
at /home/boss/boss_deployment/CP_source_new_v2/source_updated/C++/Algorithms/Dig_CP/src/Dig_CP_getScores.cpp:81

[msinha31]

DO you have any suggestion for this ?
-Manisha

[JonathanHenson]

I'm looking now.

[JonathanHenson]

-Wno-long-long .... we use long long, I doubt that's the issue but something to be aware of.

It looks like it's failing in the construction of the default credentials provider chain.
Two things, can you turn logging on and upload the log? directions are in the readme.
Also, can you run valgrind on the code and see where it shows the error?

[msinha31]

Thanks for your suggestion but I am still unable to solve the issue
attaching the log
Moreover I have removed the -Wno-long-long . from compilation.

#0 0x0000000000000000 in ?? ()
#1 0x00007fa87343ecda in Aws::Utils::Logging::FormattedLogSystem::Log(Aws::Utils::Logging::LogLevel, char const_, char const_, ...) ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#2 0x00007fa873413a06 in Aws::Auth::DefaultAWSCredentialsProviderChain::DefaultAWSCredentialsProviderChain() () from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#3 0x00007fa8736ecae3 in Aws::DynamoDB::DynamoDBClient::DynamoDBClient(Aws::Client::ClientConfiguration const&) ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-dynamodb.so
#4 0x0000000000820d43 in boss::Dig_CP::getScores::retrieveScore (this=this@entry=0x7fa867111770, tableName="good_wtp", wtp="d089be3a-a7d3-4543-a07a-d280c0237e89")
at /home/boss/boss_deployment/CP_source_new_v2/source_updated/C++/Algorithms/Dig_CP/src/Dig_CP_getScores.cpp:87
#5 0x0000000000822b01 in boss::Dig_CP::getScores::getScoresWRTWTP (this=this@entry=0x7fa867111770, wtp="d089be3a-a7d3-4543-a07a-d280c0237e89")
at /home/boss/boss_deployment/CP_source_new_v2/source_updated/C++/Algorithms/Dig_CP/src/Dig_CP_getScores.cpp:143
#6 0x000000000081cb01 in boss::Dig_CP::Dig_CP_LineItemBasic::bid (this=0x2437a80, request=...)
aws_sdk_2016-01-13-13.txt

[JonathanHenson]

A couple of thoughts.

a.) It looks like you ran 9 minutes before the segfault (that would have been useful to know as I thought it was failing on the first call).
b.) It is fairly expensive to allocate these clients and I would just store one inside your class as a member variable, or a global pointer somewhere. That should dramatically improve your performance as you can take advantage of connection pooling, not needing to hit disk to grab credentials etc... The clients are thread safe.
c.) It appears to be failing in a different place everytime which makes me wonder if your stack is getting corrupted elsewhere in your code base and failing inside this code. Any chance you could run valgrind --tool=memcheck on this and post the output?

[bretambrose]

Additionally, this appears to be a run of a Release build. Perhaps things would be clearer running a Debug build.

[msinha31]

running valgrind is not possible we tried some time back with no success
you want sdk to be debug build or my code

[msinha31]

i am using GNU Make 3.82

[JonathanHenson]

Without running some sort of memory analyzer, we will be looking for a needle in a haystack.... I can try pulling it down and running valgrind if you would like. All it takes is one buffer being overwritten or one bad alias and we would get this behavior--and the code base is large. Valgrind will find it in about 30 seconds.

[msinha31]

valgrind ./../../build/bin/Dig_CP_1 conf/Dig_CP_MainConf.xml --tool=memcheck -v
vex amd64->IR: unhandled instruction bytes: 0x66 0xF 0x1B 0x4 0x24 0x66 0xF 0x1B
vex amd64->IR: REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR: VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=0F
vex amd64->IR: PFX.66=1 PFX.F2=0 PFX.F3=0
==13663== valgrind: Unrecognised instruction at address 0x4015237.
==13663== at 0x4015237: _dl_runtime_resolve (in /lib64/ld-2.17.so)
==13663== by 0x6BEF2F8: __exp_finite (in /lib64/libm-2.17.so)
==13663== by 0x400B75A: _dl_relocate_object (in /lib64/ld-2.17.so)
==13663== by 0x4003AD9: dl_main (in /lib64/ld-2.17.so)
==13663== by 0x40167D4: _dl_sysdep_start (in /lib64/ld-2.17.so)
==13663== by 0x4004CC0: _dl_start (in /lib64/ld-2.17.so)
==13663== by 0x4001437: ??? (in /lib64/ld-2.17.so)
==13663== by 0x3: ???
==13663== by 0xFFF00045E: ???
==13663== by 0xFFF000479: ???
==13663== by 0xFFF000492: ???
==13663== by 0xFFF0004A2: ???
==13663== Your program just tried to execute an instruction that Valgrind
==13663== did not recognise. There are two possible reasons for this.
==13663== 1. Your program has a bug and erroneously jumped to a non-code
==13663== location. If you are running Memcheck and you just saw a
==13663== warning about a bad jump, it's probably your program's fault.
==13663== 2. The instruction is legitimate but Valgrind doesn't handle it,
==13663== i.e. it's Valgrind's fault. If you think this is the case or
==13663== you are not sure, please let us know and we'll try to fix it.
==13663== Either way, Valgrind will now raise a SIGILL signal which will
==13663== probably kill your program.
==13663==
==13663== Process terminating with default action of signal 4 (SIGILL)
==13663== Illegal opcode at address 0x4015237
==13663== at 0x4015237: _dl_runtime_resolve (in /lib64/ld-2.17.so)
==13663== by 0x6BEF2F8: __exp_finite (in /lib64/libm-2.17.so)
==13663== by 0x400B75A: _dl_relocate_object (in /lib64/ld-2.17.so)
==13663== by 0x4003AD9: dl_main (in /lib64/ld-2.17.so)
==13663== by 0x40167D4: _dl_sysdep_start (in /lib64/ld-2.17.so)
==13663== by 0x4004CC0: _dl_start (in /lib64/ld-2.17.so)
==13663== by 0x4001437: ??? (in /lib64/ld-2.17.so)
==13663== by 0x3: ???
==13663== by 0xFFF00045E: ???
==13663== by 0xFFF000479: ???
==13663== by 0xFFF000492: ???
==13663== by 0xFFF0004A2: ???
==13663== Jump to the invalid address stated on the next line
==13663== at 0x576: ???
==13663== by 0x49CCDF: ??? (in /home/boss/boss_deployment/CP_source_new_v2/build/bin/Dig_CP_1)
==13663== by 0x4003AD9: dl_main (in /lib64/ld-2.17.so)
==13663== by 0x40167D4: _dl_sysdep_start (in /lib64/ld-2.17.so)
==13663== by 0x4004CC0: _dl_start (in /lib64/ld-2.17.so)
==13663== by 0x4001437: ??? (in /lib64/ld-2.17.so)
==13663== by 0x3: ???
==13663== by 0xFFF00045E: ???
==13663== by 0xFFF000479: ???
==13663== by 0xFFF000492: ???
==13663== by 0xFFF0004A2: ???
==13663== Address 0x576 is not stack'd, malloc'd or (recently) free'd
==13663==
==13663==
==13663== Process terminating with default action of signal 11 (SIGSEGV)
==13663== Bad permissions for mapped region at address 0x576
==13663== at 0x576: ???
==13663== by 0x49CCDF: ??? (in /home/boss/boss_deployment/CP_source_new_v2/build/bin/Dig_CP_1)
==13663== by 0x4003AD9: dl_main (in /lib64/ld-2.17.so)
==13663== by 0x40167D4: _dl_sysdep_start (in /lib64/ld-2.17.so)
==13663== by 0x4004CC0: _dl_start (in /lib64/ld-2.17.so)
==13663== by 0x4001437: ??? (in /lib64/ld-2.17.so)
==13663== by 0x3: ???
==13663== by 0xFFF00045E: ???
==13663== by 0xFFF000479: ???
==13663== by 0xFFF000492: ???
==13663== by 0xFFF0004A2: ???
==13663==
==13663== HEAP SUMMARY:
==13663== in use at exit: 0 bytes in 0 blocks
==13663== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==13663==
==13663== All heap blocks were freed -- no leaks are possible
==13663==
==13663== For counts of detected and suppressed errors, rerun with: -v
==13663== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)
Segmentation fault

[msinha31]

==9793== Memcheck, a memory error detector
==9793== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==9793== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==9793== Command: ./../../build/bin/Dig_CP_1 conf/Dig_CP_MainConf.xml --tool=memcheck
==9793==
==9793== brk segment overflow in thread #1: can't grow to 0x4a24000
./../../build/bin/Dig_CP_1 starts
to load xml tree /home/boss/boss_deployment/CP_source_new/deploy/Dig_CP/conf/Dig_CP_ReadLineItemValFromAdBoard.xml
to load xml tree /home/boss/boss_deployment/CP_source_new/deploy/Dig_CP/conf/Dig_CP_ServerConf.xml
to load xml tree /home/boss/boss_deployment/CP_source_new/deploy/Dig_CP/conf/Dig_CP_data_conf.xml
to load xml tree /home/boss/boss_deployment/CP_source_new/deploy/Dig_CP/conf/Dig_CP_log4cppPath.xml
Dig_CP_LineItemFactory::create() LineItemID=1888121476 not supported, will create Dig_CP_LineItemBasic
Dig_CP_LineItemFactory::create() LineItemID=1888117390 not supported, will create Dig_CP_LineItemBasic
Dig_CP_LineItemFactory::create() LineItemID=1888121476 not supported, will create Dig_CP_LineItemBasic

[msinha31]

Hi
The logs and valgrind output
ignore the previous ones
valgrind ./../../build/bin/Dig_CP_1 conf/Dig_CP_MainConf.xml --tool=memcheck -v
aws_sdk_2016-01-15-08.txt
val.txt

Thanks
Manisha

[JonathanHenson]

Looks like we have a bug in the logger. Have you ever run this with logging turned off?

[msinha31]

Do you want me to buid aws libraries with cmake -DCMAKE_BUILD_TYPE=Release
and reinstall
-Manisha

[JonathanHenson]

no, no need for that, I'm just asking if you have ever run your program without the logging turned on?

[msinha31]

Yes I did run without logging also earlier but I will try again and send you the valgrind trace I hope this may help you
-Manisha

[JonathanHenson]

yes, if it is the logging then you shouldn't have this:

Thread 36:
==21516== Syscall param write(buf) points to uninitialised byte(s)
==21516== at 0x59211CD: ??? (syscall-template.S:81)
==21516== by 0x4EAF845: (anonymous namespace)::xwrite(int, char const_, long) (basic_file.cc:120)
==21516== by 0x4EE7BB7: std::basic_filebuf<char, std::char_traits >::M_convert_to_external(char, long) (fstream.tcc:521)
==21516== by 0x4EE8612: std::basic_filebuf<char, std::char_traits >::overflow(int) (fstream.tcc:448)
==21516== by 0x4EE614E: std::basic_filebuf<char, std::char_traits >::sync() (fstream.tcc:900)
==21516== by 0x4EC7FAD: pubsync (streambuf:271)
==21516== by 0x4EC7FAD: std::ostream::flush() (ostream.tcc:219)
==21516== by 0x54A6AF2: LogThread(Aws::Utils::Logging::DefaultLogSystem::LogSynchronizationData_, std::shared_ptrstd::ostream const&, std::basic_string<char, std::char_traits, Aws::Allocator > const&, bool) (in /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so)
==21516== by 0x54A790B: std::thread::Impl<std::Bind_simple<void ((Aws::Utils::Logging::DefaultLogSystem::LogSynchronizationData, std::shared_ptr<std::basic_ofstream<char, std::char_traits > >, std::basic_string<char, std::char_traits, Aws::Allocator >, bool))(Aws::Utils::Logging::DefaultLogSystem::LogSynchronizationData_, std::shared_ptrstd::ostream const&, std::basic_string<char, std::char_traits, Aws::Allocator > const&, bool)> >::_M_run() (in /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so)
==21516== by 0x4EE5B9F: execute_native_thread_routine (thread.cc:84)
==21516== by 0x591ADC4: start_thread (pthread_create.c:308)
==21516== by 0x71F0BDC: clone (clone.S:113)
==21516== Address 0x14372657 is 55 bytes inside a block of size 8,192 alloc'd
==21516== at 0x4C297AA: operator new[](unsigned long) (vg_replace_malloc.c:422)
==21516== by 0x4EE79EB: std::basic_filebuf<char, std::char_traits >::M_allocate_internal_buffer() (fstream.tcc:54)
==21516== by 0x4EE8001: std::basic_filebuf<char, std::char_traits >::open(char const, std::_Ios_Openmode) (fstream.tcc:102)
==21516== by 0x54A66CB: MakeDefaultLogFile(std::basic_string<char, std::char_traits, Aws::Allocator >) (in /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so)
==21516== by 0x54A736B: Aws::Utils::Logging::DefaultLogSystem::DefaultLogSystem(Aws::Utils::Logging::LogLevel, std::basic_string<char, std::char_traits, Aws::Allocator > const&) (in /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so)
==21516== by 0x828832: construct<Aws::Utils::Logging::DefaultLogSystem, Aws::Utils::Logging::LogLevel, char const (&)[9]> (new_allocator.h:120)
==21516== by 0x828832: _S_construct<Aws::Utils::Logging::DefaultLogSystem, Aws::Utils::Logging::LogLevel, char const (&)[9]> (alloc_traits.h:254)
==21516== by 0x828832: construct<Aws::Utils::Logging::DefaultLogSystem, Aws::Utils::Logging::LogLevel, char const (&)[9]> (alloc_traits.h:393)
==21516== by 0x828832: _Sp_counted_ptr_inplace<Aws::Utils::Logging::LogLevel, char const (&)[9]> (shared_ptr_base.h:399)
==21516== by 0x828832: construct<std::_Sp_counted_ptr_inplace<Aws::Utils::Logging::DefaultLogSystem, Aws::AllocatorAws::Utils::Logging::DefaultLogSystem, (__gnu_cxx::_Lock_policy)2u>, const Aws::AllocatorAws::Utils::Logging::DefaultLogSystem, Aws::Utils::Logging::LogLevel, char const (&)[9]> (new_allocator.h:120)
==21516== by 0x828832: _S_construct<std::_Sp_counted_ptr_inplace<Aws::Utils::Logging::DefaultLogSystem, Aws::AllocatorAws::Utils::Logging::DefaultLogSystem, (__gnu_cxx::_Lock_policy)2u>, const Aws::AllocatorAws::Utils::Logging::DefaultLogSystem, Aws::Utils::Logging::LogLevel, char const (&)[9]> (alloc_traits.h:254)
==21516== by 0x828832: construct<std::_Sp_counted_ptr_inplace<Aws::Utils::Logging::DefaultLogSystem, Aws::AllocatorAws::Utils::Logging::DefaultLogSystem, (__gnu_cxx::_Lock_policy)2u>, const Aws::AllocatorAws::Utils::Logging::DefaultLogSystem, Aws::Utils::Logging::LogLevel, char const (&)[9]> (alloc_traits.h:393)
==21516== by 0x828832: _shared_count<Aws::Utils::Logging::DefaultLogSystem, Aws::AllocatorAws::Utils::Logging::DefaultLogSystem, Aws::Utils::Logging::LogLevel, char const (&)[9]> (shared_ptr_base.h:502)
==21516== by 0x828832: shared_ptrAws::Allocator<Aws::Utils::Logging::DefaultLogSystem, Aws::Utils::Logging::LogLevel, char const (&)[9]> (shared_ptr_base.h:957)
==21516== by 0x828832: shared_ptrAws::Allocator<Aws::Utils::Logging::DefaultLogSystem, Aws::Utils::Logging::LogLevel, char const (&)[9]> (shared_ptr.h:316)
==21516== by 0x828832: allocate_shared<Aws::Utils::Logging::DefaultLogSystem, Aws::AllocatorAws::Utils::Logging::DefaultLogSystem, Aws::Utils::Logging::LogLevel, char const (&)[9]> (shared_ptr.h:598)
==21516== by 0x828832: std::shared_ptrAws::Utils::Logging::DefaultLogSystem Aws::MakeShared<Aws::Utils::Logging::DefaultLogSystem, Aws::Utils::Logging::LogLevel, char const (&) [9]>(char const, Aws::Utils::Logging::LogLevel&&, char const (&) [9]) (AWSAllocator.h:97)
==21516== by 0x82688E: boss::Dig_CP::getScores::getScoresWRTWTP(std::string, std::shared_ptrAws::DynamoDB::DynamoDBClient) (Dig_CP_getScores.cpp:141)
==21516== by 0x820942: boss::Dig_CP::Dig_CP_LineItemBasic::bid(boss::Dig_CP::Dig_CP_BidderRequest const&) (Dig_CP_LineItems.cc:187)
==21516== by 0x8117D6: boss::Dig_CP::Dig_CP_LineItemController::bid(boss::Dig_CP::Dig_CP_BidderRequest const&) (Dig_CP_LineItemController.cc:64)
==21516== by 0x80B3C4: boss::Dig_CP::Dig_CP_Responsor::processBidderRequest(bufferevent) (Dig_CP_Responsor.cc:360)
==21516== by 0x80C2B8: boss::Dig_CP::Dig_CP_Responsor::handleBidder(bufferevent, void*) (Dig_CP_Responsor.cc:189)
==21516== by 0x5B49DA9: bufferevent_readcb (bufferevent_sock.c:183)

[msinha31]

With log OFF this is the core valgrind output
val.txt

[JonathanHenson]

We are overwriting a stack function return pointer somewhere. Do you have anywhere in your code where you memcpy or memmove etc..., any chance I could take a peek at those places?

[msinha31]

Yes we are doing memmove and memcpy Oh this old code.. I will forward you the codes if it helps
-Manisha

[msinha31]

I didnt hear from you do you want me to share code

[msinha31]

5 0x00007f0d06096e20 in ?? ()
#6 0x00007f0d135e2458 in ?? () from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#7 0x00000000006c5381 in std::basic_string<char, std::char_traits, Aws::Allocator >::reserve (this=0x2341620, __res=)
at /usr/include/c++/4.8.3/bits/basic_string.tcc:510
#8 0x00007f0d135a5365 in Aws::External::Json::Reader::decodeString(Aws::External::Json::Reader::Token&, std::basic_string<char, std::char_traits, Aws::Allocator >&)
() from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#9 0x00007f0d135a58e9 in Aws::External::Json::Reader::decodeString(Aws::External::Json::Reader::Token&) () from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#10 0x00007f0d135a71fd in Aws::External::Json::Reader::readValue() () from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#11 0x00007f0d135a7082 in Aws::External::Json::Reader::rea

[msinha31]

Finally I managed to fix
wget http://curl.haxx.se/download/curl-7.47.0.tar.lzma
tar --lzma -xvf curl-7.47.0.tar.lzma
cd curl-7.47.0
./configure --disable-static --enable-threaded-resolver
make
sudo make install

Finally this did the trick.
-Manisha

[JonathanHenson]

wierd. Do you remember which version of curl you were using and with which options so that we can advise customers when they encounter similar problems?

Thanks for the update and I'm sorry you had such a hard time getting things working.