Cannot load any more object with static TLS while executing crnn_ctc on multi-devices #1360

wanghaoshuang · 2018-10-15T02:47:32Z

Error log:

  File "/home/users/***/lib/python2.7/site-packages/paddle/fluid/parallel_executor.py", line 260, in run
    self.executor.run(fetch_list, fetch_var_name)
paddle.fluid.core.EnforceNotMet: Failed to find dynamic library: libwarpctc.so ( dlopen: cannot load any more object with static TLS )
 Please specify its path correctly using following ways:
 Method. set environment variable LD_LIBRARY_PATH on Linux or DYLD_LIBRARY_PATH on Mac OS.
 For instance, issue command: export LD_LIBRARY_PATH=...
 Note: After Mac OS 10.11, using the DYLD_LIBRARY_PATH is impossible unless System Integrity Protection (SIP) is disabled. at [/home/users/***/Paddle/paddle/fluid/platform/dynload/dynamic_loader.cc:157]
PaddlePaddle Call Stacks:
0       0x7f635169b626p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 486
1       0x7f6352bc3e9ep paddle::platform::dynload::GetWarpCTCDsoHandle() + 1822
2       0x7f63527b7149p void std::__once_call_impl<std::_Bind_simple<decltype (get_warpctc_version({parm#1}...)) paddle::platform::dynload::DynLoad__get_warpctc_version::operator()<>()::{lambda()#1} ()> >() + 9
3       0x7f63bf6edbe0p pthread_once + 80
4       0x7f63527be4a8p paddle::operators::WarpCTCFunctor<paddle::platform::CUDADeviceContext>::operator()(paddle::framework::ExecutionContext const&, float const*, float*, int const*, int const*, int const*, unsigned long, unsigned long, unsigned long, float*) + 136
5       0x7f63527c0abbp paddle::operators::WarpCTCKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const + 2763
6       0x7f63527c2863p std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::WarpCTCKernel<paddle::platform::CUDADeviceContext, float> >::operator()(char const*, char const*) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) + 35
7       0x7f6352a3f04cp paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 492
8       0x7f6352a3b7bcp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 252
9       0x7f63529494c7p
10      0x7f6352967100p
11      0x7f6352966975p paddle::framework::details::OpHandleBase::RunAndRecordEvent(std::function<void ()> const&) + 805
12      0x7f6352948f9fp paddle::framework::details::ComputationOpHandle::RunImpl() + 95
13      0x7f6352967a05p paddle::framework::details::OpHandleBase::Run(bool) + 117
14      0x7f635290626ap
15      0x7f635176c263p std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) + 35
16      0x7f635176bb17p std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) + 39
17      0x7f63bf6edbe0p pthread_once + 80
18      0x7f6352905242p
19      0x7f635176d7f4p ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const + 404
20      0x7f636b73c070p
21      0x7f63bf6e8df3p
22      0x7f63bed0d2cdp clone + 109

The text was updated successfully, but these errors were encountered:

wanghaoshuang · 2018-10-15T03:00:56Z

baidu-research/warp-ctc#70
tensorflow/models#523

wanghaoshuang · 2018-10-16T02:08:37Z

一个解决方案是：编译的时候关闭mkldnn相关的选项，因为mkldnn相关的库也会占用大量static TLS

wanghaoshuang · 2019-06-03T07:42:31Z

pytorch/pytorch#2083

wanghaoshuang · 2019-08-05T03:03:19Z

建议将mkldnn放在platform/dynload下。

JiabinYang · 2019-08-05T03:05:27Z

Duplicate of #3007

wanghaoshuang · 2019-08-19T08:31:13Z

可以用readelf –l XXX.so 查看一下1.5 release so里program header的TLS：

./third_party/install/mkldnn/lib/libmkldnn.so
Program Headers:
  Type            Offset                              VirtAddr                        PhysAddr
                       FileSiz                             MemSiz                          Flags  Align
  TLS              0x0000000000c38928 0x0000000000e38928 0x0000000000e38928
                      0x0000000000000000 0x0000000000000020  R      8
 
./third_party/install/mklml/lib/libmklml_intel.so
Program Headers:
  Type           Offset                            VirtAddr                             PhysAddr
                      FileSiz                            MemSiz                             Flags  Align
  TLS             0x0000000007a043e8 0x0000000007c043e8   0x0000000007c043e8
                      0x0000000000000000 0x0000000000000020  R      8
 
./paddle/fluid/inference/libpaddle_fluid.so
Program Headers:
  Type           Offset                             VirtAddr                         PhysAddr
                      FileSiz                             MemSiz                          Flags  Align
  TLS             0x0000000001e89020 0x0000000002089020 0x0000000002089020
                      0x0000000000000010 0x00000000000014b0  R      20

Mkldnn/mklml动态库里都只有32字节的TLS；Paddle inference库则有0x14b0=5296字节的TLS。

liushanshan07 · 2019-10-28T09:17:32Z

求问这个是咋解决的？

Dely-Yu · 2019-10-31T02:54:49Z

任务10连跪。。

wanghaoshuang · 2019-10-31T08:57:17Z

目前唯一的解决办法是编译源码安装paddle，编译时把MKL选项关掉：

https://github.com/PaddlePaddle/Paddle/blob/develop/CMakeLists.txt#L65

继续探索更科学的解决办法...

wanghaoshuang · 2019-10-31T09:20:15Z

Failed to find dynamic library: libwarpctc.so ( dlopen: cannot load any more object with static TLS )

Check TLS memory size of libwarpctc.so

Find lbwarpctc.so in python site-packages:

find ${PythonPath}/python2.7/site-packages -name libwarpctc.so

Use readelf -l libwarpctc.so check memory size of TLS:

Elf file type is DYN (Shared object file)
Entry point 0x78f0
There are 6 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x00000000004ec0c4 0x00000000004ec0c4  R E    200000
  LOAD           0x00000000004ec0c8 0x00000000006ec0c8 0x00000000006ec0c8
                 0x0000000000002c40 0x0000000000003de0  RW     200000
  DYNAMIC        0x00000000004ee3f0 0x00000000006ee3f0 0x00000000006ee3f0
                 0x0000000000000240 0x0000000000000240  RW     8
  NOTE           0x0000000000000190 0x0000000000000190 0x0000000000000190
                 0x0000000000000024 0x0000000000000024  R      4
  GNU_EH_FRAME   0x00000000004dff90 0x00000000004dff90 0x00000000004dff90
                 0x000000000000274c 0x000000000000274c  R      4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     10

 Section to Segment mapping:
  Segment Sections...
   00     .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .nv_fatbin .eh_frame_hdr .eh_frame
   01     .init_array .fini_array .jcr .data.rel.ro .dynamic .got .got.plt .data .nvFatBinSegment .bss
   02     .dynamic
   03     .note.gnu.build-id
   04     .eh_frame_hdr
   05

There is no TLS in program headers. So we should check all the so needed by libwarpctc.so.

readelf -d libwarpctc.so

Dynamic section at offset 0x4ee3f0 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgomp.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libwarpctc.so]

ldd libwarpctc.so

linux-vdso.so.1 (0x00007ffd8bda5000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f852982f000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f852962a000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f8529422000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f852911c000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f8528e19000)
	libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00007f8528c0c000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f85289f6000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f8528634000)
	/opt/compiler/gcc-4.8.2/lib64/ld-linux-x86-64.so.2 (0x00007f852a150000)

/usr/lib64/libgomp.so.1 is an openmpi library:

readelf -l /usr/lib64/libgomp.so.1

Elf file type is DYN (Shared object file)
Entry point 0x38c0003640
There are 7 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  TLS            0x000000000000caa0 0x00000038c020caa0 0x00000038c020caa0
                 0x0000000000000000 0x0000000000000064  R      8

/usr/lib64/libgomp.so.1 need 0x0000000000000064 (100bytes) TLS.

0x0000000000000064 (100bytes) TLS is necessary for loading libwarpctc.so.

Check TLS occupied by mkl library

readelf -l libmkldnn.so.0

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  TLS            0x00000000009583b0 0x0000000000b583b0 0x0000000000b583b0
                 0x0000000000000000 0x0000000000000038  R      10

readelf -l libmklml_intel.so

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  TLS            0x0000000007a043e8 0x0000000007c043e8 0x0000000007c043e8
                 0x0000000000000000 0x0000000000000020  R      8

0x0000000000000058(88bytes) TLS is occupied by mkl library.

Check TLS occupied by paddle framework

readelf -l ./paddle1.6/lib/python2.7/site-packages/paddle/libs/libpaddle_framework.so
Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  TLS            0x0000000000b88000 0x0000000000d88000 0x0000000000d88000
                 0x0000000000000020 0x0000000000000430  R      20

readelf -l ./paddle1.6/lib/python2.7/site-packages/paddle/fluid/core_noavx.so
TLS            0x000000001fa1b020 0x000000001fc1b020 0x000000001fc1b020
                 0x0000000000000014 0x00000000000013f0  R      10

 readelf -l ./paddle1.6/lib/python2.7/site-packages/paddle/fluid/core_avx.so
TLS            0x000000001fbf3000 0x000000001fdf3000 0x000000001fdf3000
                 0x0000000000000024 0x0000000000001ad0  R      20

Analyse TLS in core_avx.so:

objdump -t ./paddle1.6/lib/python2.7/site-packages/paddle/fluid/core_avx.so | grep ".tbss" |awk -F" " '{cmd="c++filt "$5; system(cmd); print $4}'

guard variable for paddle::framework::MultiSlotDataFeed::ParseOneInstanceFromPipe(std::vector<paddle::framework::MultiSlotType, std::allocator<paddle::framework::MultiSlotType> >*)::reader
0000000000000008
paddle::framework::MultiSlotDataFeed::ParseOneInstanceFromPipe(std::vector<paddle::framework::MultiSlotType, std::allocator<paddle::framework::MultiSlotType> >*)::reader
0000000000000018
guard variable for paddle::framework::MultiSlotInMemoryDataFeed::ParseOneInstanceFromPipe(paddle::framework::Record*)::reader
0000000000000008
paddle::framework::MultiSlotInMemoryDataFeed::ParseOneInstanceFromPipe(paddle::framework::Record*)::reader
0000000000000018
guard variable for paddle::framework::FleetWrapper::LocalRandomEngine()::r
0000000000000008
paddle::framework::FleetWrapper::LocalRandomEngine()::r
0000000000000008
paddle::platform::g_event_list
0000000000000010
__tls_guard
0000000000000001
paddle::platform::g_mem_event_list
0000000000000010
paddle::platform::g_thread_id
0000000000000004
paddle::platform::g_mem_thread_id
0000000000000004
g_current_thread_worker
0000000000000008
g_current_thread_pollset
0000000000000008
g_current_thread_worker
0000000000000008
g_current_thread_pollset
0000000000000008
g_current_thread_worker
0000000000000008
g_current_thread_pollset
0000000000000008
g_initialized_sigmask
0000000000000001
g_current_thread_polling_island
0000000000000008
g_orig_sigmask
0000000000000080
__tls_guard
0000000000000001
paddle::platform::(anonymous namespace)::cur_input_shape_str
0000000000000008
paddle::platform::(anonymous namespace)::cur_mkldnn_session_id
0000000000000008
__tls_guard
0000000000000001
paddle::platform::(anonymous namespace)::block_id_stack
0000000000000050
paddle::platform::(anonymous namespace)::annotation_stack
0000000000000050
g_this_thread_state
0000000000000008
g_last_seen_min_timer
0000000000000008
g_cached_cq
0000000000000008
g_cached_event
0000000000000008
guard variable for paddle::framework::global_transfer_data_cache()::x
0000000000000008
paddle::framework::global_transfer_data_cache()::x
0000000000000008
guard variable for paddle::framework::global_transfer_scope_cache()::x
0000000000000008
paddle::framework::global_transfer_scope_cache()::x
0000000000000008
gpr_default_log(gpr_log_func_args*)::tid
0000000000000008
g_current_thread_worker
0000000000000008
g_current_thread_poller
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::MatMulTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)9>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)16>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)3>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::CRFDecodingTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)30>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VIdentityTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VExpTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)27>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUH1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)5>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VIdentityTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)6>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSquareTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)15>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart2Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::CRFDecodingTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::HMaxTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::SgdTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)24>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)29>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)25>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)26>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VBroadcastTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VScalTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSigmoidTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMCtHtTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VScalTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)30>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VExpTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::SeqPoolTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)18>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VBroadcastTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart2Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::HSumTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::SoftmaxTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::MatMulTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)17>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::SgdTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUH1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)31>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::StrideASumTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::HSumTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)9>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddReluTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)24>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)7>::Instance()::g_jit_codes
0000000000000008
paddle::platform::DeviceTracerImpl::AddMemInfoRecord(unsigned long, unsigned long, unsigned long, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long)::local_mem_info_record
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMCtHtTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUH1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)31>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart2Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)14>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)8>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)23>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddReluTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)22>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMCtHtTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)4>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSubTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VScalTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)5>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)1>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VMulTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)26>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)20>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::SoftmaxTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)14>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::CRFDecodingTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSigmoidTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddBiasTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VMulTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)13>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)15>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)29>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VReluTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)12>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSubTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::SgdTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSubTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)17>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)13>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)3>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::SeqPoolTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart2Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::SeqPoolTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::platform::DeviceTracerImpl::AddAnnotation(unsigned int, paddle::platform::Event*)::local_correlations_pairs
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)12>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::MatMulTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::HMaxTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)23>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VTanhTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)28>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VBroadcastTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMC1H1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::platform::DeviceTracerImpl::AddActiveKindRecords(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, long, long, unsigned int)::local_active_kind_records
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddBiasTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::MatMulTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::SgdTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::SeqPoolTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)16>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddReluTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)27>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)1>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMC1H1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)11>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::CRFDecodingTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::platform::DeviceTracerImpl::AddCPURecords(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, long, long)::local_cpu_records_
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::NCHW16CMulNCTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSquareTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VReluTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSubTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)20>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VTanhTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)19>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::StrideScalTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::StrideASumTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)19>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMC1H1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)4>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddReluTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)28>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMCtHtTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)6>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)18>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VScalTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)7>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VBroadcastTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)25>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)22>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUH1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)8>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::NCHW16CMulNCTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
grpc_core::ExecCtx::exec_ctx_
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMC1H1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::StrideScalTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSquareTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSquareTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)11>::Instance()::g_jit_codes
0000000000000030

luotao1 · 2019-11-11T08:19:48Z

@bingyanghuang Could you help see it?

wanghaoshuang closed this as completed Jun 5, 2019

wanghaoshuang mentioned this issue Jun 5, 2019

ocr训练找不到libwarpctc.so PaddlePaddle/Paddle#17792

Closed

wanghaoshuang mentioned this issue Aug 2, 2019

Failed to find dynamic library: libwarpctc.so ( dlopen: cannot load any more object with static TLS ) #3007

Open

JiabinYang marked this as a duplicate of #3007 Aug 5, 2019

wanghaoshuang reopened this Sep 24, 2019

AIpioneer assigned wanghaoshuang Oct 31, 2019

AIpioneer added the bug label Oct 31, 2019

luotao1 added the intel label Nov 11, 2019

sneaxiy pinned this issue Nov 11, 2019

qingqing01 unpinned this issue Nov 11, 2019

yihuaxu mentioned this issue Nov 13, 2019

Fix jit tls issue PaddlePaddle/Paddle#21151

Merged

OliverLPH mentioned this issue Dec 16, 2019

import paddle出现ImportError：dlopen：cannot load any more object with static TLS PaddlePaddle/Paddle#21760

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Cannot load any more object with static TLS while executing crnn_ctc on multi-devices #1360

Cannot load any more object with static TLS while executing crnn_ctc on multi-devices #1360

wanghaoshuang commented Oct 15, 2018

wanghaoshuang commented Oct 15, 2018 •

edited

Loading

Uh oh!

wanghaoshuang commented Oct 16, 2018

Uh oh!

wanghaoshuang commented Jun 3, 2019

Uh oh!

wanghaoshuang commented Aug 5, 2019

Uh oh!

JiabinYang commented Aug 5, 2019

Uh oh!

wanghaoshuang commented Aug 19, 2019

Uh oh!

liushanshan07 commented Oct 28, 2019

Uh oh!

Dely-Yu commented Oct 31, 2019

Uh oh!

wanghaoshuang commented Oct 31, 2019 •

edited

Loading

Uh oh!

wanghaoshuang commented Oct 31, 2019 •

edited

Loading

Uh oh!

luotao1 commented Nov 11, 2019

Uh oh!

Cannot load any more object with static TLS while executing crnn_ctc on multi-devices #1360

Cannot load any more object with static TLS while executing crnn_ctc on multi-devices #1360

Comments

wanghaoshuang commented Oct 15, 2018

wanghaoshuang commented Oct 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wanghaoshuang commented Oct 16, 2018

Uh oh!

wanghaoshuang commented Jun 3, 2019

Uh oh!

wanghaoshuang commented Aug 5, 2019

Uh oh!

JiabinYang commented Aug 5, 2019

Uh oh!

wanghaoshuang commented Aug 19, 2019

Uh oh!

liushanshan07 commented Oct 28, 2019

Uh oh!

Dely-Yu commented Oct 31, 2019

Uh oh!

wanghaoshuang commented Oct 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wanghaoshuang commented Oct 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Check TLS memory size of libwarpctc.so

Check TLS occupied by mkl library

Check TLS occupied by paddle framework

Uh oh!

luotao1 commented Nov 11, 2019

Uh oh!

wanghaoshuang commented Oct 15, 2018 •

edited

Loading

wanghaoshuang commented Oct 31, 2019 •

edited

Loading

wanghaoshuang commented Oct 31, 2019 •

edited

Loading