[C++][FlightRPC][MSVC] Potential deadlock error after C++ 20 requirement is enabled

[alinaliBQ]

Describe the bug, including details regarding any error messages, version, and platform.

On Windows MSVC, there are flakey mutex deadlock errors occurring in arrow-flight-sql-test.exe after C++ 20 is enabled 30809c6. This error impacts Arrow Flight SQL, it is is flakey and is not specific to any test.

I could not reproduce this error on MSYS2 which is the system tested in the CIs.

Log:

[ RUN      ] TestFlightSqlServer.TestCommandGetExportedKeys
[mutex.cc : 1426] RAW: Potential Mutex deadlock:
        @ 00007FFF10C00260
        @ 00007FFF10BF8EB3
        @ 00007FFF162DCE74
        @ 00007FFF15C12C4C
        @ 00007FFF15C13146
        @ 00007FFF15C14144
        @ 00007FFF156F8608
        @ 00007FFF154CD948
        @ 00007FFF154CDAB9
        @ 00007FFF162E0B26
        @ 00007FF84F09E8D7
        @ 00007FF8500EC53C

[mutex.cc : 1438] RAW: Acquiring absl::Mutex 0000023425BC70E0 while holding  0000023425B4B560; a cycle in the historical lock ordering graph has been observed
[mutex.cc : 1439] RAW: Cycle:
[mutex.cc : 1453] RAW: mutex@0000023425BC70E0 stack:
        @ 00007FFF10C00260
        @ 00007FFF10BF8EB3
        @ 00007FFF1539C999
        @ 00007FFF155DCC5C
        @ 00007FFF15921BEC
        @ 00007FFF159201B1
        @ 00007FFF15D031A3
        @ 00007FFF15B49440
        @ 00007FFF15B4B252
        @ 00007FFF15B4DC99
        @ 00007FFF15B4C2CD
        @ 00007FFF15B4A6F2
        @ 00007FFF15B495EC
        @ 00007FFF15D00323
        @ 00007FFF15CFD795
        @ 00007FFF15CF5BC4
        @ 00007FFF15CE7D3C
        @ 00007FFF15CEB341
        @ 00007FFF159F16EA
        @ 00007FFF159EE94B
        @ 00007FFF161109C3
        @ 00007FFF16110519
        @ 00007FFF16111283
        @ 00007FFF161112E6
        @ 00007FFF162E0B26
        @ 00007FF84F09E8D7
        @ 00007FF8500EC53C

[mutex.cc : 1453] RAW: mutex@0000023425B4B560 stack:
        @ 00007FFF10C00260
        @ 00007FFF10BF8EB3
        @ 00007FFF1539C999
        @ 00007FFF154CC90A
        @ 00007FFF1597861C
        @ 00007FFF15D2CE64
        @ 00007FFF15912BA3
        @ 00007FFF159128EC
        @ 00007FFF155BA141
        @ 00007FFF155BA33F
        @ 00007FFF155BA797
        @ 00007FFF155BCCF5
        @ 00007FFF155BDF9F
        @ 00007FFF15794EB5
        @ 00007FFF15793EA6
        @ 00007FFF155EB22D
        @ 00007FFF15483B81
        @ 00007FFF154548CA
        @ 00007FFF15368EBC
        @ 00007FFF152F4B0A
        @ 00007FF6C0C42D73
        @ 00007FFF24786AED
        @ 00007FFF24786723
        @ 00007FFF2474E76C
        @ 00007FFF2474F474
        @ 00007FFF2474FEF1
        @ 00007FFF247575C6
        @ 00007FFF24786BAD
        @ 00007FFF24786A33
        @ 00007FFF24750793
        @ 00007FFF24812DE3
        @ 00007FFF24812DB5
        @ 00007FF6C0D06809
        @ 00007FF6C0D066B2
        @ 00007FF6C0D0656E
        @ 00007FF6C0D0689E
        @ 00007FF84F09E8D7
        @ 00007FF8500EC53C

[mutex.cc : 1461] RAW: dying due to potential deadlock

To reproduce on PowerShell:

./arrow-flight-sql-test.exe --gtest --gtest_repeat=100

The test executable may need to be run multiple times before the error can show.

Component(s)

FlightRPC, C++

[alinaliBQ]

Hi @pitrou, I see you have worked on deadlock related issues in the commit history, is this something that you might be able to look into? 🙂
cc @lidavidm

[lidavidm]

Are you able to get a backtrace? We don't use absl::Mutex directly so this would seem to be a problem with Protobuf or gRPC instead.

[pitrou]

This may imply that your version of Abseil (and/or Protobuf or gRPC as @lidavidm mentioned) wasn't compiled with the same options as the ones Arrow uses when indirectly including Abseil headers. Google libraries are well-known for compilation options-dependent ABIs.

Another explanation would be that deadlock detection is only enabled with C++20. I have no idea.

[alinaliBQ]

Thanks both for your insights. @lidavidm If Arrow doesn't use absl::Mutex directly then I think it's likely not an issue with Arrow itself. @pitrou I think you are right that deadlock detection might be enabled in C++ 20, because Arrow didn't have deadlock errors before (to my knowledge).
The fix I found for Flight SQL ODBC (#48637) was to disable the deadlock detection with

absl::SetMutexDeadlockDetectionMode(absl::OnDeadlockCycle::kIgnore);

Other things I checked:

• [compile with same option] tried rebuilding the project from scratch but still get same deadlock error.
• [backtrace] couldn't get a backtrace as the error is from upstream. Visual Studio code doesn't report a stack for this either.

It's a good learning experience for me as absl is new to me. I am glad there is a workaround for this issue. Thanks!