lldb TestMultipleDebuggers.py sometimes fails on Linux

[DavidSpickett]

As seen on https://lab.llvm.org/buildbot/#/builders/59/builds/1324 and in Github CI runs. Sometimes this test times out:

TIMEOUT: lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py (1990 of 1990)
******************** TEST 'lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py' FAILED ********************
Script:
--
/usr/bin/python3.8 /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/local/bin/llvm-ar --env OBJCOPY=/usr/bin/llvm-objcopy --env LLVM_LIBS_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./lib --env LLVM_INCLUDE_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/include --env LLVM_TOOLS_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin --arch aarch64 --build-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex --lldb-module-cache-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/lldb --compiler /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/clang --dsymutil /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/dsymutil --llvm-tools-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin --lldb-obj-root /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/tools/lldb --lldb-libs-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./lib /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/api/multiple-debuggers -p TestMultipleDebuggers.py
--
Exit Code: -9
Timeout: Reached timeout of 600 seconds
Command Output (stdout):
--
lldb version 19.0.0git (https://github.com/llvm/llvm-project.git revision a4a8d36b8591801696aad30fa116669c7bb2348c)
  clang revision a4a8d36b8591801696aad30fa116669c7bb2348c
  llvm revision a4a8d36b8591801696aad30fa116669c7bb2348c
--
********************
********************
Timed Out Tests (1):
  lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py

My impression is that some change the 9th of July timeframe made this more common, but equally, it could just be a shift in the load on the bots themselves.

Also, running this on AArch64 I realised that none of the child processes start properly so it needs restricting to x86_64 only, or to use the host architecture name when starting the debugger.

[DavidSpickett]

#101169 to update the test to work properly on AArch64 and re-enable it.

I wasn't able to reproduce the timeout myself so if it happens again I'll disable it again. It is under a "stresstest" category, so losing Linux coverage here isn't the end of the world.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: David Spickett (DavidSpickett)

As seen on https://lab.llvm.org/buildbot/#/builders/59/builds/1324 and in Github CI runs. Sometimes this test times out: ``` TIMEOUT: lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py (1990 of 1990) ******************** TEST 'lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py' FAILED ******************** Script: -- /usr/bin/python3.8 /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/local/bin/llvm-ar --env OBJCOPY=/usr/bin/llvm-objcopy --env LLVM_LIBS_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./lib --env LLVM_INCLUDE_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/include --env LLVM_TOOLS_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin --arch aarch64 --build-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex --lldb-module-cache-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/lldb --compiler /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/clang --dsymutil /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/dsymutil --llvm-tools-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin --lldb-obj-root /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/tools/lldb --lldb-libs-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./lib /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/api/multiple-debuggers -p TestMultipleDebuggers.py -- Exit Code: -9 Timeout: Reached timeout of 600 seconds Command Output (stdout): -- lldb version 19.0.0git (https://github.com/llvm/llvm-project.git revision a4a8d36) clang revision a4a8d36 llvm revision a4a8d36 -- ******************** ******************** Timed Out Tests (1): lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py ```

My impression is that some change the 9th of July timeframe made this more common, but equally, it could just be a shift in the load on the bots themselves.

Also, running this on AArch64 I realised that none of the child processes start properly so it needs restricting to x86_64 only, or to use the host architecture name when starting the debugger.

[DavidSpickett]

1 report in the last week from AArch64 Linux, but it failed to set up the test instead of timing out:

FAIL: LLDB (/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/bin/clang-aarch64) :: test_multiple_debuggers (TestMultipleDebuggers.TestMultipleSimultaneousDebuggers)
======================================================================
ERROR: test_multiple_debuggers (TestMultipleDebuggers.TestMultipleSimultaneousDebuggers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/decorators.py", line 148, in wrapper
    return func(*args, **kwargs)
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/decorators.py", line 148, in wrapper
    return func(*args, **kwargs)
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/api/multiple-debuggers/TestMultipleDebuggers.py", line 35, in test_multiple_debuggers
    subprocess.check_call([self.driver_exe, self.inferior_exe])
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/api/multiple-debuggers/TestMultipleDebuggers.test_multiple_debuggers/multi-process-driver', '/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/api/multiple-debuggers/TestMultipleDebuggers.test_multiple_debuggers/testprog']' died with <Signals.SIGSEGV: 11>.
Config=aarch64-/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/bin/clang
----------------------------------------------------------------------

https://lab.llvm.org/buildbot/#/builders/59/builds/2798

I've never seen this before, it only happened once, and it's not the timeout issue that was much more common previously. So I will close this issue, the link to it remains in the test file for reference if these timeouts return.