Skip to content

Commit 32e90bb

Browse files
[lldb][test] Support remote run of Shell tests (#95986)
1. This commit adds LLDB_TEST_PLATFORM_URL, LLDB_TEST_SYSROOT, LLDB_TEST_PLATFORM_WORKING_DIR, LLDB_SHELL_TESTS_DISABLE_REMOTE cmake flags to pass arguments for cross-compilation and remote running of both Shell&API tests. 2. To run Shell tests remotely, it adds 'platform select' and 'platform connect' commands to %lldb substitution. 3. 'remote-linux' feature added to lit to disable tests failing with remote execution. 4. A separate working directory is assigned to each test to avoid conflicts during parallel test execution. 5. Remote Shell testing is run only when LLDB_TEST_SYSROOT is set for building test sources. The recommended compiler for that is Clang. --------- Co-authored-by: Vladimir Vereschaka <[email protected]>
1 parent 93bfa78 commit 32e90bb

11 files changed

+136
-31
lines changed

lldb/docs/resources/test.rst

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,8 @@ something like ``target.BreakpointCreateByName`` [#]_.
6060
A good rule of thumb is to prefer shell tests when what is being tested is
6161
relatively simple. Expressivity is limited compared to the API tests, which
6262
means that you have to have a well-defined test scenario that you can easily
63-
match with ``FileCheck``.
63+
match with ``FileCheck``. Though Shell tests can be run remotely, behavior
64+
specific to remote debugging must be tested with API tests instead.
6465

6566
Another thing to consider are the binaries being debugged, which we call
6667
inferiors. For shell tests, they have to be relatively simple. The
@@ -592,15 +593,17 @@ test suite, but there are two things to have in mind:
592593
multiple connections. For more information on how to setup remote debugging
593594
see the Remote debugging page.
594595
2. You must tell the test-suite how to connect to the remote system. This is
595-
achieved using the ``--platform-name``, ``--platform-url`` and
596-
``--platform-working-dir`` parameters to ``dotest.py``. These parameters
597-
correspond to the platform select and platform connect LLDB commands. You
598-
will usually also need to specify the compiler and architecture for the
599-
remote system.
600-
601-
Currently, running the remote test suite is supported only with ``dotest.py`` (or
602-
dosep.py with a single thread), but we expect this issue to be addressed in the
603-
near future.
596+
achieved using the ``LLDB_TEST_PLATFORM_URL``, ``LLDB_TEST_PLATFORM_WORKING_DIR``
597+
flags to cmake, and ``--platform-name`` parameter to ``dotest.py``.
598+
These parameters correspond to the platform select and platform connect
599+
LLDB commands. You will usually also need to specify the compiler and
600+
architecture for the remote system.
601+
3. Remote Shell tests execution is currently supported only for Linux target
602+
platform. It's triggered when ``LLDB_TEST_SYSROOT`` is provided for building
603+
test sources. It can be disabled by setting ``LLDB_TEST_SHELL_DISABLE_REMOTE=On``.
604+
Shell tests are not guaranteed to pass against remote target if the compiler
605+
being used is other than Clang.
606+
604607

605608
Running tests in QEMU System Emulation Environment
606609
``````````````````````````````````````````````````

lldb/test/API/lit.cfg.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,13 @@ def delete_module_cache(path):
303303
# In particular, (1) is visited at the top of the file, since the script
304304
# derives other information from it.
305305

306+
if is_configured("lldb_platform_url"):
307+
dotest_cmd += ["--platform-url", config.lldb_platform_url]
308+
if is_configured("lldb_platform_working_dir"):
309+
dotest_cmd += ["--platform-working-dir", config.lldb_platform_working_dir]
310+
if is_configured("cmake_sysroot"):
311+
dotest_cmd += ["--sysroot", config.cmake_sysroot]
312+
306313
if is_configured("dotest_user_args_str"):
307314
dotest_cmd.extend(config.dotest_user_args_str.split(";"))
308315

lldb/test/API/lit.site.cfg.py.in

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,9 @@ config.lua_executable = "@Lua_EXECUTABLE@"
2424
config.lua_test_entry = "TestLuaAPI.py"
2525
config.dotest_common_args_str = lit_config.substitute("@LLDB_TEST_COMMON_ARGS@")
2626
config.dotest_user_args_str = lit_config.substitute("@LLDB_TEST_USER_ARGS@")
27+
config.lldb_platform_url = lit_config.substitute("@LLDB_TEST_PLATFORM_URL@")
28+
config.lldb_platform_working_dir = lit_config.substitute("@LLDB_TEST_PLATFORM_WORKING_DIR@")
29+
config.cmake_sysroot = lit_config.substitute("@LLDB_TEST_SYSROOT@" or "@DEFAULT_SYSROOT@")
2730
config.lldb_enable_python = @LLDB_ENABLE_PYTHON@
2831
config.dotest_lit_args_str = None
2932
config.enabled_plugins = []

lldb/test/CMakeLists.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,8 @@ if (CMAKE_SIZEOF_VOID_P EQUAL 8)
235235
set(LLDB_IS_64_BITS 1)
236236
endif()
237237

238+
set(LLDB_TEST_SHELL_DISABLE_REMOTE OFF CACHE BOOL "Disable remote Shell tests execution")
239+
238240
# These values are not canonicalized within LLVM.
239241
llvm_canonicalize_cmake_booleans(
240242
LLDB_BUILD_INTEL_PT
@@ -244,6 +246,7 @@ llvm_canonicalize_cmake_booleans(
244246
LLVM_ENABLE_ZLIB
245247
LLVM_ENABLE_SHARED_LIBS
246248
LLDB_HAS_LIBCXX
249+
LLDB_TEST_SHELL_DISABLE_REMOTE
247250
LLDB_TOOL_LLDB_SERVER_BUILD
248251
LLDB_USE_SYSTEM_DEBUGSERVER
249252
LLDB_IS_64_BITS)

lldb/test/Shell/Expr/TestIRMemoryMap.test

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# UNSUPPORTED: system-windows
1+
# UNSUPPORTED: system-windows, remote-linux
22

33
# RUN: %clangxx_host %p/Inputs/call-function.cpp -g -o %t
44

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,3 @@
11
# RUN: %lldb -x -b -o 'settings set interpreter.echo-comment-commands true' -s %S/Inputs/EchoCommandsTest.in | FileCheck %S/Inputs/EchoCommandsAll.out
22
# RUN: %lldb -x -b -o 'settings set interpreter.echo-comment-commands false' -s %S/Inputs/EchoCommandsTest.in | FileCheck %S/Inputs/EchoCommandsNoComments.out
33
# RUN: %lldb -x -b -o 'settings set interpreter.echo-commands false' -s %S/Inputs/EchoCommandsTest.in | FileCheck %S/Inputs/EchoCommandsNone.out
4-
5-
RUN: echo start >%t.file
6-
RUN: %lldb -x -b --source-quietly -s %S/Inputs/EchoCommandsTest.in >>%t.file
7-
RUN: echo done >>%t.file
8-
RUN: FileCheck %S/Inputs/EchoCommandsQuiet.out <%t.file
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Can't mute platform select command.
2+
UNSUPPORTED: remote-linux
3+
RUN: echo start >%t.file
4+
RUN: %lldb -x -b --source-quietly -s %S/Inputs/EchoCommandsTest.in >>%t.file
5+
RUN: echo done >>%t.file
6+
RUN: FileCheck %S/Inputs/EchoCommandsQuiet.out <%t.file

lldb/test/Shell/Target/target-label.test

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,16 +4,16 @@
44

55
target create -l "ls" /bin/ls
66
target list
7-
# CHECK: * target #0 (ls): /bin/ls
7+
# CHECK: * target #0 (ls): [[LS_PATH:.*]]
88

99
script lldb.target.SetLabel("")
1010
target list
11-
# CHECK: * target #0: /bin/ls
11+
# CHECK: * target #0: [[LS_PATH]]
1212

1313
target create -l "cat" /bin/cat
1414
target list
15-
# CHECK: target #0: /bin/ls
16-
# CHECK-NEXT: * target #1 (cat): /bin/cat
15+
# CHECK: target #0: [[LS_PATH]]
16+
# CHECK-NEXT: * target #1 (cat): [[CAT_PATH:.*]]
1717

1818
target create -l "cat" /bin/cat
1919
# CHECK: Cannot use label 'cat' since it's set in target #1.
@@ -22,12 +22,12 @@ target create -l 42 /bin/cat
2222
# CHECK: error: Cannot use integer as target label.
2323

2424
target select 0
25-
# CHECK: * target #0: /bin/ls
26-
# CHECK-NEXT: target #1 (cat): /bin/cat
25+
# CHECK: * target #0: [[LS_PATH]]
26+
# CHECK-NEXT: target #1 (cat): [[CAT_PATH]]
2727

2828
target select cat
29-
# CHECK: target #0: /bin/ls
30-
# CHECK-NEXT: * target #1 (cat): /bin/cat
29+
# CHECK: target #0: [[LS_PATH]]
30+
# CHECK-NEXT: * target #1 (cat): [[CAT_PATH]]
3131

3232
script lldb.target.GetLabel()
3333
# CHECK: 'cat'
@@ -36,5 +36,5 @@ script lldb.debugger.GetTargetAtIndex(0).SetLabel('Not cat')
3636
# CHECK: success
3737

3838
target list
39-
# CHECK: target #0 (Not cat): /bin/ls
40-
# CHECK-NEXT: * target #1 (cat): /bin/cat
39+
# CHECK: target #0 (Not cat): [[LS_PATH]]
40+
# CHECK-NEXT: * target #1 (cat): [[CAT_PATH]]

lldb/test/Shell/helper/toolchain.py

Lines changed: 79 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,17 @@
11
import os
22
import itertools
33
import platform
4+
import re
45
import subprocess
56
import sys
67

78
import lit.util
9+
from lit.formats import ShTest
810
from lit.llvm import llvm_config
911
from lit.llvm.subst import FindTool
1012
from lit.llvm.subst import ToolSubst
1113

14+
import posixpath
1215

1316
def _get_lldb_init_path(config):
1417
return os.path.join(config.test_exec_root, "lit-lldb-init-quiet")
@@ -22,6 +25,68 @@ def _disallow(config, execName):
2225
config.substitutions.append((" {0} ".format(execName), warning.format(execName)))
2326

2427

28+
def get_lldb_args(config, suffix=""):
29+
lldb_args = []
30+
if "remote-linux" in config.available_features:
31+
lldb_args += [
32+
"-O",
33+
'"platform select remote-linux"',
34+
"-O",
35+
f'"platform connect {config.lldb_platform_url}"',
36+
]
37+
if config.lldb_platform_working_dir:
38+
dir = posixpath.join(f"{config.lldb_platform_working_dir}", "shell")
39+
if suffix:
40+
dir += posixpath.join(dir, f"{suffix}")
41+
lldb_args += [
42+
"-O",
43+
f'"platform shell mkdir -p {dir}"',
44+
"-O",
45+
f'"platform settings -w {dir}"',
46+
]
47+
lldb_args += ["--no-lldbinit", "-S", _get_lldb_init_path(config)]
48+
return lldb_args
49+
50+
51+
class ShTestLldb(ShTest):
52+
def __init__(
53+
self, execute_external=False, extra_substitutions=[], preamble_commands=[]
54+
):
55+
super().__init__(execute_external, extra_substitutions, preamble_commands)
56+
57+
def execute(self, test, litConfig):
58+
# Run each Shell test in a separate directory (on remote).
59+
60+
# Find directory change command in %lldb substitution.
61+
for i, t in enumerate(test.config.substitutions):
62+
if re.match(t[0], "%lldb"):
63+
cmd = t[1]
64+
if '-O "platform settings -w ' in cmd:
65+
# If command is present, it is added by get_lldb_args.
66+
# Replace the path with the tests' path in suite.
67+
# Example:
68+
# bin/lldb
69+
# -O "platform shell mkdir -p /home/user/shell"
70+
# -O "platform settings -w /home/user/shell" ...
71+
# =>
72+
# bin/lldb
73+
# -O "platform shell mkdir -p /home/user/shell/SymbolFile/Breakpad/inline-record.test"
74+
# -O "platform settings -w /home/user/shell/SymbolFile/Breakpad/inline-record.test" ...
75+
args_def = " ".join(get_lldb_args(test.config))
76+
args_unique = " ".join(
77+
get_lldb_args(
78+
test.config,
79+
posixpath.join(*test.path_in_suite),
80+
)
81+
)
82+
test.config.substitutions[i] = (
83+
t[0],
84+
cmd.replace(args_def, args_unique),
85+
)
86+
break
87+
return super().execute(test, litConfig)
88+
89+
2590
def use_lldb_substitutions(config):
2691
# Set up substitutions for primary tools. These tools must come from config.lldb_tools_dir
2792
# which is basically the build output directory. We do not want to find these in path or
@@ -34,7 +99,9 @@ def use_lldb_substitutions(config):
3499
build_script = os.path.join(build_script, "build.py")
35100
build_script_args = [
36101
build_script,
37-
"--compiler=any", # Default to best compiler
102+
(
103+
"--compiler=clang" if config.enable_remote else "--compiler=any"
104+
), # Default to best compiler
38105
"--arch=" + str(config.lldb_bitness),
39106
]
40107
if config.lldb_lit_tools_dir:
@@ -56,7 +123,7 @@ def use_lldb_substitutions(config):
56123
ToolSubst(
57124
"%lldb",
58125
command=FindTool("lldb"),
59-
extra_args=["--no-lldbinit", "-S", lldb_init],
126+
extra_args=get_lldb_args(config),
60127
unresolved="fatal",
61128
),
62129
ToolSubst(
@@ -138,7 +205,10 @@ def use_support_substitutions(config):
138205
# Set up substitutions for support tools. These tools can be overridden at the CMake
139206
# level (by specifying -DLLDB_LIT_TOOLS_DIR), installed, or as a last resort, we can use
140207
# the just-built version.
141-
host_flags = ["--target=" + config.host_triple]
208+
if config.enable_remote:
209+
host_flags = ["--target=" + config.target_triple]
210+
else:
211+
host_flags = ["--target=" + config.host_triple]
142212
if platform.system() in ["Darwin"]:
143213
try:
144214
out = subprocess.check_output(["xcrun", "--show-sdk-path"]).strip()
@@ -165,6 +235,12 @@ def use_support_substitutions(config):
165235
if config.cmake_sysroot:
166236
host_flags += ["--sysroot={}".format(config.cmake_sysroot)]
167237

238+
if config.enable_remote and config.has_libcxx:
239+
host_flags += [
240+
"-L{}".format(config.libcxx_libs_dir),
241+
"-lc++",
242+
]
243+
168244
host_flags = " ".join(host_flags)
169245
config.substitutions.append(("%clang_host", "%clang " + host_flags))
170246
config.substitutions.append(("%clangxx_host", "%clangxx " + host_flags))

lldb/test/Shell/lit.cfg.py

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
config.name = "lldb-shell"
2222

2323
# testFormat: The test format to use to interpret tests.
24-
config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell)
24+
config.test_format = toolchain.ShTestLldb(not llvm_config.use_lit_shell)
2525

2626
# suffixes: A list of file extensions to treat as test files. This is overriden
2727
# by individual lit.local.cfg files in the test subdirectories.
@@ -68,6 +68,13 @@
6868
lit_config.note("Running Shell tests in {} mode.".format(lldb_repro_mode))
6969
toolchain.use_lldb_repro_substitutions(config, lldb_repro_mode)
7070

71+
if config.lldb_platform_url and config.cmake_sysroot and config.enable_remote:
72+
if re.match(r".*-linux.*", config.target_triple):
73+
config.available_features.add("remote-linux")
74+
else:
75+
# After this, enable_remote == True iff remote testing is going to be used.
76+
config.enable_remote = False
77+
7178
llvm_config.use_default_substitutions()
7279
toolchain.use_lldb_substitutions(config)
7380
toolchain.use_support_substitutions(config)

lldb/test/Shell/lit.site.cfg.py.in

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,15 @@ config.lldb_src_root = "@LLDB_SOURCE_DIR@"
1010
config.lldb_obj_root = "@LLDB_BINARY_DIR@"
1111
config.lldb_libs_dir = lit_config.substitute("@LLDB_LIBS_DIR@")
1212
config.lldb_tools_dir = lit_config.substitute("@LLDB_TOOLS_DIR@")
13+
config.lldb_platform_url = lit_config.substitute("@LLDB_TEST_PLATFORM_URL@")
14+
config.lldb_platform_working_dir = lit_config.substitute("@LLDB_TEST_PLATFORM_WORKING_DIR@")
1315
# Since it comes from the command line, it may have backslashes which
1416
# should not need to be escaped.
1517
config.lldb_lit_tools_dir = lit_config.substitute(r"@LLDB_LIT_TOOLS_DIR@")
16-
config.cmake_sysroot = lit_config.substitute("@CMAKE_SYSROOT@")
18+
config.cmake_sysroot = lit_config.substitute("@LLDB_TEST_SYSROOT@" or "@DEFAULT_SYSROOT@")
19+
config.has_libcxx = @LLDB_HAS_LIBCXX@
20+
config.enable_remote = not @LLDB_TEST_SHELL_DISABLE_REMOTE@
21+
config.libcxx_libs_dir = "@LIBCXX_LIBRARY_DIR@"
1722
config.target_triple = "@LLVM_TARGET_TRIPLE@"
1823
config.python_executable = "@Python3_EXECUTABLE@"
1924
config.have_zlib = @LLVM_ENABLE_ZLIB@

0 commit comments

Comments
 (0)