[lldb][clang] A lot expression/import-std-module tests failed after #135119 in clang

[slydiman]

After the patch #135119 (comment)
the following lldb API tests failed because of the assert in ASTContext.cpp, line 5619:

  lldb-api :: commands/expression/import-std-module/array/TestArrayFromStdModule.py
  lldb-api :: commands/expression/import-std-module/deque-basic/TestDequeFromStdModule.py
  lldb-api :: commands/expression/import-std-module/deque-dbg-info-content/TestDbgInfoContentDequeFromStdModule.py
  lldb-api :: commands/expression/import-std-module/forward_list-dbg-info-content/TestDbgInfoContentForwardListFromStdModule.py
  lldb-api :: commands/expression/import-std-module/forward_list/TestForwardListFromStdModule.py
  lldb-api :: commands/expression/import-std-module/iterator/TestIteratorFromStdModule.py
  lldb-api :: commands/expression/import-std-module/list-dbg-info-content/TestDbgInfoContentListFromStdModule.py
  lldb-api :: commands/expression/import-std-module/list/TestListFromStdModule.py
  lldb-api :: commands/expression/import-std-module/non-module-type-separation/TestNonModuleTypeSeparation.py
  lldb-api :: commands/expression/import-std-module/shared_ptr-dbg-info-content/TestSharedPtrDbgInfoContentFromStdModule.py
  lldb-api :: commands/expression/import-std-module/shared_ptr/TestSharedPtrFromStdModule.py
  lldb-api :: commands/expression/import-std-module/vector-dbg-info-content/TestDbgInfoContentVectorFromStdModule.py
  lldb-api :: commands/expression/import-std-module/vector-of-vectors/TestVectorOfVectorsFromStdModule.py
  lldb-api :: commands/expression/import-std-module/weak_ptr-dbg-info-content/TestDbgInfoContentWeakPtrFromStdModule.py
  lldb-api :: commands/expression/import-std-module/weak_ptr/TestWeakPtrFromStdModule.py

At least the following buildbots are red: lldb-remote-linux-ubuntu, lldb-remote-linux-win.
Should we revert this patch?

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Dmitry Vasilyev (slydiman)

After the patch https://github.com//pull/135119#issuecomment-2794910840 the following lldb API tests failed because of the assert in ASTContext.cpp, line 5619: ``` lldb-api :: commands/expression/import-std-module/array/TestArrayFromStdModule.py lldb-api :: commands/expression/import-std-module/deque-basic/TestDequeFromStdModule.py lldb-api :: commands/expression/import-std-module/deque-dbg-info-content/TestDbgInfoContentDequeFromStdModule.py lldb-api :: commands/expression/import-std-module/forward_list-dbg-info-content/TestDbgInfoContentForwardListFromStdModule.py lldb-api :: commands/expression/import-std-module/forward_list/TestForwardListFromStdModule.py lldb-api :: commands/expression/import-std-module/iterator/TestIteratorFromStdModule.py lldb-api :: commands/expression/import-std-module/list-dbg-info-content/TestDbgInfoContentListFromStdModule.py lldb-api :: commands/expression/import-std-module/list/TestListFromStdModule.py lldb-api :: commands/expression/import-std-module/non-module-type-separation/TestNonModuleTypeSeparation.py lldb-api :: commands/expression/import-std-module/shared_ptr-dbg-info-content/TestSharedPtrDbgInfoContentFromStdModule.py lldb-api :: commands/expression/import-std-module/shared_ptr/TestSharedPtrFromStdModule.py lldb-api :: commands/expression/import-std-module/vector-dbg-info-content/TestDbgInfoContentVectorFromStdModule.py lldb-api :: commands/expression/import-std-module/vector-of-vectors/TestVectorOfVectorsFromStdModule.py lldb-api :: commands/expression/import-std-module/weak_ptr-dbg-info-content/TestDbgInfoContentWeakPtrFromStdModule.py lldb-api :: commands/expression/import-std-module/weak_ptr/TestWeakPtrFromStdModule.py ``` At least the following buildbots are red: [lldb-remote-linux-ubuntu](https://lab.llvm.org/buildbot/#/builders/195/builds/7429), [lldb-remote-linux-win](https://lab.llvm.org/buildbot/#/builders/197/builds/3835). Should we revert this patch?

[slydiman]

cc @mizvekov, @JDevlieghere

[Michael137]

Yes i think we should since it's been causing the bots to be red for a while now. If people need help reproducing this, feel free to ping us here or on the original PR

[slydiman]

#135354
Thanks

[llvmbot]

@llvm/issue-subscribers-clang-frontend

Author: Dmitry Vasilyev (slydiman)

After the patch https://github.com//pull/135119#issuecomment-2794910840 the following lldb API tests failed because of the assert in ASTContext.cpp, line 5619: ``` lldb-api :: commands/expression/import-std-module/array/TestArrayFromStdModule.py lldb-api :: commands/expression/import-std-module/deque-basic/TestDequeFromStdModule.py lldb-api :: commands/expression/import-std-module/deque-dbg-info-content/TestDbgInfoContentDequeFromStdModule.py lldb-api :: commands/expression/import-std-module/forward_list-dbg-info-content/TestDbgInfoContentForwardListFromStdModule.py lldb-api :: commands/expression/import-std-module/forward_list/TestForwardListFromStdModule.py lldb-api :: commands/expression/import-std-module/iterator/TestIteratorFromStdModule.py lldb-api :: commands/expression/import-std-module/list-dbg-info-content/TestDbgInfoContentListFromStdModule.py lldb-api :: commands/expression/import-std-module/list/TestListFromStdModule.py lldb-api :: commands/expression/import-std-module/non-module-type-separation/TestNonModuleTypeSeparation.py lldb-api :: commands/expression/import-std-module/shared_ptr-dbg-info-content/TestSharedPtrDbgInfoContentFromStdModule.py lldb-api :: commands/expression/import-std-module/shared_ptr/TestSharedPtrFromStdModule.py lldb-api :: commands/expression/import-std-module/vector-dbg-info-content/TestDbgInfoContentVectorFromStdModule.py lldb-api :: commands/expression/import-std-module/vector-of-vectors/TestVectorOfVectorsFromStdModule.py lldb-api :: commands/expression/import-std-module/weak_ptr-dbg-info-content/TestDbgInfoContentWeakPtrFromStdModule.py lldb-api :: commands/expression/import-std-module/weak_ptr/TestWeakPtrFromStdModule.py ``` At least the following buildbots are red: [lldb-remote-linux-ubuntu](https://lab.llvm.org/buildbot/#/builders/195/builds/7429), [lldb-remote-linux-win](https://lab.llvm.org/buildbot/#/builders/197/builds/3835). Should we revert this patch?

[mizvekov]

How can I get a stack trace from this? The buildbots don't seem to include it for some reason.

I can't build lldb locally to test this either. On macOS machine I get:

[5560/5970 63.297]Building C object llvm/tools/lldb/tool...ce/CMakeFiles/lldbDebugserverCommon.dir/mach_excUser.c.o
FAILED: llvm/tools/lldb/tools/debugserver/source/CMakeFiles/lldbDebugserverCommon.dir/mach_excUser.c.o 
CCACHE_DIR=/Users/mizvekov/Repos/llvm/cache/c CCACHE_MAXSIZE=100G CCACHE_CPP2=yes CCACHE_HASHDIR=yes /opt/homebrew/bin/ccache /usr/bin/clang -DGTEST_HAS_RTTI=0 -DHAVE_ROUND -DLLDB_USE_OS_LOG -D_DEBUG -D_GLIBCXX_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/Users/mizvekov/Repos/llvm/3/build/rwd/llvm/tools/lldb/tools/debugserver/source -I/Users/mizvekov/Repos/llvm/3/lldb/tools/debugserver/source -I/Users/mizvekov/Repos/llvm/3/lldb/include -I/Users/mizvekov/Repos/llvm/3/build/rwd/llvm/tools/lldb/include -I/Users/mizvekov/Repos/llvm/3/build/rwd/llvm/include -I/Users/mizvekov/Repos/llvm/3/llvm/include -I/opt/homebrew/opt/python@3.13/Frameworks/Python.framework/Versions/3.13/include/python3.13 -I/Users/mizvekov/Repos/llvm/3/llvm/../clang/include -I/Users/mizvekov/Repos/llvm/3/build/rwd/llvm/tools/lldb/../clang/include -I/opt/homebrew/include -I/Users/mizvekov/Repos/llvm/3/build/rwd/llvm/tools/lldb/tools/debugserver/source/.. -I/Users/mizvekov/Repos/llvm/3/lldb/source -I/Users/mizvekov/Repos/llvm/3/lldb/tools/debugserver/source/MacOSX -I/Users/mizvekov/Repos/llvm/3/lldb/tools/debugserver/source/MacOSX/x86_64 -I/Users/mizvekov/Repos/llvm/3/lldb/tools/debugserver/source/MacOSX/arm64 -fPIC -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -O2 -g -DNDEBUG -arch arm64 -UNDEBUG -MD -MT llvm/tools/lldb/tools/debugserver/source/CMakeFiles/lldbDebugserverCommon.dir/mach_excUser.c.o -MF llvm/tools/lldb/tools/debugserver/source/CMakeFiles/lldbDebugserverCommon.dir/mach_excUser.c.o.d -o llvm/tools/lldb/tools/debugserver/source/CMakeFiles/lldbDebugserverCommon.dir/mach_excUser.c.o -c /Users/mizvekov/Repos/llvm/3/build/rwd/llvm/tools/lldb/tools/debugserver/source/mach_excUser.c
clang: error: no such file or directory: '/Users/mizvekov/Repos/llvm/3/build/rwd/llvm/tools/lldb/tools/debugserver/source/mach_excUser.c'
clang: error: no input files
``

[slydiman]

How can I get a stack trace from this?

Note building the debug version is 10x longer. So buildbots use release builds with asserts and the stack trace does not contain symbols. It is hard to get the binaries from buildbots. They must be copied manually within a few minutes for a specific build.

I can't build lldb locally to test this either. On macOS machine I get:

Probably @JDevlieghere or @Michael137 can help with macOS build.

[mizvekov]

I could work around that problem by adding -DLLDB_USE_SYSTEM_DEBUGSERVER=ON.

But now I get a different problem:

Exit Code: 1

Command Output (stdout):
--
lldb version 21.0.0git (git@github.com:llvm/llvm-project.git revision b55bbc9418806494fe133d1b44ef45eff6e01e37)
  clang revision b55bbc9418806494fe133d1b44ef45eff6e01e37
  llvm revision b55bbc9418806494fe133d1b44ef45eff6e01e37
Unable to load lldb extension module.  Possible reasons for this include:
  1) LLDB was built with LLDB_ENABLE_PYTHON=0
  2) PYTHONPATH and PYTHONHOME are not set correctly.  PYTHONHOME should refer to
     the version of Python that LLDB built and linked against, and PYTHONPATH
     should contain the Lib directory for the same python distro, as well as the
     location of LLDB's site-packages folder.
  3) A different version of Python than that which was built against is exported in
     the system's PATH environment variable, causing conflicts.
  4) The executable '/Users/mizvekov/Repos/llvm/3/build/rwd/llvm/bin/lldb' could not be found.  Please check
     that it exists and is executable.

If I add PYTHONHOME=/opt/homebrew/bin/python3 to my environment, I get another error:

cd /Users/mizvekov/Repos/llvm/3/build/rwd/llvm/tools/lldb && /opt/homebrew/bin/cmake -E copy /Users/mizvekov/Repos/llvm/3/build/rwd/llvm/tools/lldb/bindings/python/lldb.py /Users/mizvekov/Repos/llvm/3/build/rwd/./lib/python3.13/site-packages/lldb/__init__.py && cd /Users/mizvekov/Repos/llvm/3/build/rwd/llvm/tools/lldb && /opt/homebrew/bin/cmake -E copy /Users/mizvekov/Repos/llvm/3/lldb/source/Interpreter/embedded_interpreter.py /Users/mizvekov/Repos/llvm/3/build/rwd/./lib/python3.13/site-packages/lldb && cd /Users/mizvekov/Repos/llvm/3/build/rwd/lib/python3.13/site-packages/lldb && /opt/homebrew/bin/cmake -E make_directory formatters/cpp && /opt/homebrew/bin/cmake -E copy /Users/mizvekov/Repos/llvm/3/lldb/examples/synthetic/gnu_libstdcpp.py /Users/mizvekov/Repos/llvm/3/lldb/examples/synthetic/libcxx.py formatters/cpp && /opt/homebrew/Frameworks/Python.framework/Versions/3.13/bin/python3.13 /Users/mizvekov/Repos/llvm/3/lldb/bindings/python/createPythonInit.py formatters/cpp /Users/mizvekov/Repos/llvm/3/lldb/examples/synthetic/gnu_libstdcpp.py /Users/mizvekov/Repos/llvm/3/lldb/examples/synthetic/libcxx.py && cd /Users/mizvekov/Repos/llvm/3/build/rwd/lib/python3.13/site-packages/lldb && /opt/homebrew/bin/cmake -E make_directory formatters && /opt/homebrew/bin/cmake -E copy /Users/mizvekov/Repos/llvm/3/lldb/examples/summaries/cocoa/cache.py /Users/mizvekov/Repos/llvm/3/lldb/examples/summaries/synth.py /Users/mizvekov/Repos/llvm/3/lldb/examples/summaries/cocoa/metrics.py /Users/mizvekov/Repos/llvm/3/lldb/examples/summaries/cocoa/attrib_fromdict.py /Users/mizvekov/Repos/llvm/3/lldb/examples/summaries/cocoa/Logger.py formatters && /opt/homebrew/Frameworks/Python.framework/Versions/3.13/bin/python3.13 /Users/mizvekov/Repos/llvm/3/lldb/bindings/python/createPythonInit.py formatters /Users/mizvekov/Repos/llvm/3/lldb/examples/summaries/cocoa/cache.py /Users/mizvekov/Repos/llvm/3/lldb/examples/summaries/synth.py /Users/mizvekov/Repos/llvm/3/lldb/examples/summaries/cocoa/metrics.py /Users/mizvekov/Repos/llvm/3/lldb/examples/summaries/cocoa/attrib_fromdict.py /Users/mizvekov/Repos/llvm/3/lldb/examples/summaries/cocoa/Logger.py && cd /Users/mizvekov/Repos/llvm/3/build/rwd/lib/python3.13/site-packages/lldb && /opt/homebrew/bin/cmake -E make_directory utils && /opt/homebrew/bin/cmake -E copy /Users/mizvekov/Repos/llvm/3/lldb/examples/python/in_call_stack.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/symbolication.py utils && /opt/homebrew/Frameworks/Python.framework/Versions/3.13/bin/python3.13 /Users/mizvekov/Repos/llvm/3/lldb/bindings/python/createPythonInit.py utils /Users/mizvekov/Repos/llvm/3/lldb/examples/python/in_call_stack.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/symbolication.py && cd /Users/mizvekov/Repos/llvm/3/build/rwd/lib/python3.13/site-packages/lldb && /opt/homebrew/bin/cmake -E make_directory plugins && /opt/homebrew/bin/cmake -E copy /Users/mizvekov/Repos/llvm/3/lldb/examples/python/templates/parsed_cmd.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/templates/scripted_process.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/templates/scripted_platform.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/templates/operating_system.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/templates/scripted_thread_plan.py plugins && /opt/homebrew/Frameworks/Python.framework/Versions/3.13/bin/python3.13 /Users/mizvekov/Repos/llvm/3/lldb/bindings/python/createPythonInit.py plugins /Users/mizvekov/Repos/llvm/3/lldb/examples/python/templates/parsed_cmd.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/templates/scripted_process.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/templates/scripted_platform.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/templates/operating_system.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/templates/scripted_thread_plan.py && cd /Users/mizvekov/Repos/llvm/3/build/rwd/lib/python3.13/site-packages/lldb && /opt/homebrew/bin/cmake -E make_directory macosx && /opt/homebrew/bin/cmake -E copy /Users/mizvekov/Repos/llvm/3/lldb/examples/python/crashlog.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/crashlog_scripted_process.py /Users/mizvekov/Repos/llvm/3/lldb/examples/darwin/heap_find/heap.py macosx && /opt/homebrew/Frameworks/Python.framework/Versions/3.13/bin/python3.13 /Users/mizvekov/Repos/llvm/3/lldb/bindings/python/createPythonInit.py macosx /Users/mizvekov/Repos/llvm/3/lldb/examples/python/crashlog.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/crashlog_scripted_process.py /Users/mizvekov/Repos/llvm/3/lldb/examples/darwin/heap_find/heap.py && cd /Users/mizvekov/Repos/llvm/3/build/rwd/lib/python3.13/site-packages/lldb && /opt/homebrew/bin/cmake -E make_directory macosx/heap && /opt/homebrew/bin/cmake -E copy /Users/mizvekov/Repos/llvm/3/lldb/examples/darwin/heap_find/heap/heap_find.cpp /Users/mizvekov/Repos/llvm/3/lldb/examples/darwin/heap_find/heap/Makefile macosx/heap && cd /Users/mizvekov/Repos/llvm/3/build/rwd/lib/python3.13/site-packages/lldb && /opt/homebrew/bin/cmake -E make_directory diagnose && /opt/homebrew/bin/cmake -E copy /Users/mizvekov/Repos/llvm/3/lldb/examples/python/diagnose_unwind.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/diagnose_nsstring.py diagnose && /opt/homebrew/Frameworks/Python.framework/Versions/3.13/bin/python3.13 /Users/mizvekov/Repos/llvm/3/lldb/bindings/python/createPythonInit.py diagnose /Users/mizvekov/Repos/llvm/3/lldb/examples/python/diagnose_unwind.py /Users/mizvekov/Repos/llvm/3/lldb/examples/python/diagnose_nsstring.py && cd /Users/mizvekov/Repos/llvm/3/build/rwd/lib/python3.13/site-packages/lldb && /opt/homebrew/bin/cmake -E create_symlink ../../../../llvm/lib/liblldb.dylib _lldb.cpython-313-darwin.so && cd /Users/mizvekov/Repos/llvm/3/build/rwd/lib/python3.13/site-packages/lldb && /opt/homebrew/bin/cmake -E create_symlink ../../../../llvm/bin/lldb-argdumper lldb-argdumper
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

[jimingham]

You have to build and run against the same python since lldb uses the "non-binary stable" portions of the Python API. If you didn't specify a Python when you built, the build will use the Python that's in Xcode, and the test suite has to also use that python. macOS puts a shim binary /usr/bin/python3 which is a redirect to the current Xcode's Python, so you don't have to actually find the Python in Xcode.app, just make that shim the python that your path chooses.

Or you can explicitly build against some other python by specifying:

-DPython3_EXECUTABLE=

in the CMake invocation.