From 5bd179e5ee8b3551313f7931f297dc4bc724bda4 Mon Sep 17 00:00:00 2001 From: Michael Gottesman Date: Wed, 14 Aug 2024 14:38:21 -0700 Subject: [PATCH] [docs] Move LSANFailureSymbolication information to DebuggingTheCompiler.md instead of having its own document. This information is generally centralized in DebuggingTheCompiler.md to make it easier to find. --- docs/DebuggingTheCompiler.md | 70 ++++++++++++++++++++++++++++++++ docs/LSANFailureSymbolication.md | 68 ------------------------------- 2 files changed, 70 insertions(+), 68 deletions(-) delete mode 100644 docs/LSANFailureSymbolication.md diff --git a/docs/DebuggingTheCompiler.md b/docs/DebuggingTheCompiler.md index d74fa12febdf4..e1ea9bc115ca4 100644 --- a/docs/DebuggingTheCompiler.md +++ b/docs/DebuggingTheCompiler.md @@ -37,6 +37,7 @@ benefit of all Swift developers. - [Using git-bisect in the presence of branch forwarding/feature branches](#using-git-bisect-in-the-presence-of-branch-forwardingfeature-branches) - [Reducing SIL test cases using bug_reducer](#reducing-sil-test-cases-using-bug_reducer) - [Disabling PCH Verification](#disabling-pch-verification) + - [Diagnosing LSAN Failures in the Compiler](#diagnosing-lsan-failures-in-the-compiler) - [Debugging the Compiler Build](#debugging-the-compiler-build) - [Build Dry Run](#build-dry-run) - [Debugging the Compiler Driver](#debugging-the-compiler-driver-build) @@ -846,6 +847,75 @@ checking by passing in to swift: NOTE: If there are actual differences in between the on disk PCH format and the format expected by the compiler crashes and undefined behavior may result. +## Diagnosing LSAN Failures in the Compiler + +### Create Ubuntu Container + +1. Use an x86 machine. The following instructions currently don’t work on arm64. It might be easy to adjust them or not, I have not tried +2. Clone (or pull) swift-docker: https://github.com/swiftlang/swift-docker +3. Build the Ubuntu 22.04 container: `cd swift-ci/master/ubuntu/22.04; docker build .` +4. `docker run -it --cpus --memory -v ~/:/src-on-host:cached --name lsan-reproducer --cap-add=SYS_PTRACE --security-opt seccomp=unconfined bash` + - The `-cap-add` and `-security-opt` arguments are needed to run LLDB inside the Docker container +5. Copy the sources to inside the Docker container: `cp -r /src-on-host/* ~` + - We need to to this because the build needs a case-sensitive file system and your host machine probably has a case-insensitive file system + +Build inside the Container + +1. `utils/build-script --preset buildbot_incremental_linux,lsan,tools=RDA,stdlib=DA,test=no` +2. This should reproduce the LSAN failure +3. Now, disassemble the failing CMake invocation to a swiftc invocation. I needed to set one environment variable and could the copy the swiftc invocation (but this might change as the build changes) + +``` +export LD_LIBRARY_PATH=/opt/swift/5.8.1/usr/lib/swift/linux +/home/build-user/build/buildbot_incremental_lsan/swift-linux-x86_64/./bin/swiftc +``` + +### Symbolicating the LSAN report + +For reasons that are not clear to me, LSAN does not symbolicate the report. To get the functions at the reported offsets, perform the following steps (there might be easier steps, please update this document if you know any). + +1. Run the swiftc invocation that fails and copy the leak report to somewhere. The leak report should look like the following. +``` +==3863==ERROR: LeakSanitizer: detected memory leaks + +Direct leak of 120 byte(s) in 3 object(s) allocated from: + #0 0x55b91c0b59b8 (/home/build-user/build/buildbot_incremental_lsan/swift-linux-x86_64/bin/swift-frontend+0x14d09b8) + #1 0x55b91d51281c (/home/build-user/build/buildbot_incremental_lsan/swift-linux-x86_64/bin/swift-frontend+0x292d81c) + #2 0x55b91c1b8700 (/home/build-user/build/buildbot_incremental_lsan/swift-linux-x86_64/bin/swift-frontend+0x15d3700) + +SUMMARY: LeakSanitizer: 120 byte(s) leaked in 3 allocation(s). +``` +2. `lldb -- ` +3. Start running swiftc inside lldb by executing `r` +4. Find the loaded offset of swift-frontend by running `image list` +For example, this might output +``` +[ 0] 0AEA10C1 0x0000555555554000 /home/build-user/build/buildbot_incremental_lsan/swift-linux-x86_64/bin/swift-frontend +[ 1] D52BB67A-BBBB-E429-6E87-FC16144CA7CE-55276DD6 0x00007ffff7ffb000 [vdso] (0x00007ffff7ffb000) +[ 2] 9EA8014C-F020-21A2-9E57-AA3E0512E9BB-6E30541D 0x00007ffff7dd3000 /lib/x86_64-linux-gnu/ld-2.27.so +``` +The loaded offset is `0x0000555555554000` +5. For the frame that you want to symbolicate,, add the offset you computed above to the stack frame in the LSAN report, eg. to symbolicate frame 1 `0x555555554000 + 0x292d81c = 0x555557E8181C` +6. Look up the address using `image lookup -a
`. This should output something like + +``` +(lldb) image lookup -a 0x555557E8181C + Address: swiftc[0x000000000292d81c] (swiftc.PT_LOAD[0]..text + 22056284) + Summary: swiftc`registerFunctionTest(BridgedStringRef, void*) + 28 at SILBridging.cpp:148:3 +``` + +7. Hoorray, you know which function is leaking. + +### Making Local Changes Inside the Container + +For example, to install vim in the container run + +``` +docker exec -u 0:0 -it lsan-reproducer bash +$ apt update +$ apt install vim +``` + # Debugging the Compiler Build ## Build Dry Run diff --git a/docs/LSANFailureSymbolication.md b/docs/LSANFailureSymbolication.md deleted file mode 100644 index 4cefd7e901bc3..0000000000000 --- a/docs/LSANFailureSymbolication.md +++ /dev/null @@ -1,68 +0,0 @@ -# Diagnose LSAN Failures in the Compiler - -### Create Ubuntu Container - -1. Use an x86 machine. The following instructions currently don’t work on arm64. It might be easy to adjust them or not, I have not tried -2. Clone (or pull) swift-docker: https://github.com/swiftlang/swift-docker -3. Build the Ubuntu 22.04 container: `cd swift-ci/master/ubuntu/22.04; docker build .` -4. `docker run -it --cpus --memory -v ~/:/src-on-host:cached --name lsan-reproducer --cap-add=SYS_PTRACE --security-opt seccomp=unconfined bash` - - The `-cap-add` and `-security-opt` arguments are needed to run LLDB inside the Docker container -5. Copy the sources to inside the Docker container: `cp -r /src-on-host/* ~` - - We need to to this because the build needs a case-sensitive file system and your host machine probably has a case-insensitive file system - -Build inside the Container - -1. `utils/build-script --preset buildbot_incremental_linux,lsan,tools=RDA,stdlib=DA,test=no` -2. This should reproduce the LSAN failure -3. Now, disassemble the failing CMake invocation to a swiftc invocation. I needed to set one environment variable and could the copy the swiftc invocation (but this might change as the build changes) - -``` -export LD_LIBRARY_PATH=/opt/swift/5.8.1/usr/lib/swift/linux -/home/build-user/build/buildbot_incremental_lsan/swift-linux-x86_64/./bin/swiftc -``` - -### Symbolicating the LSAN report - -For reasons that are not clear to me, LSAN does not symbolicate the report. To get the functions at the reported offsets, perform the following steps (there might be easier steps, please update this document if you know any). - -1. Run the swiftc invocation that fails and copy the leak report to somewhere. The leak report should look like the following. -``` -==3863==ERROR: LeakSanitizer: detected memory leaks - -Direct leak of 120 byte(s) in 3 object(s) allocated from: - #0 0x55b91c0b59b8 (/home/build-user/build/buildbot_incremental_lsan/swift-linux-x86_64/bin/swift-frontend+0x14d09b8) - #1 0x55b91d51281c (/home/build-user/build/buildbot_incremental_lsan/swift-linux-x86_64/bin/swift-frontend+0x292d81c) - #2 0x55b91c1b8700 (/home/build-user/build/buildbot_incremental_lsan/swift-linux-x86_64/bin/swift-frontend+0x15d3700) - -SUMMARY: LeakSanitizer: 120 byte(s) leaked in 3 allocation(s). -``` -2. `lldb -- ` -3. Start running swiftc inside lldb by executing `r` -4. Find the loaded offset of swift-frontend by running `image list` -For example, this might output -``` -[ 0] 0AEA10C1 0x0000555555554000 /home/build-user/build/buildbot_incremental_lsan/swift-linux-x86_64/bin/swift-frontend -[ 1] D52BB67A-BBBB-E429-6E87-FC16144CA7CE-55276DD6 0x00007ffff7ffb000 [vdso] (0x00007ffff7ffb000) -[ 2] 9EA8014C-F020-21A2-9E57-AA3E0512E9BB-6E30541D 0x00007ffff7dd3000 /lib/x86_64-linux-gnu/ld-2.27.so -``` -The loaded offset is `0x0000555555554000` -5. For the frame that you want to symbolicate,, add the offset you computed above to the stack frame in the LSAN report, eg. to symbolicate frame 1 `0x555555554000 + 0x292d81c = 0x555557E8181C` -6. Look up the address using `image lookup -a
`. This should output something like - -``` -(lldb) image lookup -a 0x555557E8181C - Address: swiftc[0x000000000292d81c] (swiftc.PT_LOAD[0]..text + 22056284) - Summary: swiftc`registerFunctionTest(BridgedStringRef, void*) + 28 at SILBridging.cpp:148:3 -``` - -7. Hoorray, you know which function is leaking. - -### Making Local Changes Inside the Container - -For example, to install vim in the container run - -``` -docker exec -u 0:0 -it lsan-reproducer bash -$ apt update -$ apt install vim -```