Create runtime stacktraces

[certik]

The generated LLVM code needs the ability to create Python stacktraces. So that when bounds check error, or string->int conversion error happens, we print a runtime stacktrace. So we need to get the stack addresses, convert them to LLVM functions, then use LLVM debugging information to find ASR functions that they correspond to, and finally ASR location information (that we need to store in the binary) to find the original Python file and line/column number where the error comes from. Then we can print Python style stacktraces at runtime.

[certik]

One (orthogonal) issue to fix is:

• Extend LocationManager to handle multiple files #1218

[certik]

To create runtime stacktraces, I would do the following to get started:

• On all platforms we would follow the macOS convention that creates a .dSYM with lines.txt and lines.dat.
• We port the relevant parts from src/libasr/stacktrace.cpp into the runtime library (we have to port from C++ to C)
• We have to figure out how to create lines.txt and lines.dat in LPython/LFortran at compilation time:
  • For each LLVM instruction that we emit, we store the filename+line (pass it to LLVM's debugging information). (Add an option: --debug-with-line-column to convert linear location to line+column in the debug info #1256)
    • We study how LLVM debugging works: https://www.llvm.org/docs/SourceLevelDebugging.html (see the comment below)
    • Add a -g option to lpython, if enabled, we will store debugging information (Initial debugging info implementation #1240)
    • Store the !dbg !23 stuff for each emitted instruction (where it makes sense) (Initial debugging info implementation #1240)
  • LLVM generates code with this info
  • We then extract this info from the generated binary, and create lines.txt and lines.dat files.

[certik]

One can test this by:

int main() {
    int i;
    i = 1;
    i = i + 2;
    i = i + 3;
    i = i + 4;
    return i;
}

Compile using:

clang -emit-llvm -S -g a.c

and LLVM then stores debugging information for each instruction (and function), such as:

define i32 @main() #0 !dbg !13 {
...
  store i32 1, i32* %2, align 4, !dbg !19
  %3 = load i32, i32* %2, align 4, !dbg !20
  %4 = add nsw i32 %3, 2, !dbg !21
  store i32 %4, i32* %2, align 4, !dbg !22
  %5 = load i32, i32* %2, align 4, !dbg !23
  %6 = add nsw i32 %5, 3, !dbg !24
  store i32 %6, i32* %2, align 4, !dbg !25
  %7 = load i32, i32* %2, align 4, !dbg !26
  %8 = add nsw i32 %7, 4, !dbg !27
  store i32 %8, i32* %2, align 4, !dbg !28
  %9 = load i32, i32* %2, align 4, !dbg !29
  ret i32 %9, !dbg !30
}
...
!13 = distinct !DISubprogram(name: "main", scope: !10, file: !10, line: 2, type: !14, scopeLine: 2, spFlags: DISPFlagDefinition, unit: !9, retainedNodes: !11)
...
!18 = !DILocation(line: 3, column: 9, scope: !13)
!19 = !DILocation(line: 4, column: 7, scope: !13)
!20 = !DILocation(line: 5, column: 9, scope: !13)
!21 = !DILocation(line: 5, column: 11, scope: !13)
!22 = !DILocation(line: 5, column: 7, scope: !13)
!23 = !DILocation(line: 6, column: 9, scope: !13)
...

[Thirumalai-Shaktivel]

Useful links:
https://llvm.org/docs/LangRef.html#metadata
https://stackoverflow.com/questions/19743861/what-is-llvm-metadata
Blog:
http://blog.llvm.org/2010/04/extensible-metadata-in-llvm-ir.html
https://llvm.org/docs/tutorial/MyFirstLanguageFrontend/LangImpl09.html#dwarf-emission-setup

[Thirumalai-Shaktivel]

We should use the ASR node itself to pass information about the name of the array or list that is out of bounds and also store the line/column number with the error message.
#1093 (comment)

..

[Thirumalai-Shaktivel]

I get the following error while using dsymutil:

$ lpython examples/expr2.py -g --debug-with-line-column -o a.out
$ dsymutil a.out
error: cannot parse the debug map for 'a.out': The file was not recognized as a valid object file

Do we have any other alternative for this? Can we fix this error?

[Thirumalai-Shaktivel]

I get the expected output using this:

$ llvm-dwarfdump --debug-line a.out
a.out:	file format elf64-x86-64

.debug_line contents:
debug_line[0x00000000]
Line table prologue:
    total_length: 0x00000054
          format: DWARF32
         version: 4
 prologue_length: 0x00000029
 min_inst_length: 1
max_ops_per_inst: 1
 default_is_stmt: 1
       line_base: -5
      line_range: 14
     opcode_base: 13
standard_opcode_lengths[DW_LNS_copy] = 0
standard_opcode_lengths[DW_LNS_advance_pc] = 1
standard_opcode_lengths[DW_LNS_advance_line] = 1
standard_opcode_lengths[DW_LNS_set_file] = 1
standard_opcode_lengths[DW_LNS_set_column] = 1
standard_opcode_lengths[DW_LNS_negate_stmt] = 0
standard_opcode_lengths[DW_LNS_set_basic_block] = 0
standard_opcode_lengths[DW_LNS_const_add_pc] = 0
standard_opcode_lengths[DW_LNS_fixed_advance_pc] = 1
standard_opcode_lengths[DW_LNS_set_prologue_end] = 0
standard_opcode_lengths[DW_LNS_set_epilogue_begin] = 0
standard_opcode_lengths[DW_LNS_set_isa] = 1
include_directories[  1] = "examples"
file_names[  1]:
           name: "expr2.py"
      dir_index: 1
       mod_time: 0x00000000
         length: 0x00000000

Address            Line   Column File   ISA Discriminator Flags
------------------ ------ ------ ------ --- ------------- -------------
0x0000000000001130      0      0      1   0             0  is_stmt
0x0000000000001131      6      1      1   0             0  is_stmt prologue_end
0x0000000000001140      0      0      1   0             0  is_stmt
0x0000000000001140      3      5      1   0             0  is_stmt prologue_end
0x0000000000001150      0      0      1   0             0  is_stmt
0x0000000000001151      1      1      1   0             0  is_stmt prologue_end
0x000000000000115a      1      1      1   0             0  is_stmt end_sequence

[Thirumalai-Shaktivel]

I think I was able to generate lines.txt and lines.dat?
How do I move forward with this issue?
@certik

[Thirumalai-Shaktivel]

lpython/src/bin/CMakeLists.txt

Lines 27 to 38 in 67251c2

	if (WITH_DWARFDUMP)
	add_custom_command(
	TARGET lpython
	POST_BUILD
	COMMAND llvm-dwarfdump --debug-line lpython.dSYM > lpython.dSYM/raw.txt
	)
	add_custom_command(
	TARGET lpython
	POST_BUILD
	COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/dwarf_convert.py lpython.dSYM/raw.txt lpython.dSYM/lines.txt lpython.dSYM/lines.dat
	)
	endif()

I used the same method as this. There seems to be a problem. lpython.dSYM/lines.dat is empty.