native-image generates corrupted debuginfo

[avimalka]

Describe the issue
When generating native image with -g it's impossible to debug it with gdb as debuginfo section is corrupted.
When running gdb following message is showing:

Reading symbols from App...
Dwarf Error: wrong version in compilation unit header (is 0, should be 2, 3, 4 or 5) [in module /tmp/native-image/App]
(No debugging symbols found in App)

Also objdump show the following warning:

objdump: Warning: Debug info is corrupted, .debug_info header at 0xce has length 6c0000

Steps to reproduce the issue
native-image -g --no-fallback -H:DebugInfoSourceSearchPath=/tmp/graal-src.zip -jar App.jar

Describe GraalVM and your environment:
GraalVM version: 21.1.0
JDK major version: 11
OS: Linux Suse 12
Architecture: amd64

More details
objdump output: https://pastebin.com/AmLEWu1R
-H:Log=DebugTypeInfo:3,DebugCodeInfo:3,dwarf:3 output: https://mega.nz/file/vIwTzIDB#k9NRGt5enSyb6hyzBfvvQ-o89MUepMHPuJ4hVc0Y2JQ (160MB)
Compiled file from my machine: https://mega.nz/file/qYo3QSbQ#X4Nz7szNDo1nKmny1EoULT3ZOadSDiO-z06Sgub-Mlk

[adinn]

@avimalka Hi Avi,

I have had a look at the output log file, objdump output and the binary and I am fairly sure this problem is to do with the version of assembler you are using rather than an error in the GraalVM Native generated debuginfo.

Can you provide me with a copy of the source code and class files in graal-src.zip and App.jar that you mention in the steps listed above? I would like to try building on my own machine to see if my version of gcc leads to the same problems.

I have provided an explanation of what I think is going wrong below.

----- 8< -------- 8< -------- 8< -------- 8< -------- 8< -------- 8< -------- 8< ---

The problem appears to be an extra pair of bytes in the info section.

The info dump shows 3 records before the one that has the wrong length:

  Compilation Unit @ offset 0x0:
   Length:        0x2a (32-bit)
   Version:       2
   Abbrev Offset: 0x0
   Pointer Size:  8
 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <c>   DW_AT_stmt_list   : 0x0
    <10>   DW_AT_low_pc      : 0x405200
    <18>   DW_AT_high_pc     : 0x40522a
    <20>   DW_AT_name        : (indirect string, offset: 0x0): ../sysdeps/x86_64/start.S
    <24>   DW_AT_comp_dir    : (indirect string, offset: 0x1a): /home/abuild/rpmbuild/BUILD/glibc-2.22/csu
    <28>   DW_AT_producer    : (indirect string, offset: 0x45): GNU AS 2.31.1
    <2c>   DW_AT_language    : 32769	(MIPS assembler)
  Compilation Unit @ offset 0x2e:
   Length:        0x7a (32-bit)
   Version:       4
   Abbrev Offset: 0x14
   Pointer Size:  8
 <0><39>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <3a>   DW_AT_producer    : (indirect string, offset: 0x66): GNU C 4.8.5 -mtune=generic -march=x86-64 -g -O2 -std=gnu99 -fgnu89-inline -fasynchronous-unwind-tables -fmerge-all-constants -fmessage-length=0 -frounding-math -fstack-clash-protection -funwind-tables -ftls-model=initial-exec
    <3e>   DW_AT_language    : 1	(ANSI C)
    <3f>   DW_AT_name        : (indirect string, offset: 0x855c2): init.c
    <43>   DW_AT_comp_dir    : (indirect string, offset: 0x1a): /home/abuild/rpmbuild/BUILD/glibc-2.22/csu
    <47>   DW_AT_stmt_list   : 0x5c
 <1><4b>: Abbrev Number: 2 (DW_TAG_base_type)
    <4c>   DW_AT_byte_size   : 1
    <4d>   DW_AT_encoding    : 8	(unsigned char)
    <4e>   DW_AT_name        : (indirect string, offset: 0x161): unsigned char
. . .
 <1><a6>: Abbrev Number: 5 (DW_TAG_const_type)
    <a7>   DW_AT_type        : <0x75>
 <1><ab>: Abbrev Number: 0
  Compilation Unit @ offset 0xac:
   Length:        0x1e (32-bit)
   Version:       2
   Abbrev Offset: 0x52
   Pointer Size:  8
 <0><b7>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <b8>   DW_AT_stmt_list   : 0x83
    <bc>   DW_AT_ranges      : 0x0
    <c0>   DW_AT_name        : (indirect string, offset: 0x178): ../sysdeps/x86_64/crti.S
    <c4>   DW_AT_comp_dir    : (indirect string, offset: 0x1a): /home/abuild/rpmbuild/BUILD/glibc-2.22/csu
    <c8>   DW_AT_producer    : (indirect string, offset: 0x45): GNU AS 2.31.1
    <cc>   DW_AT_language    : 32769	(MIPS assembler)
  Compilation Unit @ offset 0xce:
   Length:        0x6c0000 (32-bit)
   Version:       0
   Abbrev Offset: 0x4
   Pointer Size:  4
<end of output>

As you can see form the producer info this part of the info table has been generated by gcc and gas.

One weird thing to notice right away is that the records for the assembly code input files are labelled with language code 32769 (0x8001) which identifies as MIPS assembler. That's already an indication that something is a bit odd here.

Looking at the original binary allows one to see where things have gone wrong. The object header dump indicates that the info section starts at offset 0x0092fc00 in the file. Here is a hex dump of the relevant data:

0092fc00: 2a00 0000 0200 0000 0000 0801 0000 0000  *...............
0092fc10: 0052 4000 0000 0000 2a52 4000 0000 0000  .R@.....*R@.....
0092fc20: 0000 0000 1a00 0000 4500 0000 0180 7a00  ........E.....z.
0092fc30: 0000 0400 1400 0000 0801 6600 0000 01c2  ..........f.....
. . .
0092fca0: 0000 0000 0000 0547 0000 0000 1e00 0000  .......G........
0092fcb0: 0200 5200 0000 0801 8300 0000 0000 0000  ..R.............
0092fcc0: 7801 0000 1a00 0000 4500 0000 0180 0000  x.......E.......
0092fcd0: 6c00 0000 0400 0000 0000 0801 0b05 0101  l...............
0092fce0: 027d 9103 0005 0210 0575 2904 0005 0840  .}.......u)....@
. . .

The first line at offset zero

0092fc00: (2a00 0000) (0200) (0000 0000) (08)01 0000 0000  *...............

defines the properties of compilation unit representing the first assembled file

  Compilation Unit @ offset 0x0:
   Length:        0x2a (32-bit)  (4 bytes)
   Version:       2  (2 bytes)
   Abbrev Offset: 0x0 (4 bytes)
   Pointer Size:  8 (1 byte)

I have bracketed the relevant bytes so you can see how the header is laid out. Note that low to high bytes in a short or word are bytes are laid out left to right.

The next byte is an abbrev section tag byte (0x1) identifying the layout of the rest of the record in this compilation unit. I'll skip the details of that part of the record as it is not relevant.

We can step through the records by adding the length to the address of the byte following the length field (the length excludes the 4 length bytes). So, the next compilation unit starts at offset 0x2e with length 0x7a (0x0092fc2e - 0x0092fcab)

0092fc20: 0000 0000 1a00 0000 4500 0000 0180 (7a00  ........E.....z.
0092fc30: 0000) (0400) (1400 0000) (08)01 6600 0000 01c2  ..........f.....

  Compilation Unit @ offset 0x2e:
   Length:        0x7a (32-bit)
   Version:       4
   Abbrev Offset: 0x14
   Pointer Size:  8

and the next one at offset 0xac with length 0x1e (0x0092fcac - 0x0092fccd)

0092fca0: 0000 0000 0000 0547 0000 0000 (1e00 0000)  .......G........
0092fcb0: (0200) (5200 0000) (08)01 8300 0000 0000 0000  ..R.............

  Compilation Unit @ offset 0xac:
   Length:        0x1e (32-bit)
   Version:       2
   Abbrev Offset: 0x52
   Pointer Size:  8

When objdump starts reading the fourth record it reports that it is reading from offset 0xce which is the expected location:

0092fcc0: 7801 0000 1a00 0000 4500 0000 0180 (0000  x.......E.......
0092fcd0: 6c00) (0000) (0400 0000) (00)00 0801 0b05 0101  l...............

Compilation Unit @ offset 0xce:
Length: 0x6c0000 (32-bit)
Version: 0
Abbrev Offset: 0x4
Pointer Size: 4

However, the 3rd record appears to have been padded with an extra pair of zero bytes present at offset 0xce. It seems these extra bytes are causing the reader to read the wrong length for the 4th compilation unit. If we skip those two bytes the next compilation unit actually makes sense:

0092fcc0: 7801 0000 1a00 0000 4500 0000 0180 0000  x.......E.......
0092fcd0: (6c00 0000) (0400) (0000 0000) (08)01 0b05 0101  l...............

If the reader were to start from here this would get processed as

Compilation Unit @ offset 0xd0:
Length: 0x6c (32-bit)
Version: 0x4
Abbrev Offset: 0x0
Pointer Size: 8

This adjustment places the next compilation unit at 0092fd40 which also makes sense

    0092fd30: 0000 0119 ad34 0400 2500 0000 0801 0000  .....4..%.......
    0092fd40: (5900 0000) (0400) (0000 0000) (08)03 0b95 7200  Y.............r.

So, the problem appears to be invalid debuginfo generated by your assembler.

[avimalka]

Can you provide me with a copy of the source code and class files in graal-src.zip and App.jar that you mention in the steps listed above? I would like to try building on my own machine to see if my version of gcc leads to the same problems.

graal-src.zip is simply lib/src.zip from GraalVM 21.1.0 CE and the App I used to reproduce is simple "Hellow World" nothing special.

So, the problem appears to be invalid debuginfo generated by your assembler.

Is there a way to see gcc output when appending --native-compiler-options=-v ? I would like to see the gas command gcc is running.

[olpaw]

Is there a way to see gcc output when appending --native-compiler-options=-v ? I would like to see the gas command gcc is running.

Have you tried:

-H:+TraceNativeToolUsage Trace all native tool invocations as part of image building.

[adinn]

Hi @avimalka

graal-src.zip is simply lib/src.zip from GraalVM 21.1.0 CE and the App I used to reproduce is simple "Hellow World" nothing special.

Ok, I already know that Hello World works on my machine with my installed gcc. So, I don't think I am going to be able to reproduce this locally.

Is there a way to see gcc output when appending --native-compiler-options=-v ? I would like to see the gas command gcc is running.

I don't know the answer to that. I suggest following Paul's advice.

Also, I'm afraid I am not really in a position to advise on how gas operates, not even how it produces its debuginfo. You probably need to ask a gcc project developer. However, that said, if you are able to grab the object file output by gas before it gets linked into the final image I can take a look at it and check whether the debuginfo it contains is correct or not.

That might help determine whether the extra bytes are being inserted by gas itself or are inserted by the linker when it merges all the debuginfo from the different input object files into the final executable. My money is on it being a problem in gas but you never know.

Also, it would be good to know why the language code for the assembly compile units is coming out as MIPS assembler. That's definitely weird. Once again it would probably be best to consult a gcc project developer.

[avimalka]

Hi,

After retrying again in Ubuntu host it works perfectly.
I will try to figure out what's wrong with my gcc installation.

Thank you very much for your help.

[fniephaus]

It seems that #8046 fixes this issue! 🎉