cmd/link: test TestPIESize fails for ppc64le added in CL 208897

[laboger]

cmd/link test TestPIESize fails on tip on the ppc64le builders, both power8 and power9.

On some systems, it fails consistently after this test was added. On some systems, which have newer distros it passes. On the dashboard, it didn't start failing until a few commits after that test was added, not sure why.

When it works for me, ld is >= 2.31. It works for me on a Debian Buster machine with gcc 8.3.0 and ld 2.31 and I thought that was what the builders were using so that's a mystery too. The failures occur when ld <= 2.30. Possibly this is related to which systems have gcc pass -pie by default to ld but haven't verified that. @ianlancetaylor

[cherrymui]

Yeah, I'm looking at it.

[cherrymui]

The test starts to fail after I checked in CL https://go-review.googlesource.com/c/go/+/209659, which doesn't seem to be quite related. That CL could make the file size increases or decreases slightly (as it changes some PCDATA), for both exe and pie.

What seems weird to me is that the file size of PIE binary increases significantly, but none of the sections' sizes actually increase much. For example, the sizes reported by the size command is actually very close.

ls -l (the first is old, the second is new):

-rwxr-xr-x 1 root root 2527816 Dec  6 19:21 /tmp/go-link-TestPieSize-external318734918/pieexternal
-rwxr-xr-x 1 root root 2593272 Dec  6 19:12 /tmp/go-link-TestPieSize-external338302853/pieexternal

Difference is 65456.

size command:

   text	   data	    bss	    dec	    hex	filename
1183569	 738251	 247224	2169044	 2118d4	/tmp/go-link-TestPieSize-external318734918/pieexternal
1183569	 742859	 247224	2173652	 212ad4	/tmp/go-link-TestPieSize-external338302853/pieexternal

Difference is 4608.

readelf -S output are attached at the end. .data.rel.ro section size increases slightly due to PCDATA change, which should be fine.

old:

  [21] .data.rel.ro      PROGBITS         00000000001312c0  001212c0
       000000000009ea13  0000000000000000  WA       0     0     32

new:

  [21] .data.rel.ro      PROGBITS         00000000001400c0  001300c0
       000000000009fc13  0000000000000000  WA       0     0     32

What's interesting is that the file offset of .tbss section changes significantly, even all the previous sections sizes and offsets are identical, and the alignments of all the sections are identical and small.

old:

  [17] .eh_frame         PROGBITS         0000000000120edc  00120edc
       0000000000000310  0000000000000000   A       0     0     4
  [18] .tbss             NOBITS           00000000001312b0  001212b0
       0000000000000010  0000000000000000 WAT       0     0     8

new:

  [17] .eh_frame         PROGBITS         0000000000120edc  00120edc
       0000000000000310  0000000000000000   A       0     0     4
  [18] .tbss             NOBITS           00000000001400b0  001300b0
       0000000000000010  0000000000000000 WAT       0     0     8

@ianlancetaylor any idea why the file offset changes so much?
If the external linker could just leave holes in the binaries, I wonder if we want to check the sizes of all the sections, instead of file sizes?

===========================

Full readelf -S output:
old:

File: pieexternal
There are 45 section headers, starting at offset 0x268708:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .interp           PROGBITS         0000000000000270  00000270
       0000000000000011  0000000000000000   A       0     0     1
  [ 2] .note.ABI-tag     NOTE             0000000000000284  00000284
       0000000000000020  0000000000000000   A       0     0     4
  [ 3] .note.go.buildid  NOTE             00000000000002a4  000002a4
       0000000000000064  0000000000000000   A       0     0     4
  [ 4] .note.gnu.build-i NOTE             0000000000000308  00000308
       0000000000000024  0000000000000000   A       0     0     4
  [ 5] .gnu.hash         GNU_HASH         0000000000000330  00000330
       00000000000000f0  0000000000000000   A       6     0     8
  [ 6] .dynsym           DYNSYM           0000000000000420  00000420
       00000000000005d0  0000000000000018   A       7     3     8
  [ 7] .dynstr           STRTAB           00000000000009f0  000009f0
       0000000000000354  0000000000000000   A       0     0     1
  [ 8] .gnu.version      VERSYM           0000000000000d44  00000d44
       000000000000007c  0000000000000002   A       6     0     2
  [ 9] .gnu.version_r    VERNEED          0000000000000dc0  00000dc0
       0000000000000040  0000000000000000   A       7     2     8
  [10] .rela.dyn         RELA             0000000000000e00  00000e00
       000000000006afb0  0000000000000018   A       6     0     8
  [11] .rela.plt         RELA             000000000006bdb0  0006bdb0
       0000000000000258  0000000000000018  AI       6    24     8
  [12] .init             PROGBITS         000000000006c020  0006c020
       000000000000005c  0000000000000000  AX       0     0     32
  [13] .text             PROGBITS         000000000006c080  0006c080
       00000000000978f4  0000000000000000  AX       0     0     32
  [14] .fini             PROGBITS         0000000000103974  00103974
       0000000000000024  0000000000000000  AX       0     0     4
  [15] .rodata           PROGBITS         00000000001039a0  001039a0
       000000000001d490  0000000000000000   A       0     0     32
  [16] .eh_frame_hdr     PROGBITS         0000000000120e30  00120e30
       00000000000000ac  0000000000000000   A       0     0     4
  [17] .eh_frame         PROGBITS         0000000000120edc  00120edc
       0000000000000310  0000000000000000   A       0     0     4
  [18] .tbss             NOBITS           00000000001312b0  001212b0
       0000000000000010  0000000000000000 WAT       0     0     8
  [19] .init_array       INIT_ARRAY       00000000001312b0  001212b0
       0000000000000008  0000000000000008  WA       0     0     8
  [20] .fini_array       FINI_ARRAY       00000000001312b8  001212b8
       0000000000000008  0000000000000008  WA       0     0     8
  [21] .data.rel.ro      PROGBITS         00000000001312c0  001212c0
       000000000009ea13  0000000000000000  WA       0     0     32
  [22] .dynamic          DYNAMIC          00000000001cfcd8  001bfcd8
       0000000000000210  0000000000000010  WA       7     0     8
  [23] .got              PROGBITS         00000000001cff00  001bff00
       0000000000000048  0000000000000008  WA       0     0     256
  [24] .plt              NOBITS           00000000001d0000  001bff48
       00000000000000d8  0000000000000008  WA       0     0     8
  [25] .data             PROGBITS         00000000001d00e0  001c00e0
       0000000000007730  0000000000000000  WA       0     0     32
  [26] .go.buildinfo     PROGBITS         00000000001d7810  001c7810
       0000000000000020  0000000000000000  WA       0     0     16
  [27] .noptrdata        PROGBITS         00000000001d7840  001c7840
       000000000000e000  0000000000000000  WA       0     0     32
  [28] .bss              NOBITS           00000000001e5840  001d5840
       000000000002fb08  0000000000000000  WA       0     0     32
  [29] .noptrbss         NOBITS           0000000000215360  001d5840
       000000000000c9c8  0000000000000000  WA       0     0     32
  [30] .comment          PROGBITS         0000000000000000  001d5840
       000000000000001c  0000000000000001  MS       0     0     1
  [31] .zdebug_aranges   PROGBITS         0000000000000000  001ffa30
       0000000000000084  0000000000000000           0     0     16
  [32] .zdebug_pubnames  PROGBITS         0000000000000000  001ffab4
       00000000000021f3  0000000000000000           0     0     1
  [33] .zdebug_info      PROGBITS         0000000000000000  00201ca7
       000000000002fe4a  0000000000000000           0     0     1
  [34] .zdebug_abbrev    PROGBITS         0000000000000000  00231af1
       000000000000038b  0000000000000000           0     0     1
  [35] .zdebug_line      PROGBITS         0000000000000000  00231e7c
       0000000000015271  0000000000000000           0     0     1
  [36] .zdebug_frame     PROGBITS         0000000000000000  002470ed
       0000000000004a47  0000000000000000           0     0     1
  [37] .zdebug_str       PROGBITS         0000000000000000  0024bb34
       00000000000003ad  0000000000000001  MS       0     0     1
  [38] .zdebug_loc       PROGBITS         0000000000000000  0024bee1
       0000000000012eae  0000000000000000           0     0     1
  [39] .zdebug_pubtypes  PROGBITS         0000000000000000  0025ed8f
       000000000000323b  0000000000000000           0     0     1
  [40] .zdebug_ranges    PROGBITS         0000000000000000  00261fca
       0000000000006530  0000000000000000           0     0     1
  [41] .debug_gdb_script PROGBITS         0000000000000000  002684fa
       0000000000000028  0000000000000000           0     0     1
  [42] .symtab           SYMTAB           0000000000000000  001d5860
       0000000000016188  0000000000000018          43   3712     8
  [43] .strtab           STRTAB           0000000000000000  001eb9e8
       000000000001403d  0000000000000000           0     0     1
  [44] .shstrtab         STRTAB           0000000000000000  00268522
       00000000000001e4  0000000000000000           0     0     1

new:

File: pieexternal
There are 45 section headers, starting at offset 0x2786b8:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .interp           PROGBITS         0000000000000270  00000270
       0000000000000011  0000000000000000   A       0     0     1
  [ 2] .note.ABI-tag     NOTE             0000000000000284  00000284
       0000000000000020  0000000000000000   A       0     0     4
  [ 3] .note.go.buildid  NOTE             00000000000002a4  000002a4
       0000000000000064  0000000000000000   A       0     0     4
  [ 4] .note.gnu.build-i NOTE             0000000000000308  00000308
       0000000000000024  0000000000000000   A       0     0     4
  [ 5] .gnu.hash         GNU_HASH         0000000000000330  00000330
       00000000000000f0  0000000000000000   A       6     0     8
  [ 6] .dynsym           DYNSYM           0000000000000420  00000420
       00000000000005d0  0000000000000018   A       7     3     8
  [ 7] .dynstr           STRTAB           00000000000009f0  000009f0
       0000000000000354  0000000000000000   A       0     0     1
  [ 8] .gnu.version      VERSYM           0000000000000d44  00000d44
       000000000000007c  0000000000000002   A       6     0     2
  [ 9] .gnu.version_r    VERNEED          0000000000000dc0  00000dc0
       0000000000000040  0000000000000000   A       7     2     8
  [10] .rela.dyn         RELA             0000000000000e00  00000e00
       000000000006afb0  0000000000000018   A       6     0     8
  [11] .rela.plt         RELA             000000000006bdb0  0006bdb0
       0000000000000258  0000000000000018  AI       6    24     8
  [12] .init             PROGBITS         000000000006c020  0006c020
       000000000000005c  0000000000000000  AX       0     0     32
  [13] .text             PROGBITS         000000000006c080  0006c080
       00000000000978f4  0000000000000000  AX       0     0     32
  [14] .fini             PROGBITS         0000000000103974  00103974
       0000000000000024  0000000000000000  AX       0     0     4
  [15] .rodata           PROGBITS         00000000001039a0  001039a0
       000000000001d490  0000000000000000   A       0     0     32
  [16] .eh_frame_hdr     PROGBITS         0000000000120e30  00120e30
       00000000000000ac  0000000000000000   A       0     0     4
  [17] .eh_frame         PROGBITS         0000000000120edc  00120edc
       0000000000000310  0000000000000000   A       0     0     4
  [18] .tbss             NOBITS           00000000001400b0  001300b0
       0000000000000010  0000000000000000 WAT       0     0     8
  [19] .init_array       INIT_ARRAY       00000000001400b0  001300b0
       0000000000000008  0000000000000008  WA       0     0     8
  [20] .fini_array       FINI_ARRAY       00000000001400b8  001300b8
       0000000000000008  0000000000000008  WA       0     0     8
  [21] .data.rel.ro      PROGBITS         00000000001400c0  001300c0
       000000000009fc13  0000000000000000  WA       0     0     32
  [22] .dynamic          DYNAMIC          00000000001dfcd8  001cfcd8
       0000000000000210  0000000000000010  WA       7     0     8
  [23] .got              PROGBITS         00000000001dff00  001cff00
       0000000000000048  0000000000000008  WA       0     0     256
  [24] .plt              NOBITS           00000000001e0000  001cff48
       00000000000000d8  0000000000000008  WA       0     0     8
  [25] .data             PROGBITS         00000000001e00e0  001d00e0
       0000000000007730  0000000000000000  WA       0     0     32
  [26] .go.buildinfo     PROGBITS         00000000001e7810  001d7810
       0000000000000020  0000000000000000  WA       0     0     16
  [27] .noptrdata        PROGBITS         00000000001e7840  001d7840
       000000000000e000  0000000000000000  WA       0     0     32
  [28] .bss              NOBITS           00000000001f5840  001e5840
       000000000002fb08  0000000000000000  WA       0     0     32
  [29] .noptrbss         NOBITS           0000000000225360  001e5840
       000000000000c9c8  0000000000000000  WA       0     0     32
  [30] .comment          PROGBITS         0000000000000000  001e5840
       000000000000001c  0000000000000001  MS       0     0     1
  [31] .zdebug_aranges   PROGBITS         0000000000000000  0020fa30
       0000000000000084  0000000000000000           0     0     16
  [32] .zdebug_pubnames  PROGBITS         0000000000000000  0020fab4
       00000000000021ee  0000000000000000           0     0     1
  [33] .zdebug_info      PROGBITS         0000000000000000  00211ca2
       000000000002fe42  0000000000000000           0     0     1
  [34] .zdebug_abbrev    PROGBITS         0000000000000000  00241ae4
       000000000000038b  0000000000000000           0     0     1
  [35] .zdebug_line      PROGBITS         0000000000000000  00241e6f
       0000000000015271  0000000000000000           0     0     1
  [36] .zdebug_frame     PROGBITS         0000000000000000  002570e0
       0000000000004a47  0000000000000000           0     0     1
  [37] .zdebug_str       PROGBITS         0000000000000000  0025bb27
       00000000000003ad  0000000000000001  MS       0     0     1
  [38] .zdebug_loc       PROGBITS         0000000000000000  0025bed4
       0000000000012e70  0000000000000000           0     0     1
  [39] .zdebug_pubtypes  PROGBITS         0000000000000000  0026ed44
       0000000000003238  0000000000000000           0     0     1
  [40] .zdebug_ranges    PROGBITS         0000000000000000  00271f7c
       000000000000652a  0000000000000000           0     0     1
  [41] .debug_gdb_script PROGBITS         0000000000000000  002784a6
       0000000000000028  0000000000000000           0     0     1
  [42] .symtab           SYMTAB           0000000000000000  001e5860
       0000000000016188  0000000000000018          43   3712     8
  [43] .strtab           STRTAB           0000000000000000  001fb9e8
       000000000001403d  0000000000000000           0     0     1
  [44] .shstrtab         STRTAB           0000000000000000  002784ce
       00000000000001e4  0000000000000000           0     0     1

[cherrymui]

@laboger on the builder, it has ld 2.31.1 and gcc 8.3.0. When linking PIE, the Go linker passes -pie to gcc, and gcc indeed passes -pie to ld.

[ianlancetaylor]

What is happening here has something to do with the relro sections. When using relro, the goal is for the relro part of the writable data segment to end at a page boundary. Then the dynamic linker can load the executable, modify the relro part, and then mark those pages as readonly.

The relro sections start with .init_array and end with .got, so the goal is for .got to end at a page boundary. The alignment of .got is 256 (I'm not sure why) so the linker wants it to start at an address ending with 0xff00 (the page size here seems to be 0x10000). We then work backward from there. (The .tbss section is SHT_NOBITS so it doesn't really matter. It's just tagging along with .init_array for some reason.)

In the first program, the addresses happen to work out with just a small gap between the end of .eh_frame and the start of .init_array. In the second program the .data.rel.ro section is larger, so the linker has move everything forward an extra page. The gap between .eh_frame and .init_array increases from 0xc4 to 0xeec4.

At least, I think that is what is happening.

Maybe what we need to do here is measure the gaps between the PT_LOAD segment file offsets and subtract them from the file size.

[thanm]

@ianlancetaylor this is an interesting observation (makes sense). Is there a design or "reference" document anywhere that talks more about how relro is implemented?

[laboger]

The test starts to fail after I checked in CL https://go-review.googlesource.com/c/go/+/209659, which doesn't seem to be quite related. That CL could make the file size increases or decreases slightly (as it changes some PCDATA), for both exe and pie.

@cherrymui I am testing outside of a container. The test starts the fail with the commit where it was added when I build and test on my Ubuntu 16.04 system. Based on Ian's last comment, it seems to be somewhat random depending on section offsets and sizes.

[cherrymui]

Thanks @ianlancetaylor ! This makes sense. I guess this only shows up on PPC64 because it has a large page size.

[ianlancetaylor]

@thanm Do you want a design doc or an explanation? The design doc, such as it is, is https://www.sourceware.org/ml/binutils/2004-01/msg00070.html. My explanation is https://www.airs.com/blog/archives/189.

[gopherbot]

Change https://golang.org/cl/210180 mentions this issue: cmd/link: skip gaps between PT_LOAD segments in TestPIESize

[gopherbot]

Change https://golang.org/cl/220654 mentions this issue: cmd/link: improve gap detection in TestPIESize