debug/pe: add APIs to support reading COMDAT info for sections

[thanm]

This is a proposal to add additional APIs to the "debug/pe" package to
enable reading of auxiliary symbol information, specifically to enable
extraction of info about COMDAT sections, needed by the Go linker.

Background + Motivation

The debug/pe package currently provides hooks for reading symbol table
information from PE object files, notably the "COFFSymbols" slice
provided in pe's "File" type:

  // A File represents an open PE file.
  type File struct {
	FileHeader
	OptionalHeader any // of type *OptionalHeader32 or *OptionalHeader64
	Sections       []*Section
	Symbols        []*Symbol    // COFF symbols with auxiliary symbol records removed
	COFFSymbols    []COFFSymbol // all COFF symbols (including auxiliary symbol records)
	StringTable    StringTable

	closer io.Closer
  }

The COFFSymbols array is something of a fiction, in that the slice
includes entries for "regular" symbols, but also for entries in the
file's symbol array that correspond to aux symbols. Both regular
symbols and aux symbols are 18 bytes in size, but the format of aux
symbols varies depending on the type of regular symbol it precedes.
Picture to illustrate:

   ...
   k+0:  regular sym k
   k+1:    1st aux symbol for k
   k+2:    2nd aux symbol for k
   k+3:  regular sym k+3
   k+4:    1st aux symbol for k+3
   k+5:  regular sym k+5
   k+6:  regular sym k+6
   ...

More can be found in the Microsoft PE format documentation.

The existing debug/pe APIs list the number of aux symbols for each
regular symbol, making it possible to skip aux entries when iterating
through the main symbol array, but there is no way to look inside the
aux symbols.

One class of aux symbol is the section definition
aux symbol, which holds additional bits of info about symbols
corresponding to defined sections in the object file.

One part of the section definition aux symbol is material related to
COMDAT sections; the Go linker needs to be able to access this COMDAT
info in order to support more modern clang-based C toolchains on
Windows (see CL stack at
https://go-review.googlesource.com/q/topic:modernize_windows_c_compiler
and associated issue at https://golang.org/issue/35006).

Proposal details

The proposal is to add these new APIs to debug/pe:

// Section characteristics flags.
const (
	IMAGE_SCN_CNT_CODE               = 0x00000020
	IMAGE_SCN_CNT_INITIALIZED_DATA   = 0x00000040
	IMAGE_SCN_CNT_UNINITIALIZED_DATA = 0x00000080
	IMAGE_SCN_LNK_COMDAT             = 0x00001000
	IMAGE_SCN_MEM_DISCARDABLE        = 0x02000000
	IMAGE_SCN_MEM_EXECUTE            = 0x20000000
	IMAGE_SCN_MEM_READ               = 0x40000000
	IMAGE_SCN_MEM_WRITE              = 0x80000000
)

// COFFSymbolAuxFormat5 describes the expected form of an aux symbol
// attached to a section definition symbol. The PE format defines a
// number of different aux symbol formats: format 1 for function
// definitions, format 2 for .be and .ef symbols, and so on. Format 5
// holds extra info associated with a section definition, including
// number of relocations + line numbers, as well as COMDAT info. See
// https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#auxiliary-format-5-section-definitions
// for more on what's going on here.
type COFFSymbolAuxFormat5 struct {
	Size           uint32
	NumRelocs      uint16
	NumLineNumbers uint16
	Checksum       uint32
	SecNum         uint16
	Selection      uint8
	_              [3]uint8 // Padding
}

// These constants make up the possible values for the 'Selection'
// field in an AuxFormat5.
const (
	IMAGE_COMDAT_SELECT_NODUPLICATES = 1
	IMAGE_COMDAT_SELECT_ANY          = 2
	IMAGE_COMDAT_SELECT_SAME_SIZE    = 3
	IMAGE_COMDAT_SELECT_EXACT_MATCH  = 4
	IMAGE_COMDAT_SELECT_ASSOCIATIVE  = 5
	IMAGE_COMDAT_SELECT_LARGEST      = 6
)

// COFFSymbolReadSectionDefAux returns a blob of axiliary information
// (including COMDAT info) for a section definition symbol. Here 'idx'
// is the index of a section symbol in the main COFFSymbol array for
// the File. Return value is a pointer to the appropriate aux symbol
// struct. For more info, see:
//
// auxiliary symbols: https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#auxiliary-symbol-records
// COMDAT sections: https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#comdat-sections-object-only
// auxiliary info for section definitions: https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#auxiliary-format-5-section-definitions
//
func (f *File) COFFSymbolReadSectionDefAux(idx int) (*COFFSymbolAuxFormat5, error) {

...
}

When a client discovers a section definition symbol that has the IMAGE_SCN_LNK_COMDAT bit set in its characteristics flags, it can call into COFFSymbolReadSectionDefAux to obtain a pointer to the aux info for the section, allowing it to read the info about the COMDAT section.

[ianlancetaylor]

Adding to proposal project for discussion, but my personal feeling is that we don't need a formal proposal to add simple accessors and constants to the debug/* packages.

[thanm]

I agree, it seems a bit heavyweight, but I am just following the instructions as laid out in https://go.googlesource.com/go/+/96567fb3ccdc8d8c052606112db7b39abd725b59/api/README

[gopherbot]

Change https://go.dev/cl/394534 mentions this issue: debug/pe: add APIs for reading section def aux info

[alexbrainman]

@thanm

Your proposal LGTM.

Thank you.

Alex

[rsc]

This seems clear enough and unobjectionable enough that I think we can just move it to accept. Thanks!

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[thanm]

https://go.dev/cl/394534 has been +2'd and is ready to submit; I am going to go ahead and close this issue out (this is now required in order for the API check test to pass during all.bash).

[gopherbot]

Change https://go.dev/cl/414515 mentions this issue: api: correct debug/pe issue number for Go 1.19 changes