runtime: pclntab is too big

[robpike]

This article contains results from applying a visualization tool:
https://www.cockroachlabs.com/blog/go-file-size/
If the analysis is correct, pclntab needs a major redesign.

Separated from #6853 to keep the conversation focused.

[networkimprov]

Also discussed in #27266

And there are some interesting suggestions in https://news.ycombinator.com/item?id=21907144

[josharian]

I spent some time looking at this earlier this year (see e.g. CLs 172079 and 171820), and then stalled. The major impediment was the inscrutability of the linker. I hope that the new linker will open doors.

There is definitely lots more room for improvement. I have had private conversations with a few Go team members about improving the string encoding in pclntab. Most strings in the pclntab never actually get used by the program, and thus could be materialized lazily, perhaps from traditional compression or from a trie-ish encoding. This isn't trivial but you could do something like: Set up an appropriately-sized bss symbol with known offsets for all materialized strings, and when you need to materialize a string, decompress/assemble it into the correct spot for all other future users (atomically or with locks somehow). I did a very rough calculation and estimated that this could shrink pclntab by about 10%.

[thanm]

@jeremyfaller has also been brainstorming about this.

The concern I would have about such a scheme (putting on my devil's advocate hat) is that rodata can be shared across multiple instances of an executable across a given system, whereas if you go the BSS route, each process has to have it's own chunk of virtual memory.

On the other hand, it's not at all clear whether people care about that (who out there is running many instances of the same Go program on the same machine?).

[heschi]

I think there's some confusion in the blog post. Sorry if I'm the one confused instead.

The blog post talks about runtime.pclntab, but that symbol actually contains many tables, correct? ld.(*Link).pclntab() writes the pcfile and pcline tables, which are roughly in line with what you'd expect to be in a symbol named pclntab, but it also writes the pcsp and pcdata tables, which are quite different.

Has anyone checked which tables are actually the problem? I think all the commentary above is discussing the file/line tables.

[josharian]

Speaking for myself, I've looked at all of them; see CL 171760 for a recent tweak. I think the most attention is going to the file/line tables because that's where the most obvious inefficiencies are. (A few years ago I experimented with using a nybble-based instead of byte-based varint encoding for pc/value pairs. It helped, but not very much.) But all that doesn't foreclose the possibility of me being confused. :)

[cherrymui]

Yes, as @heschik said, there are several tables in pclntab.

Here is a breakdown:

			hello world	cmd/compile
.text			589993		10072652
pclntab			461293		 5254373
  func table		 28364		  166220
  _func structs		145376		  879432
  pcsp			 18305		  175053
  psfile		 13466		  208905
  pcline		 80049		 1339649
  pcdata		108492		 1947840
  strings		 56025		  496601
  file table		 11216		   40673

The strings are not large portion of the pclntab.

I have another confusion: the "growth" is growing over time, or the output size grows as the input size grows? I.e. are CockroachDB v19.1 and v1.0 built with same version of Go?

For the former, there are some growth over time, as we added more kinds of tables including e.g. inline tables, stack objects, and register maps, as mentioned in #27266.

For the latter, it seems growing linearly, at least from the table above.

[aclements]

As @thanm mentioned, @jeremyfaller has been looking at this problem recently and has done some preliminary analysis. He'll be back Thursday. He was going to prototype splitting the strings into two or three components (e.g., package+type+symbol), so each component can be deduped. I'm not sure what the current state of that is. As @cherrymui points out, the strings aren't a whole lot of the pclntab, but that also seems like low-hanging fruit. @jeremyfaller has also been looking at more typical binaries that have much longer package paths than cmd/compile, and hence more string data.

@cherrymui, what tool did you use to generate that breakdown? It would be good to also see the further breakdown of "pcdata", since there are several tables in that.

An important misconception in the blog post is that pclntab only contains traceback symbolization data. It also contains a lot of critical data for the garbage collector.

I did some experiments a while back with compressing stack maps. At the time, I noted that Huffman-coding the PCDATA delta stream roughly halved the size of the stack maps. However, that depended on the linker generating an optimal Huffman table for the whole binary, which would increase link time. Here's the code for those experiments, though it's probably bitrotted.

[cherrymui]

@aclements I just added print statements in the linker. I can do that for the pcdata tables.

[jeremyfaller]

As @thanm mentions, I was taking a look at starting to better encode pclntab. His tradeoffs are accurate -- trading some read-only-mappable memory for BSS. I was pulled to thinking about it because of internal Google binaries, where path names are quite long, and some experimentation showed that as much as 50% of pcln was used for strings. I started a design encoding paths by stripping common roots, and for function names, stripping module and type names. It's still early days, but I thought we'd see reductions in binary size > 5%. As @cherrymui points out, that gain will be minimal for the majority of users.

Right now, I was writing that document, and a change to cmd/objdump to generate stats to help us figure out where we need to focus our efforts.

[aclements]

The title of this bug (as well as the blog post) claims that pclntab grows super-linearly in the number of functions in the binary, but it's completely unclear to me what that claim is based on. It seems that they're varying both the Cockroach version and the Go version, so not only did Cockroach get bigger, but we also added more pclntab tables for the garbage collector since Go 1.2. That aside, the table in the blog post that compares Cockroach in 2017 to 2019 has exactly two data points in it, with the rest "projected", so I don't know how you get any non-linear projection from that.

The pclntab is large, a significant fraction of Go binaries, and its size is starting to cause issues for cutting-edge users, so this is worth working on. I just don't understand the basis for the super-linear growth claim.

[cherrymui]

@aclements here is a breakdown of the pcdata:

			hello world	cmd/compile
.text			589993		10072652
pclntab			461293		 5254373
  func table		 28364		  166220
  _func structs		145376		  879432
  pcsp			 18305		  175053
  psfile		 13466		  208905
  pcline		 80049		 1339649
  pcdata		108492		 1947840
    [0]			 56055		 1211541
    [1]			 35454		  465937
    [2]			 16983		  270362
  strings		 56025		  496601
  file table		 11216		   40673

The register maps are the most of them.

[aclements]

The register maps are the most of them.

Great. I'm planning on getting rid of those. :)

Of all the pc tables, the pcline is actually the biggest. I wonder if that's because of instruction interleaving and inlining. I bet that's also wasting a lot of bits writing down very small deltas. Since we only use that table for symbolization, it doesn't have to be extremely efficient to decode, so we could imagine using a more compact encoding of it.

[josharian]

I bet that's also wasting a lot of bits writing down very small deltas.

That's what I was hinting at with a "nybble-based varint encoding" above.

In case it helps, here are my notes on this topic, from Feb 2016 (some things may have changed since then!):

pc/value pairs in the go1.2 symtab are encoded using varint encoding. But most values are very small, so I thought I'd try using nibble-at-a-time style varint encoding instead of byte-at-a-time
varint encoding. You lose 1/4 bits to continuation markers instead of 1/8, but with small values it doesn't matter.

This only works because there are value pairs. The precise encoding is: first four bits are value, second four bits are pc. First bit of each nibble is a continuation bit. If both continuation bits are set, continue as before. If neither continuation bit is set, we're done. If one continuation bit is set but not the other, switch to byte-at-a-time varint encoding for the outstanding pc/value.

This actually does prove to be a more space-efficient encoding, just not enough of an improvement to warrant changing the symtab, since there are many consumers of it.

Results on the Go binary, which seems fairly typical:

binary size: 12190860
total pcln symbol size: 1469269
pcsp={current:114151 nibbles:109005}
pcfile={current:32212 nibbles:30761}
pcln={current:220683 nibbles:198681}
pctotal={current:367046 nibbles:338447}

So a total of 28k of savings, or 0.23%.

Results are similar on the SSA branch.

This is all simulations based on parsing and replaying the symtab, not actually implementing the new encoding. There's an open question about the performance of a nibble-based encoding. (Better because memory efficient? Worse because more bit twiddling?)