Dartdoc fails on pub.dev for win32 because docs generated are too verbose

[timsneath]

The win32 package is big, but not ridiculously so. The lib\src directory is only 3.3MB in size.

But dart doc is now failing on pub.dev with the following error:

DocumentationTooBigException: Reached 2147483648 bytes in the output directory.
#0      PubResourceProvider._aboutToWriteBytes (package:pub_dartdoc/src/pub_hooks.dart:61:7)
#1      _File.writeAsBytesSync (package:pub_dartdoc/src/pub_hooks.dart:145:15)
#2      _File.writeAsStringSync (package:pub_dartdoc/src/pub_hooks.dart:151:5)
#3      DartdocFileWriter.write (package:dartdoc/src/dartdoc.dart:72:10)
...

Log details here: https://pub.dev/documentation/win32/4.1.1/log.txt

Yes -- this 3.3MB of Dart code has generated over 2 gigabytes of auto-generated documentation!

Looking at the output files, there are 10,000+ documentation pages. The primary problem is that we generate a single file for even the most trivial content (e.g. a constant), and that file itself contains a reference to every other constant.

For example, the line const WM_PAINT = 0x000F; is transformed into the following 500KB+ HTML file (renamed to .txt so GitHub will let me attach it):
WM_PAINT-constant.html.txt

There is about 200 bytes of meaningful content in this file. Most of the file is a list of <a> refs to other similar files:

        ...
        <li><a href="../winrt/WM_SETTEXT-constant.html">WM_SETTEXT</a></li>
        <li><a href="../winrt/WM_SETTINGCHANGE-constant.html">WM_SETTINGCHANGE</a></li>
        <li><a href="../winrt/WM_SHOWWINDOW-constant.html">WM_SHOWWINDOW</a></li>
        <li><a href="../winrt/WM_SIZE-constant.html">WM_SIZE</a></li>
        <li><a href="../winrt/WM_SIZECLIPBOARD-constant.html">WM_SIZECLIPBOARD</a></li>
        <li><a href="../winrt/WM_SPOOLERSTATUS-constant.html">WM_SPOOLERSTATUS</a></li>
        <li><a href="../winrt/WM_STYLECHANGED-constant.html">WM_STYLECHANGED</a></li>
        ...

I hope we can make some changes to tidy this up quickly, since my package currently has no working documentation :( In addition, this must be expensive for us in hosting costs.

/cc @kevmoo

[srawlins]

Thanks Tim. I think this is a duplicate of #2995. See also #1272

[srawlins]

CC @devoncarew

I've got a proof of concept writes sidebars as separate files, and loads them dynamically, removing duplication. I tested it on win32 4.1.1:

BEFORE
======

finished in 6311.0 seconds

real    105m18.523s
user    21m26.461s
sys     3m11.344s

output size: 2.0G

AFTER
=====

finished in 265.3 seconds

real    4m32.729s
user    1m31.922s
sys     2m8.875s

output size: 96M

[timsneath]

That looks... better!

Related, I wonder if we can review the content in the sidebar. It's actually not very helpful, if you look at a typical page in the Win32 package. And that's not isolated to my package: it's hugely inflated even for a Flutter library. I'd be surprised if it was used by most customers of large-scale packages.

[srawlins]

Sounds good, we should research it.

[isoos]

I think this is resolved now.

[srawlins]

The situation now:

• Output is 93 MB (was over 2 GB)
• Number of output files is 11.6k (was "over 10k")
• Took 115 seconds to generate on my laptop.