Single Page App design with static files #3079

srawlins · 2022-07-12T19:25:18Z

In order to solve output size issues (#2995), performance issues (#2799, #2998), and in order to enable some desired front-end features ([citation needed]), we can convert generated docs into a rough Single Page App design.

Typically a single page app would have a web server at the backend which is generating JSON responses to update the contents of the page, but we can carry out the minimum amount of work by just dumping slightly different HTML files, and incorporating some JS to request them and insert HTML snippets into the DOM.

Background

The long and short of our performance and output size problems come from this fact:

A library with a class with n members will generate O(n) files, each of which has a sidebar of length O(n), resulting in a total of O(n²) bytes.

Design

In a green field design, we could re-imagine dartdoc to instead dump out one or many JSON files which would be served to a JS app and drawn into the page. But in this design, I take all of our existing code which generates one HTML file for every element, and make only small changes.

Generation

Today, for each element, we generate an HTML file with the following 5 sections in its body:

a header
a left sidebar
a right sidebar
a main body
a footer

The blocks with incredible amounts of duplication are the two sidebars. The footer may also be perfectly duplicated across every file, but probably doesn't represent a lot of bytes.

In this design, for each element, we instead generate an HTML file with the following 2-3 filled sections:

a header
a main body
(probably not the footer; but removing it can be done separately)

The other sections will be present but empty (or near enough; there are some breadcrumbs for mobile at the top of some sidebars)

The HTML will look something like:

<html>
    <head>...</head>
    <body>
        <header>...</header>
        <main>...</main>
        <sidebar-left><!-- intentionally empty --></sidebar-left>
        <sidebar-right><!-- intentionally empty --></sidebar-right>
        <footer><!-- maybe empty --></footer>
    </body>
</html>

Additionally, we generate a few more files (apologies, #1272):

For every library, we generate a library sidebar, which will be displayed on the library's page, and on the page of every element exported from that library.
For every "container" (class, mixin, enum, extension), we generate a container sidebar, which will be displayed on the container's page, and on the page of every element in the container (fields, methods, ...).

Initial page load

This new design requires that docs be served from an HTTP file server. The dartdoc package uses the dhttpd package in it's grinder scripts. Python's httpd module works fine as well. Whatever currently serves docs for pub.dev, api.dart.dev, and api.flutter.dev, will also work fine.

When a page is loaded initially, the browser will fill in the DOM just as it does today, except the sidebars will be empty. On load, some JS will read a data property of some tag (body?) for the left sidebar, build a URL for the HTML file for the desired sidebar, and request it. It then inserts the sidebar into its proper place in the DOM. The JS will do the same for the right sidebar.

Linking

As an optimization, to reduce page load (typical SPA optimization): we can add click handlers in the JS for all links destined for other pages within the same doc site.

When a link to another page in the same site is clicked (like a doc comment reference, or a type in a signature, or a member with a class, ...):

the JS intercepts it and instead requests the URL (the same HTML file destination) asynchronously,
parses the HTML response,
pushes the new location via the JS Navigation API
replaces the current <header> section with that of the response
replaces the current <main> section with that of the response
replaces the <title> tag (other stuff in <head>?)
requests the new page's left sidebar and right sidebar
replaces the left sidebar with the new one, and the right sidebar with the new one

The text was updated successfully, but these errors were encountered:

Levi-Lesches · 2022-07-25T19:31:48Z

Some questions:

Would an <iframe> of the sidebar be simpler? The link to the iframe's src can be generated as part of the regular process.
Would this affect SEO? I suppose not, since the class/enum/etc's main page will still have all its members, and each member gets its page to link to.
Does this make Provide an option to generate fewer (larger) pages. #1272 unacceptably worse? It's still O(n) files, but an extra two files for every container is still a lot. On the other hand, it produces shorter files.

If embedded HTML is the way to go, not a complete JSON overhaul, I can look into this.

srawlins · 2022-07-25T20:10:42Z

Would an <iframe> of the sidebar be simpler? The link to the iframe's src can be generated as part of the regular process.

Definitely. I think it reduces or removes any ability to do JS in there.

Would this affect SEO? I suppose not, since the class/enum/etc's main page will still have all its members, and each member gets its page to link to.

I don't think so.

Does this make Provide an option to generate fewer (larger) pages. #1272 unacceptably worse? It's still O(n) files, but an extra two files for every container is still a lot. On the other hand, it produces shorter files.

I don't think so. Just a little worse.

srawlins · 2022-10-13T14:19:13Z

CC @sigurdm @jonasfj regarding the idea of serving each page as 3 files: header/main/footer + left sidebar + right sidebar

srawlins · 2023-04-07T18:39:12Z

CC @jcollins-g we talked about this design yesterday.

jonasfj · 2023-04-11T12:19:05Z

Other similar options might be:

<iframe> with CSS to make it appear as embedded HTML, and <base target="_parent"> within the iframe.
(I'm not sure if this can be made to feel nice, and whether or not loading will be janky)
Not doing client-side navigation and always just doing full navigation. Obviously, this might be more janky.
Consider server-side includes

To be clear, I think the idea in #3384 is solid, and if we maintain the ability for the javascript navigation to be easy to disable, it wouldn't be hard for pub.dev to adopt.

For pub.dev, we would probably prefer some variant of (3).
In fact, I think we would love it if the HTML file was something like:

<html>
<head>
  <!--#include file="static/template_head.html" -->
</head>
<body>
  <header><!--#include file="header.html" --></header>
  <main><!--#include file="content.html" --></main>
  <aside><!--#include file="sidebar_left.html" --></aside>
  <aside><!--#include file="sidebar_right.html" --></aside>
  <div><!--#include file="footer.html" --></div>
</body>
</html>

(maybe having header and content on two files is overkill).

But the point is that we would like to load the HTML, sanitize it and put it into a template we maintain.

We could also do that with the SPA solution, if the javascript doing the client navigation is placed in a separate file, such that we can exclude it and concatenate the files serverside before serving to the user.

Our new code path for serving dartdoc is here (we haven't migrated traffic to this code path yet):

https://github.com/dart-lang/pub-dev/blob/master/app/lib/task/handlers.dart
- Loads HTML files
- Parses HTML files with DartDocPage.parse
- Caches the DartDocPage
- Renders the HTML sections into a custom template using DartDocPage.render
https://github.com/dart-lang/pub-dev/blob/master/app/lib/dartdoc/dartdoc_page.dart
- Parses an HTML files generated by dartdoc into sections: header, footer, main, sidebar left/right, title, description, breadcrumbs.
- Runs sections through sanitize_html
- Puts sections back into an HTML template that looks like what dartdoc normally uses.

The aim with this is to ensure that if the server running dartdoc is compromised, it will not be able to mess up our template structure, inject HTML, CSS or javascript.

If the VM running dartdoc is compromised it would be able to mess with the contents of the generated HTML pages sure. It could make things look weird, or provide documentation that is incorrect. But it wouldn't be able to inject harmful scripts/html into pages.

Someday, this may enable us to run {@tools when generated documentation on pub.dev. For now the primary motivation is that we could be able to embed dartdoc generated documentation on the package page, such that generated dartdoc doesn't feel like a separate website.

jcollins-g · 2023-04-11T16:12:41Z

Linking

The JS needs to also add click handlers for all links destined for other pages within the same doc site.

Why does the JS need to handle links destined for other pages? Is this an optimization?

As discussed offline, I think this aligns well with Reducing dartdoc generated file footprint. The impacts of the two problems are multiplicative and dartdoc output could get quite small if we implement both of them.

srawlins · 2023-04-11T16:51:47Z

Why does the JS need to handle links destined for other pages? Is this an optimization?

I need to correct that. I realized that's a bad design because it wouldn't support things like "Right click -> Copy link destination." and in implementing, it was easier to just make links point to the right place, statically, and every click is a full reload.

jonasfj · 2023-04-11T17:38:47Z

I realized that's a bad design because it wouldn't support things like "Right click -> Copy link destination."

It could be made to work. But it's a nin-trivial pain to do -- and testing it is difficult.

Same with history management, it's also doable, just mildly painful.

Serving plain HTML has a lot going for it.

Levi-Lesches · 2023-07-19T06:53:02Z

@srawlins I'm really impressed by the sidebars PR #3384, but it broke my docs (in version 6.3.0) :(
Here's what the sidebars should look like

And here's what a method/property/constructor page actually looks like:

Clearly the ProtoSocket-class-sidebar.html file exists and loads fine, but I noticed the constructor/method/property pages don't even request it!
Here are the network requests for the ProtoSocket page:

And here are the requests for the init method:

Here's the code for the main-content on the ProtoSocket page:

<div id="dartdoc-main-content" class="main-content" data-above-sidebar="burt_network/burt_network-library-sidebar.html" data-below-sidebar="burt_network/ProtoSocket-class-sidebar.html">

And here's the code for the init method:

<div id="dartdoc-main-content" class="main-content" data-above-sidebar="burt_network/ProtoSocket-class-sidebar.html" data-below-sidebar="">

When I edit the HTML to put the sidebar on the right as well, it loads... but only on the right side. My repository is here, feel free to ask for more info.

srawlins · 2023-07-19T16:07:45Z

Thanks for the report, @Levi-Lesches could you open an issue for what you're seeing?

Levi-Lesches · 2023-07-20T07:46:06Z

Filed #3467

srawlins · 2024-07-19T18:00:18Z

Forgot to close this when I shipped it :D

srawlins added the type-enhancement A request for a change that isn't a bug label Jul 12, 2022

jcollins-g added the P2 A bug or feature request we're likely to work on label Jul 14, 2022

jcollins-g added the type-performance Issues related to slow dartdoc generation. label Mar 24, 2023

srawlins mentioned this issue Apr 7, 2023

Split sidebars out into separate HTML files #3384

Merged

3 tasks

Levi-Lesches mentioned this issue Jul 20, 2023

Left sidebar not showing on some docs #3467

Closed

srawlins closed this as completed Jul 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Single Page App design with static files #3079

Single Page App design with static files #3079

srawlins commented Jul 12, 2022 •

edited

Loading

Levi-Lesches commented Jul 25, 2022 •

edited

Loading

srawlins commented Jul 25, 2022

srawlins commented Oct 13, 2022

srawlins commented Apr 7, 2023

jonasfj commented Apr 11, 2023 •

edited

Loading

jcollins-g commented Apr 11, 2023

Linking

srawlins commented Apr 11, 2023

jonasfj commented Apr 11, 2023

Levi-Lesches commented Jul 19, 2023 •

edited

Loading

srawlins commented Jul 19, 2023

Levi-Lesches commented Jul 20, 2023

srawlins commented Jul 19, 2024

Single Page App design with static files #3079

Single Page App design with static files #3079

Comments

srawlins commented Jul 12, 2022 • edited Loading

Background

Design

Generation

Initial page load

Linking

Levi-Lesches commented Jul 25, 2022 • edited Loading

srawlins commented Jul 25, 2022

srawlins commented Oct 13, 2022

srawlins commented Apr 7, 2023

jonasfj commented Apr 11, 2023 • edited Loading

jcollins-g commented Apr 11, 2023

Linking

srawlins commented Apr 11, 2023

jonasfj commented Apr 11, 2023

Levi-Lesches commented Jul 19, 2023 • edited Loading

srawlins commented Jul 19, 2023

Levi-Lesches commented Jul 20, 2023

srawlins commented Jul 19, 2024

srawlins commented Jul 12, 2022 •

edited

Loading

Levi-Lesches commented Jul 25, 2022 •

edited

Loading

jonasfj commented Apr 11, 2023 •

edited

Loading

Levi-Lesches commented Jul 19, 2023 •

edited

Loading