Add Mathjax (KaTeX?) Support to Rustdoc

[Gankra]

$[stuff]$ in markdown should be interpreted by mathjax on the client-side to get pretty maths. NoScript fallback plan for the moment is to simply allow the raw latex to display. Having server-side generated images for fallback would involve substantial dependencies.

[steveklabnik]

This would also fix #15285

[Gankra]

Presumably since rustdoc is expected to work locally, we can't just use Mathjax's recommended process of just requesting the files from their CDN.

Holy shit mathjax is a 30,000 file 60MB monstrosity. Let's see what we can hack off here...

[Gankra]

I'm gonna go ahead and make the call that correct local rendering in Firefox is not worth including, quote, "29,000 files and 12.5MB" of png fonts.

[Gankra]

Okay so I sliced Mathjax down to 400 files clocking in at 2MB. There could be more to cut out, like all their context-menu localization stuff, but I'm rubbing up against some limits. It's a hardcore asynchronous library with a ton of modules that I lack a dependency graph for. Shouldn't (can't) be min-concated together, since it seems to lean pretty heavily on directory structure.

Still, this magnitude of a dependency is spooking some people. MathJax's docs actually pretty aggressively push "Just link to our CDN", which would make this literally like, a 2-line change to Rustdoc (pull in the file, configure the latex delimeters). It fails gracefully enough in the sense that you just end up with a bunch of raw latex, which IMO is pretty legible most of the time, e.g. $O(n^2)$. This failstate would happen anyway with noscript.

Is depending on an external CDN to fully render parts of the docs (on the client-side) acceptable, or is that a non-starter?

CC @cmr Any opinion?

[emberian]

I think depending on a CDN is fine.

On Wed, Aug 6, 2014 at 8:52 PM, Alexis Beingessner <[email protected]

wrote:

Okay so I sliced Mathjax down to 400 files clocking in at 2MB. There could
be more to cut out, like all their context-menu localization stuff, but I'm
rubbing up against some limits. It's a hardcore asynchronous library with a
ton of modules that I lack a dependency graph for. Shouldn't (can't) be
min-concated together, since it seems to lean pretty heavily on directory
structure.

Still, this magnitude of a dependency is spooking some people. MathJax's
docs actually pretty aggressively push "Just link to our CDN", which would
make this literally like, a 2-line change to Rustdoc (pull in the file,
configure the latex delimeters). It fails gracefully enough in the sense
that you just end up with a bunch of raw latex, which IMO is pretty legible
most of the time, e.g. $O(n^2)$. This failstate would happen anyway with
noscript.

Is depending on an external CDN to fully render parts of the docs (on the
client-side) acceptable, or is that a non-starter?

CC @cmr https://github.com/cmr Any opinion?

—
Reply to this email directly or view it on GitHub
#16300 (comment).

http://octayn.net/

[steveklabnik]

For something this non-essential, I agree.

[adrientetar]

How much math do we have?

[killercup]

(First off, I don't care about loading Mathjax from a CDN, so just ignore me if Mathjax is what you want.)

Some time ago I came across a small JS library that could convert some TeX math to unicode and HTML (depending on just a font IIRC), but I can't remember the name. Something like that might solve 99% of all cases of math in rust docs.

[huonw]

General remarks:

• I think we would need special handling for $...$ segments since they should effectively be a verbatim environment like ..., that is, $x < a, y > b$ should not be turned into an HTML <a> tag. AFAICT, hoedown doesn't support this natively (although presumably it would be able to be hacked in by just handling $ as a synonym for a backtick, with the appropriate matching). Also, I think this implies that this is required to be (somewhat) built-in to rustdoc; just inserting the mathjax JS into each page via --html-after-content won't work fully (i.e. @adrientetar's question applies to all libraries that wish to use rustdoc, e.g. scientific ones, physics/gamedev ones).
• if we were doing this, it would be nice to ensure we have display math support too (example where it would be nice).
• on that note, LaTeX-style delimiters \(...\) (inline) and \[...\] (display) are better matched than TeX $...$ and $$...$$ (and avoid the problem of people casually writing a $ for money)
• we should do something purposefully for instances of $...$ in the comments in code blocks (i.e. make some explicit decision about how to handle it, I don't care much); $ outside comments would need to be excluded from MathJaX (since it's an important part of macros).

[Gankra]

@huonw thanks for the detailed response!

• You're probably right that we'll need special handling on the server side. I didn't get very far last night due to weird compilation caching issues, so I didn't get a chance to run into that. Good catch.
• MathJax also supports display math. Their defaults are \(...\) for inline and both \[...\] and $$...$$ for display mode. I had opted to override this to $$...$$ for inline and \[...\] for display ($...$, as you note would be a nightmare with Rust's current grammar, and I guess financial references). I picked those because the match closely to the LaTeX I've seen, but am fine with your suggestion. I basically just like $$ because it's easy to type and looks less noisy than \(...\) to me, which seems desirable for an in-line notation.
• With either of our proposed settings, I don't think individual $'s need to be addressed. As far as I can tell, you wouldn't reasonably end up with $$ appearing in any code outside of a string literal. Maybe a weird regex or something...?

[huonw]

hoedown/hoedown#114 "MathJax support, second attempt"

[Gankra]

:O

!!!

I'm not sure what our policy on dependencies is. Can we depend on this snapshot of master, or should we wait for a stable release?

[alexcrichton]

It's ok to depend on non-master versions of dependencies, we already do it all the time for libuv and llvm anyway!

[Gankra]

@alexcrichton's lightning reflexes have gotten our hoedown updated to master! Working on updating rustdoc. I assume we want this to be behind a flag in rustdoc as well?

[alexcrichton]

If hoedown natively supports it (with no extra requirements) and it basically does what you want almost all of the time, we may not need to worry about gating it too much. The stability of rustdoc over time is in question though as we don't necessarily want to tie rustdoc to hoedown for life!

[alexcrichton]

In other words, being conservative and landing it behind something like #[doc(enable_mathjax)] at the crate level may be a good idea for now.

[Gankra]

@alexcrichton I'm unfortunately inexperienced with crate level configs, would you be able to recommend what exactly you would add to this patch to make the changes opt-in?

(Currently building Rust with the changes to confirm correctness)

[alexcrichton]

Ah because we have to pull in some extra JS from a CDN, this should most definitely be opt-in!

Crate-level configuration (inside of a #[doc] attribute) is currently detected here, and you could probably set some TLD variable (we don't pass around enough FooContext variables in rustdoc)

[huonw]

(We could theoretically detect if a page actually has any mathematics on it to conditionally insert the JS.)

[Gankra]

We don't have to pull in the JS, but the alternative is a many MB dependency. The conditional check is plausible, although I suppose it would require asking hoedown if math was found?

Anyway, I'll add a use_mathjax flag to the main Context variable (not sure if you're joking about not passing around enough contexts).

Edit: PS: thanks for the pointer, Alex!

[alexcrichton]

Ah yes, sorry, I was actually joking about the contexts and markdown because sadly a good bit of the markdown configuration is becoming global TLS variables as opposed to scoped contexts. In general though there are plenty of other contexts for rendering! (sometimes)

[Gankra]

Nice, it works! http://cg.scs.carleton.ca/~abeinges/doc/std/vec/struct.Vec.html

Just need to work out the right solution for getting the flag down from render.rs to hoedown. Of note is that currently we don't provide any way to configure Markdown instances that I see. I think I'm going to have to make Markdown stop being a tuple struct, and start taking some Config object. Unless we want to go down the road of bizarre static configs. Or maybe I could just introduce another tuple struct like, MarkdownWithMath, because that seems to be what MarkdownWithToc seems to be accomplishing?

[huonw]

One approach would be always set hoedown to render it, detect if any math occurs (useful for conditional JS insertion too), and to emit an error if the flag isn't enabled.

I think this approach is more "backwards compatible"?

[mildsunrise]

Nice, it works! http://cg.scs.carleton.ca/~abeinges/doc/std/vec/struct.Vec.html

Happy to see the math support in action! 😃
It's not perfect, so if block/inline is not guessed right, please let us know!

[Gankra]

Oh hey @jmendeth! <3 <3

Do you think it would be reasonable to add some way for hoedown to signal that it found Math, considering that mathjax isn't free to include clientside?

[mildsunrise]

Totally. I see you're already overriding some of Hoedown callbacks, then override the math callback too, and set a flag when it's called. I.e. add a field to your custom data structure on markdown.rs

 struct MyOpaque {
     dfltblk: extern "C" fn(*mut hoedown_buffer, *const hoedown_buffer,
                            *const hoedown_buffer, *mut libc::c_void),
     toc_builder: Option<TocBuilder>,
+    has_math: bool,
 }

then expose the math callback to Rust

 struct hoedown_renderer {
     opaque: *mut hoedown_html_renderer_state,
     blockcode: Option<extern "C" fn(*mut hoedown_buffer, *const hoedown_buffer,
                                     *const hoedown_buffer, *mut libc::c_void)>,
     blockquote: Option<extern "C" fn(*mut hoedown_buffer, *const hoedown_buffer,
                                      *mut libc::c_void)>,
     blockhtml: Option<extern "C" fn(*mut hoedown_buffer, *const hoedown_buffer,
                                     *mut libc::c_void)>,
     header: Option<extern "C" fn(*mut hoedown_buffer, *const hoedown_buffer,
                                  libc::c_int, *mut libc::c_void)>,
+    math: Option<extern "C" fn(*mut hoedown_buffer, *const hoedown_buffer,
+                                   libc::c_int, *mut libc::c_void)>,
     other: [libc::size_t, ..28],
 }

and override it with your own callback that, when called, sets has_math to true and passes the parameters to the original callback.

[killercup]

Just saw that this alternative to mathjax was published recently: https://github.com/Khan/KaTeX

It seems to be a lot faster and smaller than Mathjax (one 110k JS file, and 20k of CSS) and can be easily bundled with rustdoc. And even though it's 0.1.0, it appears quite stable and complete (from what I can tell after just entering some math into the online demo).

This will need some hand-coded JS to do the replacements, though.

[mahkoh]

Apparently this doesn't support \mathcal and friends.

[vks]

It does not seem to support any amsmath functionality. This includes matrices (i.e. \begin{pmatrix}) and fonts as @mahkoh mentioned. See [1] and [2].

[1] KaTeX/KaTeX#43
[2] KaTeX/KaTeX#42

[xymostech]

Hello! I'm one of the creators of KaTeX. I'd definitely love to get KaTeX to support rustdoc. We've just released this to get the word out, so it's not even close to feature complete, but are definitely planning on adding new functionality in the near future. Let me know what you need.

[huonw]

If we offer a way to just 'parse' the $$...$$ as mathematics without automatically inserting the MathJax JS then the KaTeX (or any other processor) can be inserted itself.

Providing --math-js URL flag to have the same lazy insertion also seems reasonable (but I'd be very happy doing this as a future extension).

[Gankra]

@xymostech I think our largest use case will just be asymptotics, which is trivial. Beyond that, libraries that are based on, or leverage, mathematical concepts might want to use this. Currently (in an experimental branch) we're using an external library to convert our math into the standard form MathJax supports, but it should be fairly easy to change the output to something else if necessary.

I'm concerned by the statement of "synchronous" on the project page. I certainly don't want page loading/reading to be blocked by richly rendering the math, which likely doesn't need to be rendered instantly. I'd much rather the math be lazily rendered while the user gets their bearings quickly scanning the page for the fns they want. Also blocking (the 100% clientside) search from working while the math is trying to grind through stuff strikes me as a no-no.

[xymostech]

@gankro Definitely agree! Blocking UI is a big problem. I guess the point of making KaTeX synchronous is that if you want to make it asynchronous, then you can. It doesn't have to happen with page load if you don't want. If you start with it asynchronous, it's hard to go the other way and make it synchronous. We'll probably add a way to to asynchronous loading in the future. However, if you're rendering a small amount of simple expressions, KaTeX can be nearly instantaneous.

Let me know if you want to try it out, I'd be happy to answer any questions along the way.

[Gankra]

I've closed my PR to add MathJax because I'm a bit busy with other stuff, and it was developing merge conflicts from age. If anyone wants to take a shot at it, they're welcome to steal what I had as a good starting point, though.

KaTeX also definitely seems worth evaluating!

[steveklabnik]

Given the decision in #17390 (comment), I'm closing this as well.