Markdown processing pipeline design

Create a diagram and brief explanation of the phases in markdown processing and how data will be passed between phases.

Demonstrate the contents of the markdown after each transformation

Related to #222 #243 #229 #137

Possible steps:

• use ocaml-mdx and dune to compile snippets
• generate table of contents and inject heading ids into markdown or generated html
• render markdown to html
• apply code highlight at build time or when displaying
• check that internal links point to valid pages - can implement this with two phases: dump links to a file, and check link file

Are there hooks in the nextjs build process to fire a task before all pages build or after all pages have been built? Alternatively, should we introduce inotify-wait to implement a third command make markdown:watch?

We should use a different file extension for inputs / outputs of each stage. Possibly:

.omdx, .md, .mdtoc

Leaning towards Ashish's suggestion of invoking an external operating system process (invoking ocaml logic) from rescript as the first step of processing a markdown file, where the process writes the result to stdout, and rescript continues working with the result.

We need to experiment with invoking ocaml-mdx soon, in order to get more clarity on what is possible. In particular, will dune become a runtime dependency of the nextjs build?

[agarwal]

So far I'm thinking of a system with 3 build tools in this order:

1. dune. Builds an ocaml CLI tool ocamlorg. Then contains other rules that call ocamlorg to build various files. These rules might generate .md, .yaml, and even .res files.
2. rescript. Now ReScript takes over. It continues the build by compiling .res to .js files.
3. next. Last, NextJS does its work.

All 3 tools support a watch mode, so ideally a developer launches all 3 and clearly gets incremental builds across all stages.

I would move that comment into a new issue. This issue is for the specifics of integrating ReScript with remark, omd, and ocaml-mdx, possibly using nodejs child process. The comment above addresses this at a high level, but also is useful for structuring the build for other processing like #140 .

It will be useful to understand rescript-lang's highlightjs integration (https://github.com/rescript-association/rescript-lang.org/blob/master/src/common/HighlightJs.res) while contemplating this design.

Starting to add diagrams and descriptions of options here - https://docs.google.com/document/d/1dCfHnYwCM8-lCZZ38ZwpLBrP_Ru6NISBHS12qO2Wo90/edit?usp=sharing

I'm going to run with spikes to explore how the following combination might work:

• when a page is building and runs getStaticProps, we call out to a node child process which runs ocaml-mdx to deal with code snippets. the result of the step will be a string written to stdout and read by the node child process library. (related to Evaluate ocaml code in markdown files #229)
• parse the values in the string from prior phases
• generate the table of contents with custom code operating on the AST generated by remark-js
• pass the table of contents data to the table of contents component. pass the markdown body to rehype-react (or mdxjs)
• integrate with highlight.js to enhance output from rehype-react (is this feasible?) (related to Implement code highlighting #137)

Based on the findings from the spikes, I will either continue pursuing that design or regroup and come up with a new design or select one of other alternatives.