Pre- and post-process hooks

[noamross]

What is the scope or timeline for pre- and post-process hooks for developers to create complex formats that require more than templates and lua filters? Some existing examples of this in the R Markdown include capturing and including source data as base64-encoded data in the final document, or more complex MS Word and Powerpoint formats that implement features not in Pandoc's data model. Both of these uses pre-parse and capture information from the document to store as knitr metadata or sidecar files, then use that data to insert into the document before or after passing through Pandoc. Is this functionality expected in Quarto? Would pre- or post-processors need to be implemented in Typescript or would they be able to use the developer's language of choice?

Understanding this would help me decide whether to continue to develop on top of R Markdown for some use-cases or start porting existing work to Quarto.

[rgaiacs]

Hello @noamross.

capturing and including source data as base64-encoded data in the final document

What type of data are we talking about? Pandoc support convert images to base64. For example,

When writing, I use

---
title: "MWE"
author: "Myself"
format: html
---

but, for the final document, I use

---
title: "MWE"
author: "Myself"
format: 
  html:
    self-contained: true
---

It is faster to update the HTML without the base64 images but the base64 images help when sharing the document over email as I don't need to zip all the files together.

[noamross]

Thanks, @rgaiacs. I actually speaking of a different set of features. html_document, for instance has a code_download option that embeds the entire source .Rmd file to download. This is implemented with the pre-rendering hook (https://github.com/rstudio/rmarkdown/blob/d23e479017ceec894cc62c6b98f40f4188a4a75f/R/html_document.R#L510-L518), and similarly, for prerendered shiny documents, a pre-knit hook parses the Rmd file code to find other files, such as data files, that should be packaged with the document: https://github.com/rstudio/rmarkdown/blob/d23e479017ceec894cc62c6b98f40f4188a4a75f/R/shiny_prerendered.R#L242

[jjallaire]

Hi @noamross, there are actually two answers to this question:

(1) We didn't implement pre and post processing hooks yet mostly because we wanted to fully understand the use cases / requirements (i.e. raw content processing, file system side effects, synthesizing metadata, etc.) so we could create the right API. I was actually planning on reaching out to you to discuss further as I know you've probably done more than almost anyone with these hooks in R Markdown. Ping me over email and we can setup a time to talk realtime.

(2) We have attempted to create common implementations for many of the common operations of pre and post-processing hooks (e.g. keep-tex, keep-md, etc. One of these happens to be keep-source which if you set to true will embed the source code of the input file into the document (within a script tag). There is also a code-tools option which builds on this to let the user see the source code (but you could just set keep-source e.g. in a custom format and then find the script tag and do what you like with it).

Look forward to talking and getting this API right! One other thing to address: we are biasing in favor of having extensions/formats written in a runtime guaranteed to be available for all users of Quarto (which implies either Lua or Typescript). If we start having people use R and Python they will then also have package dependencies (likely with version sensitivities) which will make it exceedingly difficult to create extensions that can be successfully installed by the majority of users.