Parallel creation of parameterized reports fails due to file conflicts

[mschubert]

Bug description

I'm trying to generate parameterized reports in parallel on Ubuntu 22.04 LTS using quarto=1.2.335. For this, I'm using the following test.qmd file:

---
title: paralell report generation fails
params:
  counter: 1
format:
  html:
    embed-resources: true
---

\```{r}
Sys.sleep(5)
\```

This is report number `r params$counter`

Generating both report out1.html and out2.html works fine when running sequentially:

quarto render test.qmd -P counter:1 --output out1.html
quarto render test.qmd -P counter:2 --output out2.html

However, when I try to run them in parallel I get:

quarto render test.qmd -P counter:1 --output out1.html &;
quarto render test.qmd -P counter:2 --output out2.html
# Error in readLines(con, warn = FALSE) : cannot open the connection
# Calls: .main ... partition_yaml_front_matter -> grep -> is.factor -> read_utf8 -> readLines

I suspect this is because both reports store their temporary files in test_files/. Wouldn't it make more sense to use the output file base as directory, because this should not conflict?

Checklist

• Please include a minimal, fully reproducible example in a single .qmd file? Please provide the whole file rather than the snippet you believe is causing the issue.
• Please format your issue so it is easier for us to read the bug report.
• Please document the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
• Please document the operating system you're running. If on Linux, please provide the specific distribution.

[cscheid]

Yes, this is a known issue for us. We will improve this treatment in the future.

There are a large number of complicating factors with filenames and parameterized reports. Centrally, we don't perfectly control the creation of all files in all formats (for example, when going through the latex engine, it's not easy to ensure we don't end up with overwriting files).

In the meantime, the recommended solution is unfortunately to either serialize the creation or to use differently-named inputs.

[cderv]

Another workaround we shared in R Markdown too, as the issue is the same (difficulties are identical and not easy to overcome, that is why even in quarto it is not yet improved):

Use project scripts for the workflow (quarto project script or makefile for example), and duplicate project in temp folder (or other folder) - a different one per rendering in parallel - for the time of the rendering, and move results back in main project. This way conflicts should not happen as they are probably project related.

In the R Markdown ecosystem, someone has done a R package to help with this: https://github.com/petrelharp/templater

Related R Markdown issues with discussions:

• parallel render()'s with different output_file's overwrite each other rstudio/rmarkdown#499
• [FR] *.knit.md should have a unique name to allow parallelization rstudio/rmarkdown#2454

Just sharing for reference of past discussions on this topic

[mschubert]

My current workaround is:

# {out_base} is "out1" or "out2" in the above example
ln -s test.qmd {out_base}.qmd
quarto render {out_base}.qmd -P counter:1 --output {out_base}.html
unlink {out_base}.qmd

This works well for generating my parameterized reports 🚀