format keyword in document not overriding _metadata.yml

[anielsen001]

Bug description

the format keyword in the yaml block of a document is not overriding the format settings from the _metadata.yml in the same directory.

Steps to reproduce

I've created a repo here which demonstrates the issue https://github.com/anielsen001/quarto-yaml-test

in this project, run

quarto render test/test.qmd

and get the following output:

Executing 'test.ipynb'
  Cell 1/1: ''...Done

pandoc 
  to: html
  output-file: test.html
  standalone: true
  title-prefix: Yamltest
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  css:
    - ../styles.css
  toc: true
  
metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  theme: cosmo
  title: test file
  

Executing 'test.ipynb'
  Cell 1/1: ''...Done

pandoc 
  to: ipynb
  output-file: test.ipynb
  standalone: true
  default-image-extension: png
  
metadata
  ipynb-title-block: /opt/quarto/share/formats/ipynb/templates/title-block.md
  title: test file
  
Output created: ../_site/test/test.html

While I've set the format keyword in the test/test.qmd file to be only html, I still get the jupyter notebook output specified in the test/_metadata.yml file.

Expected behavior

I expect the yaml block in the file to override the directory settings in the_metadata.yml file. At least that's my interpretation of this section of the documentation: https://quarto.org/docs/projects/quarto-projects.html#metadata-merging

Actual behavior

I'm trying to suppress the jupyter notebook output on a per file basis when the _metadata.yml file specifies this behavior for the entire directory.

Your environment

text file editor

Quarto check output

quarto check
Quarto 1.4.554
[✓] Checking versions of quarto binary dependencies...
Pandoc version 3.1.11: OK
Dart Sass version 1.69.5: OK
Deno version 1.37.2: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
Version: 1.4.554
Path: /opt/quarto/bin

[✓] Checking tools....................OK
TinyTeX: (not installed)
Chromium: (not installed)

[✓] Checking LaTeX....................OK
Using: Installation From Path
Path: /usr/bin
Version: 2019

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
Version: 3.12.1
Path: /home/apn/proj/quarto/metadata_override/yamltest/venv/bin/python3
Jupyter: 5.7.2
Kernels: python3

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........(None)

  Unable to locate an installed version of R.
  Install R from https://cloud.r-project.org/

[mcanouil]

Thanks, I can reproduce.
Contrary to what is documented, format is merged.

The one exception to metadata merging is format. If the document-level YAML defines a format, it must define the complete list of formats to be rendered.

I made a slightly different example: https://github.com/mcanouil/quarto-issues-experiments/tree/main/quarto-cli-9605

This behaviour was already there in 1.0.38 so not really new.
I am not sure when format from document frontmatter was indeed not merge.

[cderv]

I believe this is related to

• [docs] Format in _metadata.yml are merged with in-document and used in priority over project config #8968

The directory level metadata and the document level metadata are considered a single set of metadata. If they provide format key then it will be considered as formats over the project directory.

The one exception to metadata merging is format. If the document-level YAML defines a format, it must define the complete list of formats to be rendered.

What this means is that

• format can be defined at project level, and it will apply to all document in the project.
  • For example it can defines defaults to typst and html format
• Though if a document metadata provide configuration for one of them (html or typst in the exemple), then only those formats will be considered
  • For example, _quarto.yml defines typst and html
  • about.qmd defines only html - then this document should not be rendered to typst.
• Directory metadata with _metadata.yml are also considered the same.
  • For example, _quarto.yml defines typst and html
  • dir/about.qmd defines only html
  • dir/_metadata.yml defines only docx
  • Then it means about.qmd can be rendered to docx or html but won't be rendered to typst when quarto render --to typst is called at project level.

This is the merging exception and why complete list must be defined. It is not a bug

I hope this clarify how it works.

• The important part is If the document-level YAML defines a format, it must define the complete list of formats to be rendered.
• The missing part is that it also applies to document-level AND directory level

I made a slightly different example: mcanouil/quarto-issues-experiments@main/quarto-cli-9605

In this example

• _metadata.yml defines format: typst
• _quarto.yml only defines format: html
• document.qmd defines html
  • It will be rendered to typst and html when calling quarto render
• metadat.qmd and project.qmd defines no format. So only the directory level will apply and they will be render to typst pdf only when calling quarto render

[mcanouil]

If the document-level YAML defines a format, it must define the complete list of formats to be rendered.

I don't read this as you do.

If document.qmd has format: whatever then it only renders "whatever". For document.qmd to be rendered to "all formats", "it must define the complete list of formats".

If there is an explicit format at a document level, it should never be merged with _metadata.yml or _quarto.yml.

Currently, it means you cannot set a directory level format and change that for a subset of document.
In regards, to format, I strongly feel that _metadata.yml should work as _quarto.yml.
Otherwise, I don't see benefits of setting format in _metadata.yml unless you want this to apply to absolutely all documents with no exception.

This thread is indeed a duplicate of:

• [docs] Format in _metadata.yml are merged with in-document and used in priority over project config #8968

Although, I don't believe this to be a documentation issue, see above.

---

Edit: an example to highlight why the current behaviour is not desirable

Assume you have a directory with ten documents.
You want a default to Typst and HTML in this directory.
Unfortunately, for one of the document, you are using something that cannot work in Typst.

Currently, it means to set:

• format: html in _metadata.yml

• in nine out of the ten documents:

  format:
    html: default
    typst: default

Because you cannot disable a format set in _metadata.yml contrary to _quarto.yml.

What seems more logical would be to set:

• in _metadata.yml (but you don't want to set the equivalent for a website with listings and add format everywhere):

  format:
    html: default
    typst: default

• in one out of the ten documents: format: html.

[cderv]

I think you disagree with how it was designed from the start, which is fine but this does not make it a bug.

It can still be rethought with consideration of possible breaking change. What I am sharing is not based on documentation interpretation but on how our code works as of now.

I believe it was introduced three years ago, with the commit message "correct format merging for directory level _metadata.yml"
c42196a

Comment in our code mentions this specificity for format keys.

quarto-cli/src/command/render/render-contexts.ts

Line 418 in 76d4ae6

// determine formats (treat dir format keys as part of 'input' format keys)

This is only the case for the list of target formats—the configuration of formats and any other metadata are merged, in the overwrite sense.

So I labeled this a document issue because our documentation does not clearly explain this specific current behavior. I am not saying it is perfect; I am just saying our document does not explain this specificity well.

unless you want this to apply to absolutely all documents with no exception.

Honestly, I think it was designed like this: to avoid adding a format: key to each file if you want to have a whole folder with a different format, like a format: revealjs for a presentation folder. The exclusion case may not have been envision at the time, probably only the configuration merging case.

My understanding is that

• You can have the format defined at the _quarto.yml level, which will apply to all docs in the project.
• Though there is a mechanism so that the list of target format can be made different at the document level (possibly to have a subset only)
• _metadata.yml is not a _quarto.yml-like subdirectory but a way to split some document metadata to be shared across documents.

I understand your specific example, and there could also be opposite specific examples where current behavior is easier.

For example, you want a default to Typst and HTML in this directory.
Unfortunately, for one of the documents, you need to overwrite a metadata value for the format: html. You want this doc to have a specific config different from all others. for example, you want smooth scrolling on this page for some reason.

format: 
   html: 
      smooth-scroll: true

By doing this, this document will now not be rendered to Typst because only format: html is listed. You would need to add typst: default to explicitly add to the list of formats so that the format: typst config from _metadata.yml can apply. If you need to configure some other differences per format, then you would need to add all formats in each document. _metadata.yml is less useful in this case.

In this case, I think one is happy that _metadata.yml applies to all docs to avoid duplication while allowing customization of some formats at the document level.

What I want to illustrate here is that this kind of feature is hard to make perfect and work for all use cases. They are design choices that have implications.

Both examples are opposite and have problems. It is a matter of choosing one behavior or the other. To me, it is more of a design choice, possibly changing our previous choices. It does not make it a bug, IMO.
It is not always easy to change, so we need to think through it carefully and keep in mind that if we change this one, we can possibly break some existing projects.

So, current situation has to be explained first, then this has to be discussed if we want to change.

I hope this explains my previous reply and clarifies what I wanted to share

[mcanouil]

Thanks for the code references.
From that, it's indeed a documentation issue.

My point is mostly based on several use cases and this report.
The current behaviour feels counterintuitive and less beneficial or used, but yes that's a design choice that would need further discussion.

Meanwhile about the documentation:

• _metadata.yml / document front matter overwrite format in _quarto.yml.
• document front matter format is merged with _metadata.yml format.

Note that by "overwrite" I mean merge the option of the formats in common under format and by "merged" I mean merge everything under format.

Is that correct?

[cderv]

Note that by "overwrite" I mean merge the option of the formats in common under format and by "merged" I mean merge everything under format.

Here's the relevant code piece

quarto-cli/src/command/render/render-contexts.ts

Lines 420 to 424 in 76d4ae6

	const projFormatKeys = formatKeys(projMetadata);
	const dirFormatKeys = formatKeys(directoryMetadata);
	const inputFormatKeys = ld.uniq(
	formatKeys(inputMetadata).concat(dirFormatKeys),
	);

This means that formats list is built in two set

• formats defined in project
• formats defined in document and shared metadata from directory

The latter overwrite the former, except in specific project type where only project formats are valid (only book for now)

Then for each format on this computed formats list, metadata are resolved for each type (Project, Directory, Document)

quarto-cli/src/command/render/render-contexts.ts

Lines 453 to 473 in 76d4ae6

	// resolve formats for each type of metadata
	const projFormats = await resolveFormatsFromMetadata(
	projMetadata,
	target.input,
	formats,
	options.flags,
	);
	const directoryFormats = await resolveFormatsFromMetadata(
	directoryMetadata,
	target.input,
	formats,
	options.flags,
	);
	const inputFormats = await resolveFormatsFromMetadata(
	inputMetadata,
	target.input,
	formats,
	options.flags,
	);

This is where the format list derived from project or document + directory matters.

Then all the selected formats' configuration are merged together

• For all formats
  

  quarto-cli/src/command/render/render-contexts.ts

  Lines 489 to 497 in 76d4ae6

  	// A list of all the format keys included
  	const allFormatKeys = ld.uniq(
  	Object.keys(projFormats).concat(Object.keys(directoryFormats)).concat(
  	Object.keys(inputFormats),
  	),
  	);
  	const mergedFormats: Record<string, Format> = {};
  	for (const format of allFormatKeys) {

• Merging in this order : configurations for a format from Input overrides those from directory which override those from project.
  

  quarto-cli/src/command/render/render-contexts.ts

  Lines 536 to 541 in 76d4ae6

  	// combine user formats
  	const userFormat = mergeFormatMetadata(
  	projFormat || {},
  	directoryFormat || {},
  	inputFormat || {},
  	);

This shows this is a two steps process for Quarto rendering

• Which are the target formats to consider ?
• How are the configuration merged for each targeted format ?

Hope this clarifies and answer your last question.

[mcanouil]

I am changing the labels to enhancement and "to open" a discussion on a possible change.

[anielsen001]

@cderv thank you for sharing your description of how and why the format key works the way it does at present. I had not considered that use case.

Perhaps the one possible enhancement at the document level would be to explicitly disable a particular output type by setting the format keyword to false.

format:
    html: false

In my particular case I have many files which I would like rendered to html and ipynb, but some of them don't need the ipynb output. I'd like to define the format spec in the _metadata.yml to share with all the files in the directory, but then disable some outputs for some files.