Should a MetaString be rendered literally or escaped?

[bmaupin]

Pandoc version: 2.0, 1.19.2.1

Command:

echo "# Test" > test.md
pandoc -o test.epub -M title="<this> & that" test.md

Expected: The title should be correctly escaped in title_page.xhtml as it is in content.opf:

<dc:title id="epub-title-1">&lt;this&gt; &amp; that</dc:title>

Actual: Title in title_page.xhtml isn't properly escaped:

<h1 class="title"><this> & that</h1>

This causes validation errors:

Type	File	Line	Position	Message
ERROR	EPUB/text/title_page.xhtml	17	27	Error while parsing file 'element "this" not allowed here; expected the element end-tag, text or element "a", "abbr", "area", "audio", "b", "bdi", "bdo", "br", "button", "canvas", "cite", "code", "command", "datalist", "del", "dfn", "em", "embed", "epub:switch", "i", "iframe", "img", "input", "ins", "kbd", "keygen", "label", "link", "map", "mark", "meta", "meter", "ns1:math", "ns2:svg", "object", "output", "progress", "q", "ruby", "s", "samp", "script", "select", "small", "span", "strong", "sub", "sup", "textarea", "time", "u", "var", "video" or "wbr" (with xmlns:ns1="http://www.w3.org/1998/Math/MathML" xmlns:ns2="http://www.w3.org/2000/svg")'.
FATAL	EPUB/text/title_page.xhtml	17	29	Fatal Error while parsing file 'The entity name must immediately follow the '&' in the entity reference.'.

[jgm]

Variables are treated as literal strings. If you set title as a variable, it's your responsibility to do your own escaping. If you set it using --metadata, by contrast, pandoc will do the escaping.

[jgm]

PS. I would not recommend using `--variable` to set the title or other epub metadata. `--variable` says: put this literal content in the appropriately marked spot in the template. But for EPUB titles (and authors etc.) you want this to be part of the document's metadata. Summary: Use a YAML metadata block or `--metadata` on the command line.

[bmaupin]

Summary: Use a YAML metadata block or --metadata on the
command line.

-M is the same as --metadata, right?

pandoc --help | grep -- --metadata
  -M KEY[:VALUE]        --metadata=KEY[:VALUE]

If you set it using --metadata, by contrast, pandoc will do the escaping.

It seems pandoc is escaping the title everywhere except within the body of the title page, which causes a fatal error during validation:

$ unzip test.epub
...
$ grep this * -R
EPUB/nav.xhtml:  <title>&lt;this&gt; &amp; that</title>
EPUB/nav.xhtml:  <h1 id="toc-title">&lt;this&gt; &amp; that</h1>
EPUB/toc.ncx:    <text>&lt;this&gt; &amp; that</text>
EPUB/toc.ncx:        <text>&lt;this&gt; &amp; that</text>
EPUB/content.opf:    <dc:title id="epub-title-1">&lt;this&gt; &amp; that</dc:title>
EPUB/content.opf:    <reference type="toc" title="&lt;this&gt; &amp; that" href="nav.xhtml" />
EPUB/text/title_page.xhtml:  <title>&lt;this&gt; &amp; that</title>
EPUB/text/title_page.xhtml:  <h1 class="title"><this> & that</h1>

[jgm]

Sorry I missed that you were using --metadata all along. I understand the issue now.

[jgm]

If you specify the title in metadata inside the Markdown document (YAML block or a pandoc title block), everything should work. Metadata values specified that way are parsed as Markdown and will be escaped properly. But metadata values specified using --metadata are stored as MetaString rather than e.g. MetaInline values, and currently these will be copied literally into the corresponding variables.

I think it would probably make most sense to modify metaValueToJSON in Text.Pandoc.Writers.Shared so that MetaString s is treated like MetaInlines [Str s]. Then we'd get appropriate escaping. However, it's possible that this might break things for some current users, who may be expecting a MetaString to render literally.