Processing translated document with footnote makes sphinx-builder crash

[k-yaegashi]

Summary

When processing a document using footnote text and translated using gettext, unexpected runtime error occured and sphinx crashed.

Detail

After running the following shell script as sphinx-test.sh on my home directory,

#!/bin/sh
DOCS_DEST=$HOME/sphinx-test
LF=$(printf '\\\012_')
LF=${LF%_}
sphinx-quickstart -q -p 'Test Document' -a 'Test Author' -v '0.0.1' --sep --makefile --batchfile $DOCS_DEST
cd $DOCS_DEST
sed -i -e 's/language = None/language = "ja"'"$LF"'locale_dirs = ["locale"]/' $DOCS_DEST/source/conf.py
sed -i -e 's/Indices and tables/This is a test footnote [#f1]_ .'"$LF"''"$LF"'.. [#f1] This is a test footnote text.'"$LF"''"$LF"'Indices and tables/' $DOCS_DEST/source/index.rst
make gettext
cd source
sphinx-intl update -p ../build/gettext -l ja
cd ..
POFILE=$DOCS_DEST/source/locale/ja/LC_MESSAGES/index.po
sed -i -e '23 s/msgstr ""/msgstr "テスト文書のドキュメンテーションへようこそ!"/' $POFILE
sed -i -e '27 s/msgstr ""/msgid "これはテストの脚注です [#f1]_ 。"/' $POFILE
sed -i -e '31 s/msgstr ""/msgid "これはテストの脚注のテキストです。"/' $POFILE
make html

I've got the following console output:

Creating file /Users/k-yaegashi/sphinx-test/source/conf.py.
Creating file /Users/k-yaegashi/sphinx-test/source/index.rst.
Creating file /Users/k-yaegashi/sphinx-test/Makefile.
Creating file /Users/k-yaegashi/sphinx-test/make.bat.

Finished: An initial directory structure has been created.

You should now populate your master file /Users/k-yaegashi/sphinx-test/source/index.rst and create other documentation
source files. Use the Makefile to build the docs, like so:
   make builder
where "builder" is one of the supported builders, e.g. html, latex or linkcheck.

Running Sphinx v1.5.1
making output directory...
loading translations [ja]... done
loading pickled environment... not yet created
building [gettext]: targets for 0 template files
building [gettext]: targets for 1 source files that are out of date
updating environment: 1 added, 0 changed, 0 removed
reading sources... [100%] index                                                 
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index                                                  
writing message catalogs... [100%] index                                        
build succeeded.

Build finished. The message catalogs are in build/gettext.
Create: locale/ja/LC_MESSAGES/index.po
Running Sphinx v1.5.1
making output directory...
loading translations [ja]... done
loading pickled environment... not yet created
building [mo]: targets for 1 po files that are out of date
writing output... [100%] locale/ja/LC_MESSAGES/index.mo                         
Exception occurred:
  File "/usr/local/lib/python2.7/site-packages/babel/messages/pofile.py", line 153, in _add_message
    string = denormalize(translations[0][1])
IndexError: list index out of range
The full traceback has been saved in /var/folders/hj/0rrp_ghx1qd5fwmyz1cvtxlh24332s/T/sphinx-err-ymWCcz.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [html] Error 1

and the following log error:

# Sphinx version: 1.5.1
# Python version: 2.7.9 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.3
# Last messages:
#   Running Sphinx v1.5.1
#   making output directory...
#   loading translations [ja]...
#   done
#   loading pickled environment...
#   not yet created
#   building [mo]: targets for 1 po files that are out of date
#   writing output... [100%] locale/ja/LC_MESSAGES/index.mo
# Loaded extensions:
#   sphinx.domains.cpp (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/cpp.pyc
#   sphinx.builders.applehelp (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/applehelp.pyc
#   sphinx.directives (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/directives/__init__.pyc
#   sphinx.builders.changes (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/changes.pyc
#   sphinx.builders.websupport (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/websupport.pyc
#   sphinx.domains.c (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/c.pyc
#   sphinx.builders.dummy (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/dummy.pyc
#   sphinx.builders.texinfo (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/texinfo.pyc
#   sphinx.builders.epub (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/epub.pyc
#   sphinx.builders.linkcheck (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/linkcheck.pyc
#   sphinx.domains.std (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/std.pyc
#   sphinx.directives.other (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/directives/other.pyc
#   sphinx.builders.epub3 (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/epub3.pyc
#   sphinx.domains.javascript (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/javascript.pyc
#   sphinx.directives.code (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/directives/code.pyc
#   alabaster (0.7.9) from /usr/local/lib/python2.7/site-packages/alabaster/__init__.pyc
#   sphinx.builders.text (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/text.pyc
#   sphinx.builders.qthelp (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/qthelp.pyc
#   sphinx.roles (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/roles.pyc
#   sphinx.builders.htmlhelp (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/htmlhelp.pyc
#   sphinx.directives.patches (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/directives/patches.pyc
#   sphinx.builders.latex (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/latex.pyc
#   sphinx.builders.html (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/html.pyc
#   sphinx.builders.devhelp (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/devhelp.pyc
#   sphinx.builders.gettext (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/gettext.pyc
#   sphinx.builders.xml (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/xml.pyc
#   sphinx.builders.manpage (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/manpage.pyc
#   sphinx.domains.rst (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/rst.pyc
#   sphinx.domains.python (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/python.pyc
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/sphinx/cmdline.py", line 296, in main
    app.build(opts.force_all, filenames)
  File "/usr/local/lib/python2.7/site-packages/sphinx/application.py", line 332, in build
    self.builder.compile_update_catalogs()
  File "/usr/local/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 205, in compile_update_catalogs
    self.compile_catalogs(catalogs, message)
  File "/usr/local/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 170, in compile_catalogs
    catalog.write_mo(self.config.language)
  File "/usr/local/lib/python2.7/site-packages/sphinx/util/i18n.py", line 59, in write_mo
    write_mo(mo, read_po(po, locale))
  File "/usr/local/lib/python2.7/site-packages/babel/messages/pofile.py", line 244, in read_po
    _process_message_line(lineno, line)
  File "/usr/local/lib/python2.7/site-packages/babel/messages/pofile.py", line 186, in _process_message_line
    _add_message()
  File "/usr/local/lib/python2.7/site-packages/babel/messages/pofile.py", line 153, in _add_message
    string = denormalize(translations[0][1])
IndexError: list index out of range

If any outputs are wrong on your environment, please use the following intended files instead and try make again.

$HOME/sphinx-test/source/index.rst:

.. Test Document documentation master file, created by
   sphinx-quickstart on Thu Jan 12 20:26:15 2017.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to Test Document's documentation!
=========================================

.. toctree::
   :maxdepth: 2
   :caption: Contents:



This is a test footnote [#f1]_ .

.. [#f1] This is a test footnote text.

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

$HOME/sphinx-test/source/locale/ja/LC_MESSAGES/index.po:

# SOME DESCRIPTIVE TITLE.
# Copyright (C) 2017, Test Author
# This file is distributed under the same license as the Test Document
# package.
# FIRST AUTHOR <EMAIL@ADDRESS>, 2017.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: Test Document 0.0.1\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2017-01-12 20:26+0900\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.3.4\n"

#: ../../source/index.rst:7
msgid "Welcome to Test Document's documentation!"
msgstr "テスト文書のドキュメンテーションへようこそ!"

#: ../../source/index.rst:15
msgid "This is a test footnote [#f1]_ ."
msgid "これはテストの脚注です [#f1]_ 。"

#: ../../source/index.rst:17
msgid "This is a test footnote text."
msgid "これはテストの脚注のテキストです。"

#: ../../source/index.rst:20
msgid "Indices and tables"
msgstr ""

#: ../../source/index.rst:22
msgid ":ref:`genindex`"
msgstr ""

#: ../../source/index.rst:23
msgid ":ref:`modindex`"
msgstr ""

#: ../../source/index.rst:24
msgid ":ref:`search`"
msgstr ""

My environment

• Sphinx (sphinx-build) 1.5.1
• Sphinx-intl 0.9.9
  • (Modules above are installed by pip)
• Python version 2.7.9
• Mac OS X 10.9.5
• All translated Japanese texts above are correctly encoded in UTF-8.

[shimizukawa]

Reading po causes crush because the index.po file is incorrect.

#: ../../source/index.rst:15
msgid "This is a test footnote [#f1]_ ."
msgid "これはテストの脚注です [#f1]_ 。"

msgid and msgid. later one must be msgstr.
Please correct your po file.

[shimizukawa]

Next version of sphinx will emits WARNING: reading error: /path/to/project/locale/ja/LC_MESSAGES/index.po if index.po is incorrect.

Now I close this issue. Please feel free to reopen if you still have problem. Thanks for you reporting!

[k-yaegashi]

Sorry for my mistake and I fixed the repro script:

#!/bin/sh
DOCS_DEST=$HOME/sphinx-test
MARK=*
PERFORM_TRANSLATE=true
LF=$(printf '\\\012_')
LF=${LF%_}
sphinx-quickstart -q -p 'Test Document' -a 'Test Author' -v '0.0.1' --sep --makefile --batchfile $DOCS_DEST
cd $DOCS_DEST
if [ "$PERFORM_TRANSLATE" == "true" ]; then
    sed -i -e 's/language = None/language = "ja"'"$LF"'locale_dirs = ["locale"]/' $DOCS_DEST/source/conf.py
fi
sed -i -e 's/Indices and tables/This is a test footnote ['"$MARK"']_ .'"$LF"''"$LF"'.. ['"$MARK"'] This is a test footnote text.'"$LF"''"$LF"'Indices and tables/' $DOCS_DEST/source/index.rst
make gettext
cd source
sphinx-intl update -p ../build/gettext -l ja
cd ..
POFILE=$DOCS_DEST/source/locale/ja/LC_MESSAGES/index.po
sed -i -e '23 s/msgstr ""/msgstr "テスト文書のドキュメンテーションへようこそ"/' $POFILE
sed -i -e '27 s/msgstr ""/msgstr "これはテストの脚注です ['"$MARK"']_ 。"/' $POFILE
sed -i -e '31 s/msgstr ""/msgstr "これはテストの脚注のテキストです。"/' $POFILE
make html

It will produce,
$HOME/sphinx-test/source/index.rst:

.. Test Document documentation master file, created by
   sphinx-quickstart on Fri Jan 13 09:14:21 2017.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to Test Document's documentation!
=========================================

.. toctree::
   :maxdepth: 2
   :caption: Contents:



This is a test footnote [*]_ .

.. [*] This is a test footnote text.

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

$HOME/sphinx-test/source/locale/ja/LC_MESSAGES/index.po:

# SOME DESCRIPTIVE TITLE.
# Copyright (C) 2017, Test Author
# This file is distributed under the same license as the Test Document
# package.
# FIRST AUTHOR <EMAIL@ADDRESS>, 2017.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: Test Document 0.0.1\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2017-01-13 09:14+0900\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.3.4\n"

#: ../../source/index.rst:7
msgid "Welcome to Test Document's documentation!"
msgstr "テスト文書のドキュメンテーションへようこそ"

#: ../../source/index.rst:15
msgid "This is a test footnote [*]_ ."
msgstr "これはテストの脚注です [*]_ 。"

#: ../../source/index.rst:17
msgid "This is a test footnote text."
msgstr "これはテストの脚注のテキストです。"

#: ../../source/index.rst:20
msgid "Indices and tables"
msgstr ""

#: ../../source/index.rst:22
msgid ":ref:`genindex`"
msgstr ""

#: ../../source/index.rst:23
msgid ":ref:`modindex`"
msgstr ""

#: ../../source/index.rst:24
msgid ":ref:`search`"
msgstr ""

and also produces following error on console:

Exception occurred:
  File "/usr/local/lib/python2.7/site-packages/docutils/nodes.py", line 567, in __getitem__
    return self.attributes[key]
KeyError: 'refid'

with stacktrace:

# Sphinx version: 1.5.1
# Python version: 2.7.9 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.3
# Last messages:
#   
#   looking for now-outdated files...
#   none found
#   pickling environment...
#   done
#   checking consistency...
#   done
#   preparing documents...
#   done
#   writing output... [100%] index
# Loaded extensions:
#   sphinx.domains.cpp (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/cpp.pyc
#   sphinx.builders.applehelp (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/applehelp.pyc
#   sphinx.directives (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/directives/__init__.pyc
#   sphinx.builders.changes (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/changes.pyc
#   sphinx.builders.websupport (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/websupport.pyc
#   sphinx.domains.c (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/c.pyc
#   sphinx.builders.dummy (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/dummy.pyc
#   sphinx.builders.texinfo (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/texinfo.pyc
#   sphinx.builders.epub (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/epub.pyc
#   sphinx.builders.linkcheck (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/linkcheck.pyc
#   sphinx.domains.std (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/std.pyc
#   sphinx.directives.other (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/directives/other.pyc
#   sphinx.builders.epub3 (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/epub3.pyc
#   sphinx.domains.javascript (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/javascript.pyc
#   sphinx.directives.code (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/directives/code.pyc
#   alabaster (0.7.9) from /usr/local/lib/python2.7/site-packages/alabaster/__init__.pyc
#   sphinx.builders.text (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/text.pyc
#   sphinx.builders.qthelp (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/qthelp.pyc
#   sphinx.roles (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/roles.pyc
#   sphinx.builders.htmlhelp (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/htmlhelp.pyc
#   sphinx.directives.patches (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/directives/patches.pyc
#   sphinx.builders.latex (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/latex.pyc
#   sphinx.builders.html (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/html.pyc
#   sphinx.builders.devhelp (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/devhelp.pyc
#   sphinx.builders.gettext (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/gettext.pyc
#   sphinx.builders.xml (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/xml.pyc
#   sphinx.builders.manpage (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/builders/manpage.pyc
#   sphinx.domains.rst (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/rst.pyc
#   sphinx.domains.python (builtin) from /usr/local/lib/python2.7/site-packages/sphinx/domains/python.pyc
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/sphinx/cmdline.py", line 296, in main
    app.build(opts.force_all, filenames)
  File "/usr/local/lib/python2.7/site-packages/sphinx/application.py", line 333, in build
    self.builder.build_update()
  File "/usr/local/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 251, in build_update
    'out of date' % len(to_build))
  File "/usr/local/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 322, in build
    self.write(docnames, list(updated_docnames), method)
  File "/usr/local/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 360, in write
    self._write_serial(sorted(docnames), warnings)
  File "/usr/local/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 368, in _write_serial
    self.write_doc(docname, doctree)
  File "/usr/local/lib/python2.7/site-packages/sphinx/builders/html.py", line 452, in write_doc
    self.docwriter.write(doctree, destination)
  File "/usr/local/lib/python2.7/site-packages/docutils/writers/__init__.py", line 80, in write
    self.translate()
  File "/usr/local/lib/python2.7/site-packages/sphinx/writers/html.py", line 48, in translate
    self.document.walkabout(visitor)
  File "/usr/local/lib/python2.7/site-packages/docutils/nodes.py", line 174, in walkabout
    if child.walkabout(visitor):
  File "/usr/local/lib/python2.7/site-packages/docutils/nodes.py", line 174, in walkabout
    if child.walkabout(visitor):
  File "/usr/local/lib/python2.7/site-packages/docutils/nodes.py", line 174, in walkabout
    if child.walkabout(visitor):
  File "/usr/local/lib/python2.7/site-packages/docutils/nodes.py", line 166, in walkabout
    visitor.dispatch_visit(self)
  File "/usr/local/lib/python2.7/site-packages/docutils/nodes.py", line 1882, in dispatch_visit
    return method(node)
  File "/usr/local/lib/python2.7/site-packages/docutils/writers/html4css1/__init__.py", line 501, in visit_footnote_reference
    href = '#' + node['refid']
  File "/usr/local/lib/python2.7/site-packages/docutils/nodes.py", line 567, in __getitem__
    return self.attributes[key]
KeyError: 'refid'

Please modify the repro script as follows:
If:

MARK=#note
PERFORM_TRANSLATE=true

it works without error (same as my first report without mistake ...).

And if:

MARK=*
PERFORM_TRANSLATE=false

it also works with no trouble. It means using asterisk * for footnote is naturally available without any translation.

But if:

MARK=*
PERFORM_TRANSLATE=true

it produces error.

Some documents written in sphinx seem to use asterisk * directly to realize footnote
(e.g. http://docs.bakefile.org/en/latest/intro.html)
even though "reStructuredText Primer" doesn't instruct so:
http://www.sphinx-doc.org/en/1.4.9/rest.html

So the essential distress I'd like to say is, when translating document using * as footnote mark, I cannot complete translation of troubled paragraph.

And I seem to be unable to reopen this issue by myself due to lack of privileges.

[tk0miya]

Reproduced. Surely, refid attribute was lost on translation::

-        <paragraph translated="True">
+        <paragraph>
-            これはテストの脚注です
+            This is a test footnote
-            <footnote_reference auto="*" ids="id1">
+            <footnote_reference auto="*" ids="id1" refid="id2">
+                *
-             。
+             .

[Endeavourl]

Spent 2 days dealing with this. Seems that [*]_ style footnotes crash builder.
Failed RTD build example: https://readthedocs.org/projects/pgcodekeeper/builds/7392040/