Nondeterministic build failures in `Tex('The horse does not eat cucumber salad.')`

[markromanmiller]

Description of bug / unexpected behavior

As I've been figuring out the test coverage toolset for Manim, I've done quite a lot of automated builds on a copied (not forked) repository (https://github.com/MrMallIronmaker/manim-cov/). For some reason, occasionally one of the Ubuntu builds fails while I've been playing around with coverage settings. Whether it's 3.7, 3.8, or 3.9 is apparently random.

All of them fail in the Tex doctest.

Expected behavior

It should pass the test.

How to reproduce the issue

Code for reproducing the problem

In theory the problematic code is here: ```py Tex('The horse does not eat cucumber salad.') ```

However, I've not been able to replicate this issue

Additional media files

Build output

Run poetry run pytest --cov-append --doctest-modules manim
  poetry run pytest --cov-append --doctest-modules manim
  shell: /usr/bin/bash -e {0}
  env:
    POETRY_VIRTUALENVS_CREATE: false
    pythonLocation: /opt/hostedtoolcache/Python/3.8.8/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.8/x64/lib
Skipping virtualenv creation, as specified in config file.
============================= test session starts ==============================
platform linux -- Python 3.8.8, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/runner/work/manim-cov/manim-cov, configfile: pyproject.toml
plugins: cov-2.11.1
collected 30 items

manim/_config/__init__.py .                                              [  3%]
manim/_config/utils.py ..                  
## Logs
<details><summary>Terminal output</summary>
<!-- Add "-v DEBUG" when calling manim to generate more detailed logs -->

PASTE HERE OR PROVIDE LINK TO https://pastebin.com/ OR SIMILAR

<!-- Insert screenshots here (only when absolutely necessary, we prefer copy/pasted output!) -->

</details>


## System specifications

<details><summary>System Details</summary>

- OS (with version, e.g Windows 10 v2004 or macOS 10.15 (Catalina)):
- RAM:
- Python version (`python/py/python3 --version`):
- Installed modules (provide output from `pip list`):

PASTE HERE

</details>

<details><summary>LaTeX details</summary>

+ LaTeX distribution (e.g. TeX Live 2020):
+ Installed LaTeX packages:
<!-- output of `tlmgr list --only-installed` for TeX Live or a screenshot of the Packages page for MikTeX -->
</details>

<details><summary>FFMPEG</summary>

Output of `ffmpeg -version`:

PASTE HERE

</details>

## Additional comments
<!-- Add further context that you think might be relevant for this issue here. -->
                              [ 10%]
manim/animation/animation.py .                                           [ 13%]
manim/mobject/geometry.py .........                                      [ 43%]
manim/mobject/mobject.py ......                                          [ 63%]
manim/mobject/svg/tex_mobject.py ..F                                     [ 73%]
manim/mobject/svg/text_mobject.py ...                                    [ 83%]
manim/mobject/types/vectorized_mobject.py ....                           [ 96%]
manim/utils/color.py .                                                   [100%]

=================================== FAILURES ===================================
_________________ [doctest] manim.mobject.svg.tex_mobject.Tex __________________
491 A string compiled with LaTeX in normal mode.
492 
493     Tests
494     -----
495 
496     Check whether writing a LaTeX string works::
497 
498         >>> Tex('The horse does not eat cucumber salad.')
UNEXPECTED EXCEPTION: IndexError('list index out of range')
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.8/x64/lib/python3.8/doctest.py", line 1336, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest manim.mobject.svg.tex_mobject.Tex[0]>", line 1, in <module>
  File "/home/runner/work/manim-cov/manim-cov/manim/mobject/svg/tex_mobject.py", line 506, in __init__
    MathTex.__init__(
  File "/home/runner/work/manim-cov/manim-cov/manim/mobject/svg/tex_mobject.py", line 392, in __init__
    self.break_up_by_substrings()
  File "/home/runner/work/manim-cov/manim-cov/manim/mobject/svg/tex_mobject.py", line 434, in break_up_by_substrings
    sub_tex_mob.move_to(self.submobjects[last_submob_index], RIGHT)
IndexError: list index out of range
/home/runner/work/manim-cov/manim-cov/manim/mobject/svg/tex_mobject.py:498: UnexpectedException
----------------------------- Captured stdout call -----------------------------
                    INFO     Writing "The horse does not  tex_file_writing.py:81
                             eat cucumber salad." to medi                       
                             a/Tex/3ecc83aec1683253.tex                         
------------------------------ Captured log call -------------------------------
INFO     manim:tex_file_writing.py:81 Writing "The horse does not eat cucumber salad." to media/Tex/3ecc83aec1683253.tex
=============================== warnings summary ===============================
manim/mobject/mobject.py::manim.mobject.mobject.Mobject.set
  <doctest manim.mobject.mobject.Mobject.set[1]>:1: DeprecationWarning: This method is not guaranteed to stay around. Please prefer setting the attribute normally or with Mobject.set().

manim/mobject/mobject.py::manim.mobject.mobject.Mobject.set
  <doctest manim.mobject.mobject.Mobject.set[2]>:1: DeprecationWarning: This method is not guaranteed to stay around. Please prefer getting the attribute normally.

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-84, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-104, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-101, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-111, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-114, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-115, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-100, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-110, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-116, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-97, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-99, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-117, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-109, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-98, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-108, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-46, which is not recognized
    warnings.warn(warning_text)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

----------- coverage: platform linux, python 3.8.8-final-0 -----------
Coverage XML written to file coverage.xml

=========================== short test summary info ============================
FAILED manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
================== 1 failed, 29 passed, 18 warnings in 6.56s ===================
Error: Process completed with exit code 1.

[naveen521kk]

with these warning

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-84, which is not recognized
    warnings.warn(warning_text)

manim/mobject/svg/tex_mobject.py::manim.mobject.svg.tex_mobject.Tex
  /home/runner/work/manim-cov/manim-cov/manim/mobject/svg/svg_mobject.py:261: UserWarning: media/Tex/3ecc83aec1683253.svg contains a reference to id #g0-104, which is not recognized
    warnings.warn(warning_text)

the problem is LaTeX installation, is it on CI?
(this was a well-known problem from Miktex I think. I think @behackl know more about this?)

[markromanmiller]

It's on CI, yeah. Github Actions - I copied over the regular manim ci system.

Thanks for the tip!

[naveen521kk]

I think dvisvgm? Don't know though. Also, why it isn't failing here?

[markromanmiller]

Good q. That does make me pause. It only fails 1/6 - 1/8 of the time (per ubuntu run), so maybe it hasn't been a problem yet? I haven't looked through the build indicators to see if any of them have the same issue but if this is still a mystery as we get closer to merging #1071 then I'll dig deeper.

The other possibilities are that it is something specific to my manim-cov repo because I copied it without forking, or specific to my changes, or it may have its roots in some recent change to the manim codebase. Seeing if there are errors on other branches here helps me narrow it down... Or sometimes it's just like MikTex out of the blue.

Another thing I want to try is modifying the code to dump the SVG file to the log when it breaks to see what it looks like and if it's an SVG parsing error on manim's end... The fact it's random is weird though.

[eulertour]

I encountered this in our CI on #1075

[behackl]

Now also when merging into master: https://github.com/ManimCommunity/manim/runs/2050314745?check_suite_focus=true

As of right now, it is completely unclear to me why this would fail. It looks like the error that occurs when the cm-super LaTeX package is not installed (and it would be curious to see whether changing the test to Tex("...", tex_template=TexTemplateLibrary.simple) would change anything) -- but as far as I can tell, cm-super should be included with the other system packages that are installed.

It worries me a little that this occurs randomly.

[markromanmiller]

Ok, so the problem appears to be upstream of each of these things... I copied based on e3359a5, so if it's manim-based problem, its origin is there or earlier.

I've not been able to replicate the bug on my local machine with the simple code I gave.

Are the hashed filenames different? I always see the 3ecc83... filename with the error, but it may be different normally? And the .tex files determine the hash, right? It may narrow down where in the pipeline the problem lies...

The warnings text is a warning I wrote, given off by

manim/manim/mobject/svg/svg_mobject.py

Line 261 in 0de873c

warnings.warn(warning_text)

- in short, the svg element defines a lot of objects (e.g., the path to write "e") and then the svg body has tags to insert those definitions into the document so it doesn't have to repeat itself.

In short, each warning is saying that it couldn't find the defined letter (84=T, 104=h, 101=e, etc...) and I think the index error is indexing the svgmobject at zero, but can't because there are no valid vmobject children.

The cause that feels simplest to me is that there is some strangeness in creating the SVG, but it's still a valid SVG file and should be read normally, but the SVG parser is fragile and so couldn't parse it correctly.

But that is still a pretty weird situation.

[GameDungeon]

Something also strange is the warning in #1149 is the same way, randomly triggering. The only difference is it happens locally.

[naveen521kk]

The only difference is it happens locally.

Does this happen locally? Can you give me your OS configuration?

[GameDungeon]

I run windows 10 pro