4.3 fails if there are HTML files in the source code

[nedbat]

Originally reported by amercader (Bitbucket: amercader, GitHub: amercader)

---

It seems like 4.3 is trying to parse HTML files that are part of the source code, while 4.2 didn't. We noticed this in our Travis builds:

https://travis-ci.org/frictionlessdata/goodtables.io/builds/192669485

This is running our tests with pytest with pytest --cov goodtablesio --cov-report term-missing but running coverage report -m locally also raises the exception:

#!traceback

coverage report -m
Traceback (most recent call last):
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/parser.py", line 364, in __init__
    self.code = compile_unicode(text, filename, "exec")
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/phystokens.py", line 286, in compile_unicode
    code = compile(source, filename, mode)
  File "/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/templates/error404.html", line 1
    {% extends "layout.html" %}
     ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/summary.py", line 46, in report
    analysis = self.coverage._analyze(fr)
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/control.py", line 899, in _analyze
    return Analysis(self.data, it)
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/results.py", line 19, in __init__
    self.statements = self.file_reporter.lines()
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/python.py", line 176, in lines
    return self.parser.statements
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/python.py", line 171, in parser
    self._parser.parse_source()
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/parser.py", line 238, in parse_source
    self._raw_parse()
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/parser.py", line 207, in _raw_parse
    self.raw_statements.update(self.byte_parser._find_statements())
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/parser.py", line 96, in byte_parser
    self._byte_parser = ByteParser(self.text, filename=self.filename)
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/parser.py", line 368, in __init__
    filename, synerr.msg, synerr.lineno
coverage.misc.NotPython: Couldn't parse '/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/templates/error404.html' as Python source: 'invalid syntax' at line 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/adria/dev/pyenvs/gt/bin/coverage", line 11, in <module>
    sys.exit(main())
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/cmdline.py", line 756, in main
    status = CoverageScript().command_line(argv)
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/cmdline.py", line 507, in command_line
    skip_covered=options.skip_covered, **report_args)
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/control.py", line 977, in report
    return reporter.report(morfs, outfile=file)
  File "/home/adria/dev/pyenvs/gt/lib/python3.5/site-packages/coverage/summary.py", line 58, in report
    except StopEverything:
TypeError: catching classes that do not inherit from BaseException is not allowed

4.2 works as expected

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/556

[nedbat]

Can you show me the output of "coverage debug data" when your "coverage report" fails?

[nedbat]

Original comment by amercader (Bitbucket: amercader, GitHub: amercader)

---

@nedbat here you go:

coverage debug data
-- data ------------------------------------------------------
path: /home/adria/dev/pyenvs/gt/src/goodtables.io/.coverage
has_arcs: False

72 files:
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/__init__.py: 5 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/app.py: 24 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/auth.py: 19 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/blueprints/__init__.py: 1 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/blueprints/api.py: 20 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/blueprints/site.py: 18 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/blueprints/user.py: 12 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/celery_app.py: 5 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/exceptions.py: 6 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/models/__init__.py: 2 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/models/base.py: 15 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/models/job.py: 28 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/models/user.py: 25 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/__init__.py: 4 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/blueprint.py: 25 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/models/__init__.py: 1 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/models/repo.py: 23 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/signals.py: 5 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/tasks/__init__.py: 1 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/tasks/jobconf.py: 12 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/tasks/repos.py: 7 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/utils/__init__.py: 1 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/utils/hook.py: 5 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/utils/repos.py: 2 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/utils/signature.py: 4 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/plugins/github/utils/status.py: 5 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/services.py: 2 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/settings.py: 24 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/signals.py: 9 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tasks/__init__.py: 1 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tasks/validate.py: 6 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/templates/error404.html: 14 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/templates/header.html: 14 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/templates/home.html: 19 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/templates/job.html: 36 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/templates/jobs.html: 22 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/templates/layout.html: 44 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/__init__.py: 1 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/blueprints/__init__.py: 1 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/blueprints/test_api.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/blueprints/test_site.py: 24 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/blueprints/test_user.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/conftest.py: 17 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/factories.py: 41 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/models/__init__.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/models/test_base.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/models/test_job.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/models/test_user.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/__init__.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/__init__.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/conftest.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/models/__init__.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/models/test_repo.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/tasks/__init__.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/tasks/test_jobconf.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/tasks/test_repos.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/test_blueprint.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/utils/__init__.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/utils/test_hook.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/utils/test_repos.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/utils/test_signature.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/plugins/github/utils/test_status.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/tasks/__init__.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/tasks/test_validate.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/test_factories.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/utils/__init__.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/utils/test_jobconf.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/tests/utils/test_jobtask.py: 0 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/utils/__init__.py: 1 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/utils/database.py: 7 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/utils/jobconf.py: 16 lines
/home/adria/dev/pyenvs/gt/src/goodtables.io/goodtablesio/utils/jobtask.py: 6 lines

[nedbat]

@amercader I see you have .html files in your data. If you do this again with 4.2, does "coverage debug data" show .html files?

[nedbat]

Original comment by amercader (Bitbucket: amercader, GitHub: amercader)

---

Yes, the output for coverage debug data with 4.2 and 4.3.3 is the same.

[nedbat]

Original comment by SGSabbage (Bitbucket: SGSabbage, GitHub: SGSabbage)

---

This has also become a problem for me today with jinja2 files in the source. I've checked, it works fine in 4.3.2 but not in 4.3.3, if that helps?

[nedbat]

@sgsabbage hmm, thanks for the data point, but that seems really unlikely. There's only one line of product code different between 4.3.2 and 4.3.3, and that should only affect either 2.6 or Jython. Can you verify? Perhaps there's data carrying over?

[nedbat]

Original comment by SGSabbage (Bitbucket: SGSabbage, GitHub: SGSabbage)

---

Pretty sure it's the change between 4.3.3 and 4.3.2 - if I just change the version of the library I alternate between passing and failure.

Full traceback (with some data redacted)

(hub) ubuntu@ubuntu-xenial:/vagrant$ pip install coverage==4.3.3 --upgrade
Collecting coverage==4.3.3
Installing collected packages: coverage
  Found existing installation: coverage 4.3.2
    Uninstalling coverage-4.3.2:
      Successfully uninstalled coverage-4.3.2
Successfully installed coverage-4.3.3
(hub) ubuntu@ubuntu-xenial:/vagrant$ py.test --cov -q
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/coverage/parser.py", line 238, in parse_source
INTERNALERROR>     self._raw_parse()
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/coverage/parser.py", line 139, in _raw_parse
INTERNALERROR>     tokgen = generate_tokens(self.text)
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/coverage/phystokens.py", line 148, in generate_tokens
INTERNALERROR>     self.last_tokens = list(tokenize.generate_tokens(readline))
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/tokenize.py", line 597, in _tokenize
INTERNALERROR>     raise TokenError("EOF in multi-line statement", (lnum, 0))
INTERNALERROR> tokenize.TokenError: ('EOF in multi-line statement', (59, 0))
INTERNALERROR>
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/coverage/summary.py", line 46, in report
INTERNALERROR>     analysis = self.coverage._analyze(fr)
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/coverage/control.py", line 899, in _analyze
INTERNALERROR>     return Analysis(self.data, it)
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/coverage/results.py", line 19, in __init__
INTERNALERROR>     self.statements = self.file_reporter.lines()
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/coverage/python.py", line 176, in lines
INTERNALERROR>     return self.parser.statements
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/coverage/python.py", line 171, in parser
INTERNALERROR>     self._parser.parse_source()
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/coverage/parser.py", line 246, in parse_source
INTERNALERROR>     self.filename, err.args[0], lineno
INTERNALERROR> coverage.misc.NotPython: Couldn't parse '/vagrant/hub/templates/form.jinja2' as Python source: 'EOF in multi-line statement' at line 59
INTERNALERROR>
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/_pytest/main.py", line 98, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/_pytest/main.py", line 133, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/pytest_cov/plugin.py", line 236, in pytest_runtestloop
INTERNALERROR>     self.cov_total = self.cov_controller.summary(self.cov_report)
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/pytest_cov/engine.py", line 95, in summary
INTERNALERROR>     total = self.cov.report(**options)
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/coverage/control.py", line 977, in report
INTERNALERROR>     return reporter.report(morfs, outfile=file)
INTERNALERROR>   File "/home/ubuntu/.virtualenvs/hub/lib/python3.5/site-packages/coverage/summary.py", line 58, in report
INTERNALERROR>     except StopEverything:
INTERNALERROR> TypeError: catching classes that do not inherit from BaseException is not allowed

535 passed in 18.67 seconds
(hub) ubuntu@ubuntu-xenial:/vagrant$ pip install coverage==4.3.2 --upgrade
Collecting coverage==4.3.2
Installing collected packages: coverage
  Found existing installation: coverage 4.3.3
    Uninstalling coverage-4.3.3:
      Successfully uninstalled coverage-4.3.3
Successfully installed coverage-4.3.2
(hub) ubuntu@ubuntu-xenial:/vagrant$ py.test --cov -q
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

----------- coverage: platform linux, python 3.5.2-final-0 -----------
Name            Stmts   Miss Branch BrPart  Cover   Missing
-----------------------------------------------------------
hub/store.py        4      3      2      0    17%   2-5
hub/tweens.py       8      1      2      1    80%   8, 7->8
-----------------------------------------------------------
TOTAL            2495      4    696      1    99%

29 files skipped due to complete coverage.

535 passed in 19.85 seconds
(hub) ubuntu@ubuntu-xenial:/vagrant$

Environment:

platform linux -- Python 3.5.2, pytest-3.0.5, py-1.4.32, pluggy-0.4.0
rootdir: /vagrant, inifile: tox.ini
plugins: celery-4.0.2, hypothesis-3.6.0, mock-1.5.0, cov-2.4.0

Happy to return any additional data required

[nedbat]

Ugh, I see.... One line that caused two releases to be bad... 4.3.4 coming up... :(

[nedbat]

Original comment by SGSabbage (Bitbucket: SGSabbage, GitHub: SGSabbage)

---

It happens! :) Thank you for the quick response and turn around!

[nedbat]

Fixed in 6020dcd2c7e6 (bb)

[nedbat]

Original comment by amercader (Bitbucket: amercader, GitHub: amercader)

---

Thank you very much for the quick fix @nedbat!

[nedbat]

4.3.4 is up on PyPI, give it a try.

[nedbat]

Original comment by amercader (Bitbucket: amercader, GitHub: amercader)

---

All good with 4.3.4, thanks! 👍