Coverage misses with Python 3.10

[warsaw]

Describe the bug
I think there are still Python 3.10 compatibility problems, despite #1184 and #1106 and possibly other bugs. When testing flufl.lock against Python 3.10 (in CI and locally), I'm seeing line misses only in Python 3.10. These misses still appear to exist after testing locally with CPython 3.10 head and coveragepy head.

To Reproduce

1. git clone https://gitlab.com/warsaw/flufl.lock.git
2. cd flufl.lock
3. git switch qa
4. tox -e py39 -e py310

Expected behavior (Python 3.9)

---------- coverage: platform darwin, python 3.9.6-final-0 -----------
Name                      Stmts   Miss Branch BrPart  Cover   Missing
---------------------------------------------------------------------
flufl/lock/__init__.py        5      0      0      0   100%
flufl/lock/_lockfile.py     251      0     70      0   100%
---------------------------------------------------------------------
TOTAL                       256      0     70      0   100%
Coverage XML written to file coverage.xml

Actual behavior (Python 3.10)

---------- coverage: platform darwin, python 3.10.0-beta-4 -----------
Name                      Stmts   Miss Branch BrPart  Cover   Missing
---------------------------------------------------------------------
flufl/lock/__init__.py        5      0      0      0   100%
flufl/lock/_lockfile.py     253      0     70      3    99%   412->415, 418->420, 535->exit
---------------------------------------------------------------------
TOTAL                       258      0     70      3    99%
Coverage XML written to file coverage.xml

FAIL Required test coverage of 100.0% not reached. Total coverage: 99.09%

From local testing the lines are covered AFAICT, so there should be no functional difference between 3.9 and 3.10. The commonality is that these are conditionals inside try/excepts with raises in the true branch of the conditional.

[nedbat]

Thanks, this is reported against CPython here: https://bugs.python.org/issue44616

[warsaw]

Indeed, thanks @nedbat - that looks exactly like my case.

[warsaw]

The latest 3.10 head fixes this. Thanks!

[nedbat]

bpo 44616 is now fixed in the next 3.10 release.

[DerRidda]

@nedbat Is this actually fully resolved? I have just updated a project to Python 3.10(.8) using coverage through pytest-cov (4.0.0 using coverage 6.5.0) with no logic changes and suddenly get very weird coverage results with what looks like - to me - impossible misses. Such as:

If it is of note this code runs through the Apache beam SDK 2.43.0.
Needless to say, coverage was as expected under Python 3.9.

[nedbat]

@DerRidda Do you have a way for me to reproduce these failures?

[DerRidda]

I will try to extract an minimal example for this behavior.

[Nik-09]

Hi @nedbat, I'm encountering a similar issue where trivial lines are being marked as missing in the coverage report. However, when I add print statements for debugging, those lines are clearly being executed as they appear in the console output.

I'm attaching screenshots below to illustrate the difference — one from Python 3.9 and the other from 3.10 — showing changes in coverage without any modifications to the source files.

Python 3.10.16 and Code coverage for Python, version 7.8.0 with C extension:

Python 3.9.20 and Code coverage for Python, version 7.2.3 with C extension:

[nedbat]

@Nik-09 double-check that you are combining and reporting the correct data files. Can you give me a way to reproduce the issue?

[Nik-09]

Hi @nedbat ,

I confirm that I am combining and reporting the correct file.

Regarding the reproduction steps:
I'm using Apache Beam for a pipelined task in Python. I attempted to create a minimal example that replicates the behavior of the original code, but the coverage passes in the sample, so the issue doesn't reproduce there. To replicate the error, you may need to set up the Oppia codebase.

Here are the setup steps:

1. Set up the Oppia codebase with Python 3.9 link.
2. Install Python 3.10.16 via pyenv (pyenv install 3.10.16).
3. Update the .envrc file to use Python 3.10.
4. Delete the third-party Python libraries located in /oppia/oppia/third_party.
5. Pull the latest changes corresponding to Python 3.10 from the PR #22536
6. Start the server (python -m scripts.start)
7. Now close the server and run the backend tests with coverage enabled (python -m scripts.run_backend_tests --v --test_target=core.jobs.batch_jobs.exp_migration_jobs_test --generate_coverage_report)

I understand these steps might be time-consuming, but I'm happy to try out any suggestions you have on my system.

Thank you,
Nikhil

[nedbat]

@Nik-09 can you create a Dockerfile that does all of this? That would make it possible for us to have similar experiences.

[Nik-09]

Hi Ned,

I’ve created a Docker file for the project (pasted in the end). Please follow the steps below:

1. Create a directory and add the provided Dockerfile to that directory.
2. In the terminal, build the Docker image (this may take some time): docker build -t oppia-test .
3. Create and run a container from the image: docker run -it --rm oppia-test
4. Inside the container, run the following to install dependencies: python -m scripts.start. The server will exit automatically after setup.
5. Run backend tests and generate a coverage report:
   python -m scripts.run_backend_tests --v --test_target=core.jobs.batch_jobs.exp_migration_jobs_test --generate_coverage_report --skip-install

Observation:

After running the test, search for the keyword exp_migration in the generated coverage report.

The report shows line 524 is missing, which is a simple assignment operation "target_state_schema_version = feconf.CURRENT_STATE_SCHEMA_VERSION". But this line was covered in Python 3.9.20.

Let me know if you have any questions.

Thank you for your help.
Nikhil

---

# Base image with required tools and Python build dependencies
FROM ubuntu:24.04

# Non-interactive install
ENV DEBIAN_FRONTEND=noninteractive

# Install system dependencies
RUN apt-get update && apt-get install -y \
    curl \
    git \
    unzip \
    openjdk-11-jre \
    make \
    build-essential \
    libssl-dev \
    zlib1g-dev \
    libbz2-dev \
    libreadline-dev \
    libsqlite3-dev \
    wget \
    llvm \
    libncursesw5-dev \
    xz-utils \
    tk-dev \
    libxml2-dev \
    libxmlsec1-dev \
    libffi-dev \
    liblzma-dev \
    direnv \
    ca-certificates \
    gnupg \
    lsb-release \
    && rm -rf /var/lib/apt/lists/*

# Install Chrome
RUN curl -fsSL https://dl.google.com/linux/linux_signing_key.pub | gpg --dearmor -o /usr/share/keyrings/google-chrome.gpg \
    && echo "deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome.gpg] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list \
    && apt-get update && apt-get install -y google-chrome-stable

# Install pyenv
RUN curl https://pyenv.run | bash

# Set up pyenv environment variables
ENV PYENV_ROOT="/root/.pyenv"
ENV PATH="$PYENV_ROOT/bin:$PYENV_ROOT/shims:$PATH"

# Install Python 3.9.20 using pyenv
RUN bash -c "source ~/.bashrc && pyenv install 3.10.16 && pyenv global 3.10.16"

# Clone the Oppia repository from the custom branch
WORKDIR /root/opensource
RUN git clone -b python_migration https://github.com/Nik-09/oppia.git

WORKDIR /root/opensource/oppia

# Set up direnv hook
RUN echo 'eval "$(direnv hook bash)"' >> ~/.bashrc

# Set up .direnvrc
RUN echo 'use_python() {\n\
  local python_root=$(pyenv root)/versions/$1\n\
  load_prefix "$python_root"\n\
  if [[ -x "$python_root/bin/python" ]]; then\n\
    layout python "$python_root/bin/python"\n\
  else\n\
    echo "Error: $python_root/bin/python cannot be executed."\n\
    exit\n\
  fi\n\
}' > ~/.direnvrc

# Set up .envrc to activate Python 3.9.20 environment
RUN echo 'use python 3.10.16' > ../.envrc

# Allow direnv
RUN direnv allow ..

# Install Python dependencies
# RUN pip install pyyaml setuptools coverage configparser

# Pre-download third-party libraries (simulate initial setup)
RUN bash scripts/install_prerequisites.sh || true