Unit test failure

[prabhat00155]

self = <test_videoapi.TestVideoApi testMethod=test_frame_reading>

    @unittest.skipIf(av is None, "PyAV unavailable")
    def test_frame_reading(self):
        for test_video, config in test_videos.items():
            full_path = os.path.join(VIDEO_DIR, test_video)
    
            av_reader = av.open(full_path)
    
            if av_reader.streams.video:
                video_reader = VideoReader(full_path, "video")
                for av_frame in av_reader.decode(av_reader.streams.video[0]):
                    vr_frame = next(video_reader)
    
                    self.assertAlmostEqual(
                        float(av_frame.pts * av_frame.time_base),
                        vr_frame["pts"],
                        delta=0.1,
                    )
    
                    av_array = torch.tensor(av_frame.to_rgb().to_ndarray()).permute(
                        2, 0, 1
                    )
                    vr_array = vr_frame["data"]
                    mean_delta = torch.mean(
                        torch.abs(av_array.float() - vr_array.float())
                    )
                    # on average the difference is very small and caused
                    # by decoding (around 1%)
                    # TODO: asses empirically how to set this? atm it's 1%
                    # averaged over all frames
>                   self.assertTrue(mean_delta.item() < 2.5)
E                   AssertionError: False is not true

test/test_videoapi.py:97: AssertionError

https://app.circleci.com/pipelines/github/pytorch/vision/7915/workflows/82c134ae-52a5-4b83-8dbe-22d3f30ce3e5/jobs/561557

cc @bjuncek

[bjuncek]

This is probably something on my end, possibly related to your change on videoAPI.
I'll take a look at this tomorrow if that's ok?

[datumbox]

@bjuncek It seems to me that if you trigger the unit-tests to rerun on your original PR the test_videoapi.py tests will pass. This makes me thing it's not a dependency issue as I originally thought, but I could be wrong. Note that the test passes on my macOS on the latest master.

[prabhat00155]

@bjuncek I can't reproduce this locally, could be flaky.

(conda_py38) prabhatroy-mbp:vision prabhatroy$ pytest -vs test/test_videoapi.py 
==================================================================================================================== test session starts =====================================================================================================================
platform darwin -- Python 3.8.8, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /Users/prabhatroy/miniconda/envs/conda_py38/bin/python
cachedir: .pytest_cache
rootdir: /Users/prabhatroy/Documents/pytorch/repos/vision
collected 5 items                                                                                                                                                                                                                                            

test/test_videoapi.py::TestVideoApi::test_accurateseek_middle PASSED
test/test_videoapi.py::TestVideoApi::test_fate_suite Downloading http://fate.ffmpeg.org/fate-suite/sub/MovText_capability_tester.mp4 to /Users/prabhatroy/Documents/pytorch/repos/vision/test/assets/videos/MovText_capability_tester.mp4
124.1%
PASSED
test/test_videoapi.py::TestVideoApi::test_frame_reading PASSED
test/test_videoapi.py::TestVideoApi::test_metadata PASSED
test/test_videoapi.py::TestVideoApi::test_seek_start PASSED

===================================================================================================================== 5 passed in 8.94s ======================================================================================================================
(conda_py38) prabhatroy-mbp:vision prabhatroy$ 

[fmassa]

Another thing to look is if there are difference with PyAV / ffmpeg on those environments

[datumbox]

Another thing to look is if there are difference with PyAV / ffmpeg on those environments

@prabhat00155 This can be partially checked by restarting old runs of the test and see if it passes now. Example here. If it used to pass but now does not, it's probably due to external changes on dependencies. If it does pass, then that's the route to trace changes between the old commit and current master that contribute to its failing (changes on pinned version of software, modifications on the related methods or unit-test etc).

[prabhat00155]

Another thing to look is if there are difference with PyAV / ffmpeg on those environments

In the test, I see:

[conda] ffmpeg                    4.2                  h0a44026_0    pytorch
Requirement already satisfied: av in /Users/distiller/project/env/lib/python3.6/site-packages (from -r /Users/distiller/project/.circleci/unittest/linux/scripts/condaenv.hyuclnij.requirements.txt (line 4)) (8.0.3)

Locally, I have ffmpeg==4.2 and av==8.0.1.

av                        8.0.1            py38haad2d5e_0    conda-forge
ffmpeg                    4.2                  h0a44026_0    pytorch

Can't install 8.0.3 with ffmpeg==4.2:

Package lame conflicts for:
ffmpeg=4.2.0 -> lame[version='>=3.100,<3.101.0a0']
av=8.0.3 -> ffmpeg[version='>=4.3.1,<5.0a0'] -> lame[version='>=3.100,<3.101.0a0']

[datumbox]

You can try to force it outside of conda by installing it with pip.

I don't want to steer you towards the wrong direction, but given that the rerun worked, it might be worth checking for code changes between PR #2916 and master that affect the test.

[fmassa]

@datumbox

This can be partially checked by restarting old runs of the test and see if it passes now. Example here. If it used to pass but now does not, it's probably due to external changes on dependencies

This is partially true. If we have changed in the CircleCI config the pinning or particular versions of PyAV, then re-running old CI jobs can still pass even though it's related to a change of dependency, see #3341 for example, and in particular #3341 (comment)

[datumbox]

@fmassa I am aware. These are some directions that one can check here given that we can't reproduce it on our local setup.

Edit:

Comparing a recent failed run, it run seems to install the same versions for ffmpeg and av as when we rerun the old test.

Requirement already satisfied: av in /Users/distiller/project/env/lib/python3.8/site-packages (from -r /Users/distiller/project/.circleci/unittest/linux/scripts/condaenv._meap9h5.requirements.txt (line 4)) (8.0.3)

[conda] ffmpeg                    4.2                  h0a44026_0    pytorch

So probably these two libs are not the issue.