Better diffs in tests

[hauntsaninja]

It's annoying that one line change causes everything else to show up as a diff. Just use difflib instead. I also highlight the changed lines. We can't use FancyFormatter because it doesn't work well with pytest.

[ikonst]

Add a test to mypy/test/meta/?

[JukkaL]

Can you show some examples?

One the goals of the existing diffing approach is to make it easy to copy-paste the actual output from running a test to a test case description with only minor editing required. I'd like to preserve that. I often leave the output of a test case empty, run the test case, and after validating that the output looks good copy the output into the test case.

[ikonst]

@JukkaL did you try pytest --update-data recently? :)

[JukkaL]

did you try pytest --update-data recently?

It's pretty handy, but I only tend to use it for bulk updates. If I'm working on a single test case I tend to use copy-paste a lot, in part because I can never remember the name of the flag without looking it up. :)

[hauntsaninja]

Before:

After:

[hauntsaninja]

Okay I changed it to also mention --update-data (which is awesome)

[ikonst]

Speaking of --update-data: A wart most people would quickly discover is that (a) we default to parallelized tests and (b) those are incompatible with --update-data (since a file might be sharded between two runners, and you can't update files in parallel while relying on absolute line offsets).

I couldn't find a way to trigger -n0 when --update-data is used.

We can rewrite --update-data to work by keeping offsets relative to testcase, which would require it to at least rudimentarily parse the file during the update phase.

Or keep the wart (maybe then say "Update the test output using -n0 --update-data").

[hauntsaninja]

I'm merging, since I keep wanting this for other PRs I'm working on. I will add a unit test for the diff logic + any other feedback in follow up. If it helps I use similar code elsewhere + this logic never changes test results, so am pretty confident