Defer html generation until after testrun is complete

[noamkremen]

Hi,
I currently have a scenario where a report row needs to be augmented with links to log files collected via a setup/teardown fixture. I'm trying to achieve this via the pytest_runtest_makereport hook but as far as I understand, this is not currently possible since:
A) append_failed/passed only run for report.when=='call', thus no report rows are generated for teardown/setup.
B) Keeping a reference to the call phase report (during pytest_runtest_makereport) and then adding extras to it during the teardown phase report is useless since the relevant row html has already been generated by _appendrow.

Do I understand the limitations correctly? If so, will you be open to considering a PR that allows addition of rows for setup/teardown phases and defers html generation until the testrun has been completed?

[davehunt]

A failure in setup/teardown will be reported as an Error, and any extras associated with the report at this time should be included. This can mean that a single test has two rows, for example if the 'call' passes but the 'teardown' fails. Note that if 'setup' fails then 'call' will not be executed.

It is true that If a 'call' fails then the extras associated with the associated 'setup' will not be included in the row, and also true that the 'teardown' extras will not be included as they have not yet been executed.

I'm open to a pull request that holds off building the report row until the 'teardown' call, and includes extras associated with each step, but I suspect this will be difficult to achieve without introducing conflicts and ambiguity in the report.

I think I'd be more interested in a way to always report setup/teardown in the report as separate rows grouped by the associated test. These could perhaps be collapsed or hidden by default for anyone wanting a more succinct report. This is very similar to #131.

[mayankchugh]

While we are on the subject of holding off the report generation until test completion, I can see that the report is created at desired path only after teardown fixture has run.

Is there a way to keep on generating the report side-by-side? My use case is to send the HTML report in email in teardown. However, my email function in teardown is not able to find the report as HTML is saved only after teardown is run