Proposal: disable "live logging" by default and write to terminal writer instead

[nicoddemus]

The Live Logs feature currently is enabled by default and requires the user to pass -s to see the logs.

I see two problems:

1. If the user is interested in the feature, having to pass -s feels clumsy because now all sort of outputs will come through, reducing the actual usefulness of the feature because it will be harder to see the actual logging messages.
2. To solve 1), I believe we will need to write to the TerminalWriter, which by consequence will show up for all users from that point on even if they are not interested in the feature because the default is WARNING.

My proposal:

1. Create a new property, live_logs, defaulting to False.
2. Write the "live logs" to the TerminalWriter, probably in color according to the level, yellow for WARNING and red for CRITICAL. This means users won't need to use -s to see live logs anymore.

This feature was originally implemented in eisensheng/pytest-catchlog#33 but was never released to PyPI, so I assume "live logs" has seen little use in the wild.

cc @Thisch

[twmr]

I would introduce a logcapture (or log-capture) property instead, similar to the existing capture property. In the course of changing the property(ies) of the logging plugin I would also rename the log-cli-* properties (maybe to log-capture-*).

If we use the TerminalWriter for outputting the logs, we could hopefully write a newline character before the first log message s.t. loglines do not appear on the same lines as the testname. This is how it looks when capturing is turned off:

...
collected 1 item                                                                                                                                                                                                              

test_logging_in_pytest.py::test test_logging_in_pytest.py   27 WARNING  Some warning
PASSED                                                                                                                                                                                  [100%]

[nicoddemus]

I would introduce a logcapture (or log-capture) property instead, similar to the existing capture property.

You mean a command-line option? I ask because the "capture" configuration that exists today is --capture=X on the command-line. If so I agree, I think we should add an option to disable logging capture completely as well (similar to -s or --capture=no).

Having said that, I still would like a separate option for "live logs": seems useful to be able to be able to capture logs and only having them appear on the captured output section if a test fails.

How about:

• --capture-log

  • yes (default): capture plugin is active; captured logs are displayed next to stderr and stdout sections if a test fails.
  • no: capture plugin is not active

• --live_logs:

  • no (default): nothing special.
  • yes: captured logs are displayed in the TerminalWriter as soon as they are emitted (in color and in a newline). For this to work --capture-log must be yes.

(Alternatively, all those could be ini options instead given that it is easy to set/override them on the command line using -o)

[twmr]

Yes I mean a command line --capture-log option, which should be orthogonal to the --capture option. I also think that the capture-log option can have two possible values: yes (default) and no. --capture-log=no should correspond to the live-log mode and --capture-log=yes should display captured logs if a test fails.

I don't understand why you need an additional --live-logs option, though. I also don't understand why your --live-logs option only works when capturing is enabled (--capture-log=yes)
Isn't --capture-log=no the same as -p no:logging in your proposal?

[nicoddemus]

(@Thisch sorry for the delay)

Isn't --capture-log=no the same as -p no:logging in your proposal?

At first glance it might seem like it is but disabling the plugin entirely means that the caplog fixture would not be available and tests which use it will break, that's why I feel we need a separate option (I should have written: "log capture is not active" instead of "log capture plugin is not active").

I don't understand why you need an additional --live-logs option, though.

When I wrote the previous proposal, I was considering that --capture-log=yes and --live-logs=yes was a valid use case: logs are displayed during the test run and also displayed when a test fails; this might be useful because at least you have the captured log output for each test (otherwise you can't know which test emitted a certain log message).

But I'm thinking that having --capture-log and --live-logs might be confusing.

Perhaps we can have three values for --capture-log then:

• on-failure (default): display captured logs on failure
• live: display captured logs as they are emitted, possibly with colors
• no: no log capturing at all (3.2 behavior), fixture still available

I'm also curious, do people use the live logs feature frequently?

Also would love to hear other opinions on this.

[twmr]

Perhaps we can have three values for --capture-log then:

• on-failure (default): display captured logs on failure
• live: display captured logs as they are emitted, possibly with colors
• no: no log capturing at all (3.2 behavior), fixture still available

I like this, but I'm not sure if on-failure should be the default, given that this might cause a high memory usage (see #2977 (comment))

[twmr]

Write the "live logs" to the TerminalWriter, probably in color according to the level, yellow for WARNING
and red for CRITICAL. This means users won't need to use -s to see live logs anymore.

The output by TerminalWriter is captured if -s is not used, or am I missing sth?

[nicoddemus]

I like this, but I'm not sure if on-failure should be the default, given that this might cause a high memory usage

This is happening because of we are setting the log-level of the capture-per-item to NO_SET, which causes it to capture everything... this seems like a bad idea in general.

The output by TerminalWriter is captured if -s is not used, or am I missing sth?

TerminalWriter is responsible for writing pytest's own output, so its output is not captured.

[twmr]

This is happening because of we are setting the log-level of the capture-per-item to NO_SET, which causes it to capture everything... this seems like a bad idea in general.

Ok, so if we set the default capture-log-level to a non-zero value, i.e., != NO_SET, on-failure can be set as the default for capture-log.

[twmr]

TerminalWriter is responsible for writing pytest's own output, so its output is not captured.

I've tried to experiment a bit with TerminalWriter in twmr@6e3ddf8 but I can't get the logs to be shown when --capture-log=live is used to run the tests. Maybe I'm doing sth wrong.

[nicoddemus]

Ouch sorry, I meant the TerminalReporter plugin! My bad.