-
-
Notifications
You must be signed in to change notification settings - Fork 448
ternary ifs not taken into account by branch coverage #509
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
The Python settrace facility only offers feedback about lines executed. Supporting ternary-if and short-circuiting binary operators requires more details than Python offers. There are experiments about getting those details (http://nedbatchelder.com/blog/200804/wicked_hack_python_bytecode_tracing.html), but it's impractical to support in coverage.py for now. |
Original comment by Antony Lee (Bitbucket: anntzer, GitHub: anntzer) Perhaps it may be worth revisiting this now that Py3.7 optionally supports opcode tracing (if I understood https://bugs.python.org/issue31344 correctly)? |
Bytecode tracing is definitely an enabling technology, but there is still a lot to be done, including how to correlate tokens in the source with bytecodes at run-time. There's no information in the compiled code object about that. If someone wants to do some experiments, I'll be very interested in the results. |
this also applies to any sort of short circuiting operator eg: def true():
return True
def throw_error():
raise Exception
def error():
a = true()
b = throw_error()
return (a or b)
def no_error():
return (true() or throw_error()) # True
def test():
print(no_error()) # True
print(error()) # Exception()
if __name__ == '__main__':
test() of course when expanded to def no_error():
if x := true():
return x
return throw_error() it's caught by branch and line coverage |
This would be an amazing feature to implement, but has a number of challenges:
|
This thing just happened to me since I was curious about why I was not getting a lack of coverage.
|
Have been asking the same question today as well once I noticed that a file which contains some rather complex conditional logic has been reported to have 0 branches. It had the following structure: def resolve_status(data):
DEFAULT = -1
def priority1():
return 1 if something(data) else None
def priority2():
return 2 if something_else(data) else None
def priority3():
return 3 if data else None
return priority1() or priority2() or priority3() or DEFAULT I find this sort of style both concise and readable and tend to use the if condition():
return something
return None style yielded 16 branches. That's quite a significant difference compared with 0. Are there any plans to support this? |
Just fyi: it might be possible to implement this on 3.11 where we now expose column information for bytecode: https://docs.python.org/3.11/reference/datamodel.html#codeobject.co_positions |
See also: https://www.python.org/dev/peps/pep-0657/ for the PEP "Include Fine Grained Error Locations in Tracebacks" :-) |
Hey @nedbat . |
@saroad2 I haven't even looked at bytecode tracing. You could experiment with the information that sys.settrace offers for bytecode-level tracing. There's a lot to do to make it useful I think. |
https://docs.python.org/3.11/whatsnew/3.11.html#summary-release-highlights |
Either it will or pycoverage will be replaced by whatever new project comes along and does it. Sub-expression-level branching is simply the the correct way to measure coverage, and now it’s finally possible! |
I'm always interested in exploring how these things might work, even if it's just a discussion here. What would reporting look like? Are there other languages with this kind of instrumentation that we could learn from? |
It’s just regular branch coverage reporting. Instead of “lines of code that have been executed” it’s “nonbranching expressions that have been executed” or a similar concept. If you mean “how to visualize region coverage”, coverage.py already visualizes “partial hits” for As a first step, that could also be used for ternaries, The real deal however is to map code regions (or “spans”) that can be executed, as explained here here: Once that’s done, we can actually visualize which code regions are executed and which aren’t:
In a HTML report, hovering regions would then show how often they’ve been executed e.g. like this: |
I created a small proof of concept package that shows that it’s possible: https://github.com/flying-sheep/fine-coverage It visualizes coverage of (it would be trivial to enhance it to do more, but it’s only a proof of concept) |
Thanks, I didn't doubt that it was possible :) |
Ah I just wanted to be the first who does it 😄 and gains some insight how to do this The issue here is that the ideal data structure to represent this isn’t line based, and indeed having a line based data structure hinders implementation. I assume coverage.py uses a line based data structure to track coverage. If I’m right, it would probably need quite some refactoring to get it to a point where adding this is feasible. |
Hello @nedbat, do you have any updates on this? I think I could try to dive into concept from @flying-sheep and further refactoring, if you are interested in this feature. I have couple of projects at work using large conditionals and lambdas a lot, and I would love to see this work soon, since now coverage.py doesn't provide truthful measurements. |
Want to express my interest in this feature after we almost hit a dangerous bug, because the line was marked as tested when it wasn't actually tested (bc of a ternary operator). Maybe someone can suggest another coverage tool, that supports that? |
I'm afraid no one was strong enough to attempt to replace coverage.py successfully :) I stopped using ternaries entirely in my production code because of this. |
This is what we are discussing with the team right now. |
Until this ticket is resolved, are there existing tools to apply such bans automatically? But, I do also want to point back to where this round of comments started at #509 (comment) where interest was expressed in working on this. |
i want to point out that it's not just ternaries, it's also the
operates like this:
which means the exit branch when x is false might never be tested (and any side effect of evaluating y never happens!) it means all and and or conditions in our code are now linted out... devs get an ugly error |
Originally reported by Antony Lee (Bitbucket: anntzer, GitHub: anntzer)
coverage 4.1, python 3.5.2, arch linux
test.py
Getting branch coverage for this file shows partial coverage of the first
if cond:
, but not of the ternary (inline)if cond
, or of the short-circuiting binary operators.The text was updated successfully, but these errors were encountered: