Make getting test results from fixtures easier

[pytestbot]

Originally reported by: Pedro Rodriguez (BitBucket: prodriguez320, GitHub: prodriguez320)

---

The current way is too complex for how simple this should be.

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/230

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

could you give an example of what you mean

"getting test results from fixtures" does not compute

fixtures do not have test results

tests have a result

tests use fixtures

[pytestbot]

Original comment by Pedro Rodriguez (BitBucket: prodriguez320, GitHub: prodriguez320):

---

There is no way (well there is, but it is complex) of reacting to a test result on finalizers (adding something or saying something if a test passes or fails).

Holger Krekel and I had this conversation: http://stackoverflow.com/questions/13364868/in-pytest-how-can-i-figure-out-if-a-test-failed-from-request/13375241

He told me to file an issue for it. Sorry if I was too vague.

[pfctdayelise]

As there is now an example in the docs about how to do this (see http://pytest.org/dev/example/simple.html#making-test-result-information-available-in-fixtures ) I will close this, if someone thinks it really deserves better support feel free to reopen.

[RonnyPfannschmidt]

There should be a less expensive way to track it

[flub]

The example should also probably use a hookwrapper now

[RonnyPfannschmidt]

true

[flub]

@RonnyPfannschmidt So looking at this I'm not sure what you have in mind for less expensive. Looking at this I am kind of tempted to just put the reports on the item by default. That's not hard, not that expensive I would think and makes using it a lot simpler. On the other hand making the report available by default gives using this some more official endorsement and that may or may not be a good thing.

[RonnyPfannschmidt]

reports take up memory, its already expensive when tracking what junitxml does on huge suites, it gets more horrific when everything remembers everything

[arosszay]

Hi guys,

[EDIT: First off, I really should thank Holger for being so kind AND taking the time and effort to provide us a way, albeit complex and indirect, but at least a possible way to check the status of a test. I have yet to read the example in detail, but I am assuming my comments here are still valid. I want to make sure that my appreciation for what you all have done to help the QA community out thus far is not missed by the frustration I lay out in my comment. Please keep this in mind as you read.]

I am a software automation qa engineer and I can tell you that having the ability in our automation framework code to determine if a previous running test has failed or has passed is CRITICAL!!! I am dumbfounded that this function isn't obvious as a critical function to have available in a super easy way as described by Pedro Rodriguez.

This is what we, automation engineers, want from pytest:

1. Write a fixture function
2. Within the teardown method of the fixture function we want a super easy way to check an attribute to determine if a test has failed or passed...something like (but doesn't have to be...after all I'm not a developer, If I was I'd have probably already make these changes in pytest myself) request.function.fail or previous_test_status or something simple like that.

What memory issues are you guys talking about? How hard is it to set an attribute somewhere indicating what the status of the last running test was? I don't understand why memory would even be a consideration here. What are you guys concerned about with this?

Lastly, I just want to say that the only reason I sound frustrated is because I am. Because I know that if pytest was a paid for product, this would never be an issue. And us poor QA guys like me, who DO NOT really know how to code, are stuck with open-source software that is just not fully up to par...and we've been waiting for years just to get a simple little function like the one we're talking about included in the tool we literally depend on 100% to do our work. (This issue was reported back in 2012!!)

If it's that much of a problem, why don't you guys just start making everyone pay for the product and actually make something awesome for the world of QA to use? Our jobs DO NOT have to suck because of you. Unfortunately, the reality is that our jobs do suck because of half-baked open-source software.

How much do you guys want to be paid to add this function? I bet you I can convince my company, a tiny bootstrapped startup even, to fork over 500 bucks to develop this simple function. Any takers on that? It would be nice to get this new functionality out the door in the next 3 weeks.

Thanks guys for all the hard work. I know it sucks working for free, but why did you choose to do that for such an important project that literally every single QA automation engineer using Selenium + Python depends on. Please, we really need pytest to work the best for us.

Thanks again for all the hard work thus far guys...and I'm serious about you guys thinking about how to start charging for this tool. It's important enough trust me...companies that are using it now will DEFINITELY pay for it. All of you will see money start coming in.

In any case, when do you think we can get this new function available in pytest?

[The-Compiler]

Please tone it down a bit. Blaming people for not doing the thing you want them to do in their free time isn't helping anybody. Is there a paid Python test runner which is better than pytest (since pytest is, apparently, not "up to par")? I certainly am not aware of one.

And no, it doesn't suck to work for free. I'm going to argue pytest, or Python, or a lot of other free software (as in freedom) would never be where they are if they were paid and proprietary. What sucks is reading pointless rants like this. In fact, this is a perfect way to ensure people are going to be more motivated working on something else instead.

That being said, Merlinux (the company of @hpk42) is offering paid feature development for pytest and then contracting pytest core developers. There are by far not enough companies doing so so someone could work full-time. Can you change that?

[nicoddemus]

@arosszay, I understand your frustration, but I agree completely with @The-Compiler's points. It is free software, people work on it on their on time, and we always try very hard to be accepting of external contributions so people in need of a certain feature can code it themselves. I understand the latter is not an option for you guys because you mention you're not coders, but as Florian mentioned Merlinux accepts paid feature development so perhaps that would be a solution to your specific problem (as a somewhat recent example of where that happen was the implementation of restarting slaves after a test crashing in pytest-xdist).

About keeping the reports in memory by default, it was a concern a few years ago: pytest was initially developed to run pypy's test suite which contains 20k+ tests, so keeping test results in memory was a concern back then. It might not be a problem anymore, but I particularly like pytest's current design of triggering hooks and keeping state to a minimum; this permits people to implement plugins which keep that information in memory if they want to.

Having said that and using the sample code provided, it seems easy to provide a plugin which would append that information automatically to all test items.