Big problem with v6.16.2 (MUST REVERT)

[alekdavis]

I think we got a big problem with the v6.16.2 update.

Here is the scenario:

Testing a CRUD sequence. POST successfully creates an object, but some assertion fails (maybe some property is not quite correct, or the Location header is missing, something not essential, but causing the test to fail). There is a number of tests depending on the POST: GET, PATCH, GET after PATCH, and finally DELETE. Most of them have a @ref to the original POST. But since the response variable gets cleared, for every test with the POST test dependency, POST will be called again and again. This is really bad.

Try this sample:

### Test A
# @name a
POST https://httpbin.org/anything

?? status == 200
?? body json exists

### Test B
# @name b
# @ref a
{{
  exports.$cancel = !(aResponse.statusCode == 200);
}}
GET https://httpbin.org/anything

?? status == 200

### Test C
# @name c
# @ref a
{{
  exports.$cancel = !(aResponse.statusCode == 200);
}}
PATCH https://httpbin.org/anything

?? status == 200

### Test Z
# @name z
# @ref a
{{
  exports.$cancel = !(aResponse.statusCode == 200);
}}
DELETE https://httpbin.org/anything

?? status == 200

You see how a failed assertion in referenced tests essentially turns @ref into @forcedRef?

Not only that. Now there is absolutely no way to reference a response from a request with a failed assertion. The request could've been good and an assertion could be minor, but since assertion failed, any test referencing it will be automatically skipped. Meaning, any failed assertion after a successful POST will not allow to DELETE the object.

We definitely do not want that. And I do not think there is a way to fix it.

I think the original behavior made more sense. A request with @ref or @forceRef dependency should be skipped if the call fails and the response variable does not get set, but it should not be automatically skipped if the call completes but an assertion fails (or multiple assertion fail). If an assertion fails, we at least can detect the condition programmatically and use the $cancel variable to submit or cancel a request, but if we automatically skip it on a failed assertion, there is no way around.

Does this make sense?

[AnWeber]

What is your expected behaviour for this use case?

• the POST response variables are always set, but the depending requests are only executed if no test fails
• negative test results does not stop depending requests from executing
• ...?

[alekdavis]

These sound like behaviors (what the system does) vs use cases (what I want to do), so if you're asking about the proposed system behaviors (and there seems to be two), here is my take.

the POST response variables are always set, but the depending requests are only executed if no test fails

This solves the problem of turning @ref into @forcedRef, but unless I'm missing something, the response variables will not be usable for anything (other than not calling the referenced request repeatedly), since the dependent requests will be skipped. I still need to use @ref to get the response variable, right? But if the test with @ref is automatically skipped, the response variable in the @ref is not accessible, right? I suppose, I can save the object ID from the POST in a global (in the POST post-request script), or do some other hack, but it would be a hassle. So, unless I use some globals or whatever, I will not be able to delete an object successfully created by POST (if a POST assertion failed) by referencing the POST's response variable. No good, IMHO.

negative test results does not stop depending requests from executing

I think this makes most sense and that's how the system should behave. The depending request can always check a condition and set $cancel programmatically. I suppose, the only case when you may want to automatically skip a request is when the depending request causes an exception (like, when not being able to make a call at all due to an exception), but it would be fine if it were not automatically skipped even in this case because the pre-request script should be always checking whether to run the test after a failed dependency anyway, so automatic skipping would not make anything easier. If it's easier to automatically skip a request on a dependency exception (but not on failed assertion!), it's fine, if it's easier not to skip, it's fine too.

Another thing I want to mention is that if a referenced request fails due to an exception (not due to an assertion), it should not be repeatedly called from all @ref instances pointing to it in a single test run. Meaning, under no circumstances should @ref turn into @forcedRef. For example, if I have tests A, B, C, ... Z, where A calls POST to create an object and B, C, ...Z use @ref A, and A fails due to an exception, in a single test run (whether I execute all A-Z test, or some test with an A dependency), A should be attempted only once. If A fails (due to network issue, or whatever) and the A response variable does not get set, there is no point to repeatedly call A again and again for every dependency from B to Z.

What do you think?

[AnWeber]

I updated the behaviour:

• negative test results from reference does not stop dependent request
• errored reference is not executed multiple times if it is referenced multiple times

[alekdavis]

@AnWeber You are the best!