Document order of callback methods and extensions that run after test/container execution

[apodkutin]

Based on JUnit5 user-guide extensions execution order should correlate with extensions declaration order:

Extensions registered declaratively via @ExtendWith will be executed in the order in which they are declared in the source code. For example, the execution of tests in both MyFirstTests and MySecondTests will be extended by the FooExtension and BarExtension, in exactly that order.

But it does not.

Example: https://github.com/apodkutin/junit5-extensions-order-test/blob/master/src/test/java/junit5/extensions/order/JUnit5ExtensionsOrderTest.java

@ExtendWith({FirstExtension.class, SecondExtension.class})

Expected output

FirstExtension afterEach method call
SecondExtension afterEach method call

Actual output

SecondExtension afterEach method call
FirstExtension afterEach method call

Jupiter-api version is 5.3.1

<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.3.1</version>

Deliverables

• Document the lack of explicit ordering for all user-supplied callback methods (i.e., @BeforeEach, @AfterEach, @BeforeAll, @AfterAll) declared within a single test class/interface.

  • in Javadoc
  • in User Guide

• Document the lack of wrapping behavior for all user-supplied callback methods (e.g., @AfterAll, @AfterEach) declared within a single test class/interface.

  • in Javadoc
  • in User Guide

• Document wrapping behavior for all user-supplied callback methods (i.e., @BeforeEach, @AfterEach, @BeforeAll, @AfterAll) within test class/interface hierarchies.

  • in Javadoc
  • in User Guide

• Document wrapping behavior for all applicable extension APIs (e.g., TestExecutionExceptionHandler, AfterTestExecutionCallback, AfterEachCallback, AfterAllCallback, etc.).

  • in Javadoc
  • in User Guide

[smoyer64]

That looks correct to me ... it's assumed that AfterEach is often paired with BeforeEach. The order of post-test extension execution is inverted so that there's a symmetry around the test. See Extensions Execution Order in the Users Guide. This diagram doesn't show exactly what happens if more than one of each extension is present but the idea is that the extensions run after the test should be in the reverse order of their counterparts run before the test.

[apodkutin]

Thank for your comment @smoyer64, but I can't agree with you, because you're showing the diagram which is related to user-provided test and lifecycle methods and it looks like expected behavior for me as well.

But when I'm extending some test class with extension, and I'm declaring them in a specific order, I'm expecting that they will be invoked in the same order, I mean beforeEach from the first extension and after that beforeEach from the second extension. And I'm expecting that the same behavior should be for afterEach method based only on this statement from the guide "will be executed in the order in which they are declared in the source code."

I understand that for more than one extension order, execution order logic will be the same as for user-provided test and lifecycle methods execution order logic. But, do you think, that this statement is correct "will be executed in the order in which they are declared in the source code."? Maybe it should be changed to avoid misleading or maybe I've understood it wrong.

[marcphilipp]

@apodkutin Thanks for reporting this issue! @smoyer64 described our intention correctly: Extensions that run after a test are run in reverse order. However, as you noticed that's missing from the documentation and we should add it there.

[sbrannen]

Please note that the wrapping behavior for "after" extensions applies to callback methods as well (e.g., @AfterEach, @AfterAll).

I've updated the title of this issue accordingly.

[sbrannen]

I've added a Deliverables section as well.

[apodkutin]

@smoyer64 , @marcphilipp @sbrannen, thank you guys for the quick answers!
I'm really appreciate that you will add this information to the JavaDoc and to the user-guide as well.

But I still have only one question, do you think that this behavior and I'm talking only about callback methods from extensions should be like this? I have one example when I've faced it and why I actually raised this ticket.

Example:

We have EnvVariablesExtension which is changing environment variables in the beforeEach callback method and restoring them to original ones in the afterEach callback method.
I've tried to test EnvVariablesExtension with the second extension. Let's name it AcceptanceExtension and, of course, this extension is saving original environment variables in the beforeEach callback and asserting that environment variables which was fetched in afterEach callback method equals original environment variables.

Order of the extensions is

1. AcceptanceExtension
2. EnvVariablesExtension

As you already understood, beforeEach part works fine, but the second one afterEach part does not work, because AcceptanceExtension is called first and assertion failed.

My question is, why you've chosen this approach for execution order of callback extension's methods?
Why not just calling every callback extension's methods in that order which you've provided in the @ExtendWith({AcceptanceExtension.class, EnvVariablesExtension.class}) annotation?

AcceptanceExtension.beforeEach -> 
EnvVariablesExtension.beforeEach -> 
Test's body -> 
AcceptanceExtension.beforeEach -> 
EnvVariablesExtension.beforeEach

Thank you in advance.

[smoyer64]

I've never had a situation where I didn't want to invert the test setup when performing the tear-down. In my mind, the phrase

Let's name it AcceptanceExtension and, of course, this extension is saving original environment variables in the beforeEach callback and asserting that environment variables which was fetched in afterEach callback method equals original environment variables.

immediately raised red-flags. In my opinion, you really shouldn't be making assertions outside of test methods. In the case where I've written tests for an extension that implements both a beforeXxx() and afterXxx() extension point, I've individually tested that the beforeXxx() method functions as expected and that the afterXxx() method works as expected and then trusted that JUnit 5 will execute them at the appropriate times in the test lifecycle.

[apodkutin]

I agree with you @smoyer64 that it is a better way to test your custom extensions. But main thoughts were not about approach of testing your extensions, it was just an example to give us food for thought.

I'm just wondering maybe there are other cases where you need the same strict declared order of execution's callback methods calls, not only for beforeEach callback methods, but for afterEach callback methods as well.