Please allow TestExecutionListeners to fail the test [SPR-3960]

[spring-projects-issues]

Magnus Heino opened SPR-3960 and commented

SpringMethodRoadie.runAfters() calls all @After methods, then calls TestContextManager.afterTestMethod

TestContextManager.afterTestMethod calls all TestExecutionListeners. Any exception thrown in the TestExecutionListener is caught and logged by TestContextManager.afterTestMethod. This behaviour is good since it allows all TestExecutionListeners to execute even if one fails.

However, TestContextManager.afterTestMethod should be changed from void to boolean (or throw a custom exception). The returned value should indicate if any TestExecutionListener failed. If any TestExecutionListener failed, SpringMethodRoadie.runAfters() should then call SpringMethodRoadie.addFailure to fail the test.

I have extended TransactionalTestExecutionListener to look for @DataSet and @ExpectedDataSet annotations. @DataSet loads data using DBUnit before the test is executed, and @ExpectedDataSet compares the data in the database after test execution with the expected result. It is not currently possible to fail the test in the TestExecutionListener, with the current behaviour described above.

Please change this simple thing to make TestExecutionListeners even more powerful!

---

Affects: 2.1 M4

Issue Links:

• Introduce before/after test execution callbacks in the TestContext framework [SPR-4365] #9043 Introduce before/after test execution callbacks in the TestContext framework

[spring-projects-issues]

Magnus Heino commented

Please make SpringTestMethod and SpringMethodRoadie visible for external packages also, to make it possible to unittest custom testexecutionlisteners and annotations.

[spring-projects-issues]

Sam Brannen commented

Hi Magnus,

Thanks for the suggestion!

The Spring TestContext Framework now allows 'before' and 'after' methods of
TestExecutionListeners as well as @BeforeTransaction and @AfterTransaction
methods to fail a test.

I have extended TransactionalTestExecutionListener to look for @DataSet and
@ExpectedDataSet annotations. @DataSet loads data using DBUnit before the test
is executed, and @ExpectedDataSet compares the data in the database after test
execution with the expected result. It is not currently possible to fail the
test in the TestExecutionListener, with the current behaviour described above.

FYI: due to the "wrapped" nature of TestExecutionListener execution, you should
not need to extend TransactionalTestExecutionListener. Instead, you should just
develop your own stand-alone TestExecutionListener and configure it to run after
(i.e., wrapped 'within') the TransactionalTestExecutionListener.

For example, if you develop a CustomTestExecutionListener (CTEL) and configure
it after TransactionalTestExecutionListener (TTEL) in the list of TELs via
@TestExecutionListeners, CTEL's beforeTestMethod() will execute after TTEL's
beforeTestMethod(), and CTEL's afterTestMethod() will execute before TTEL's
afterTestMethod(). Thus CTEL's methods will execute within the transactional
context provided TTEL. Make sense?

Regards,

Sam

[spring-projects-issues]

Sam Brannen commented

Hi Magnus,

Please make SpringTestMethod and SpringMethodRoadie visible for external packages
also, to make it possible to unittest custom testexecutionlisteners and annotations.

SpringTestMethod and SpringMethodRoadie constitute specific details of the
internal implementation of the JUnit 4.4 support for the TestContext Framework.

Note that the TestContext Framework itself is completely agnostic of the
underlying testing framework (e.g., JUnit, TestNG, etc.). Thus, we currently do
not plan to open up SpringTestMethod and SpringMethodRoadie without good reason.

What is keeping you from testing your code via the public APIs for TestContext,
TestContextManager, and TestExecutionListener?

Thanks in advance for feedback.

Regards,

Sam

[spring-projects-issues]

Magnus Heino commented

afterTestMethod(). Thus CTEL's methods will execute within the transactional
context provided TTEL. Make sense

Yes.

I just tried my CTEL's with 2.5RC1, and I'm very happy about the current implementation.

Thanks,

/Magnus