MQE-1260: Create RERUN_COUNT field in Jenkins MFTF Parameters section same as MTF parameters section #251
Conversation
… same as MTF parameters section - Adding run:failed command
… same as MTF parameters section - Changing Suite Test Information to an array
… same as MTF parameters section - Added rerun file generation - Cleaned up test generation
… same as MTF parameters section - Removing unused use statement
There was a problem hiding this comment.
Changes work very well for test execution, but Allure implementation needs a fix:
- On build, suites are run by
run functional -g suiteNamefirst - Then, tests are retried via
mftf run:failed, which regenerates and runs tests viarun functionalwith no-gspecified
This is affecting allure paths to each tests thus:
- First Run:
Magento\FunctionalTestingFramework.functional\functionalSuite1.FailTest1 - Second Run:
Magento\FunctionalTestingFramework.functional.FailTest1 - Third+ Run:
Magento\FunctionalTestingFramework.functional.FailTest1
Above causes tests to be grouped incorrectly, where the first instance is not grouped with its retries.
… same as MTF parameters section - Restructured Codeception command to grab suite names properly
… same as MTF parameters section - Updated to prevent write correct tests to file after a rerun with multiple failures
| ); | ||
| $this->failedList = array_merge( | ||
| $this->failedList, | ||
| $this->readFailedTestFile(self::TESTS_FAILED_FILE) |
There was a problem hiding this comment.
Found while testing, if your last codecept run statement contains no failures, it actually deletes the failed file, so if a flaky test passes on the second try, $this->readFailedTestfile will actually fail to open the stream to the file (which was deleted).
… same as MTF parameters section - Prevented after from failing in the case of a passed test on rerun
| $output->write($buffer); | ||
| } | ||
| ); | ||
| if (realpath(self::TESTS_FAILED_FILE)) { |
There was a problem hiding this comment.
Change this to file_exists, realpath gives a false positive here depending on timing (somehow)
… same as MTF parameters section - Used shorter method for preventing test failure
|
@aljcalandra Rerun looks good to go from a local perspective. If you end up needing to make more changes to this due to jenkins implementation, I will happily re-review. |
… same as MTF parameters section - Protecting on missing testname and filepath
ACQE-2302: Unable to connect to selenium server message display
Adding a run:failed command to run the tests that failed on the last build
Contribution checklist