diff --git a/docs/commands/codeception.md b/docs/commands/codeception.md index 6a96a89c7..8e6d76c11 100644 --- a/docs/commands/codeception.md +++ b/docs/commands/codeception.md @@ -1,7 +1,7 @@ # CLI commands: vendor/bin/codecept
-We do not recommend using Codeception commands directly as they can break the MFTF basic workflow. +We do not recommend using Codeception commands directly as they can break MFTF basic workflow. All the Codeception commands you need are wrapped using the [mftf tool][]. To run the Codeception testing framework commands directly, change your directory to the ``. @@ -36,7 +36,7 @@ vendor/bin/codecept run ```
-The following documentation corresponds to Codeception 2.3.8. +The following documentation corresponds to Codeception 4.1.4.
```bash @@ -47,36 +47,44 @@ Arguments: test test to be run Options: - -o, --override=OVERRIDE Override config values (multiple values allowed) - --config (-c) Use custom path for config - --report Show output in compact style - --html Generate html with results (default: "report.html") - --xml Generate JUnit XML Log (default: "report.xml") - --tap Generate Tap Log (default: "report.tap.log") - --json Generate Json Log (default: "report.json") - --colors Use colors in output - --no-colors Force no colors in output (useful to override config file) - --silent Only outputs suite names and final results - --steps Show steps in output - --debug (-d) Show debug and scenario output - --coverage Run with code coverage (default: "coverage.serialized") - --coverage-html Generate CodeCoverage HTML report in path (default: "coverage") - --coverage-xml Generate CodeCoverage XML report in file (default: "coverage.xml") - --coverage-text Generate CodeCoverage text report in file (default: "coverage.txt") - --coverage-phpunit Generate CodeCoverage PHPUnit report in file (default: "coverage-phpunit") - --no-exit Do not finish with exit code - --group (-g) Groups of tests to be executed (multiple values allowed) - --skip (-s) Skip selected suites (multiple values allowed) - --skip-group (-x) Skip selected groups (multiple values allowed) - --env Run tests in selected environments. (multiple values allowed, environments can be merged with ',') - --fail-fast (-f) Stop after first failure - --help (-h) Display this help message. - --quiet (-q) Do not output any message. - --verbose (-v|vv|vvv) Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug - --version (-V) Display this application version. - --ansi Force ANSI output. - --no-ansi Disable ANSI output. - --no-interaction (-n) Do not ask any interactive question. + -o, --override=OVERRIDE Override config values (multiple values allowed) + -e, --ext=EXT Run with extension enabled (multiple values allowed) + --report Show output in compact style + --html[=HTML] Generate html with results [default: "report.html"] + --xml[=XML] Generate JUnit XML Log [default: "report.xml"] + --phpunit-xml[=PHPUNIT-XML] Generate PhpUnit XML Log [default: "phpunit-report.xml"] + --tap[=TAP] Generate Tap Log [default: "report.tap.log"] + --json[=JSON] Generate Json Log [default: "report.json"] + --colors Use colors in output + --no-colors Force no colors in output (useful to override config file) + --silent Only outputs suite names and final results + --steps Show steps in output + -d, --debug Show debug and scenario output + --bootstrap[=BOOTSTRAP] Execute custom PHP script before running tests. Path can be absolute or relative to current working directory [default: false] + --no-redirect Do not redirect to Composer-installed version in vendor/codeception + --coverage[=COVERAGE] Run with code coverage + --coverage-html[=COVERAGE-HTML] Generate CodeCoverage HTML report in path + --coverage-xml[=COVERAGE-XML] Generate CodeCoverage XML report in file + --coverage-text[=COVERAGE-TEXT] Generate CodeCoverage text report in file + --coverage-crap4j[=COVERAGE-CRAP4J] Generate CodeCoverage report in Crap4J XML format + --coverage-phpunit[=COVERAGE-PHPUNIT] Generate CodeCoverage PHPUnit report in path + --no-exit Don't finish with exit code + -g, --group=GROUP Groups of tests to be executed (multiple values allowed) + -s, --skip=SKIP Skip selected suites (multiple values allowed) + -x, --skip-group=SKIP-GROUP Skip selected groups (multiple values allowed) + --env=ENV Run tests in selected environments. (multiple values allowed) + -f, --fail-fast Stop after first failure + --no-rebuild Do not rebuild actor classes on start + --seed=SEED Define random seed for shuffle setting + --no-artifacts Don't report about artifacts + -h, --help Display this help message + -q, --quiet Do not output any message + -V, --version Display this application version + --ansi Force ANSI output + --no-ansi Disable ANSI output + -n, --no-interaction Do not ask any interactive question + -c, --config[=CONFIG] Use custom path for config + -v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug ``` diff --git a/docs/commands/mftf.md b/docs/commands/mftf.md index bc2e5c0ec..b7981808c 100644 --- a/docs/commands/mftf.md +++ b/docs/commands/mftf.md @@ -39,13 +39,13 @@ vendor/bin/mftf generate:tests ### Generate tests by test name ```bash -vendor/bin/mftf generate:tests AdminLoginTest StorefrontPersistedCustomerLoginTest +vendor/bin/mftf generate:tests AdminLoginSuccessfulTest StorefrontPersistedCustomerLoginTest ``` ### Generate test by test and suite name ```bash -vendor/bin/mftf generate:tests LoginSuite:AdminLoginTest +vendor/bin/mftf generate:tests WYSIWYGDisabledSuite:AdminCMSPageCreatePageTest ``` ### Generate and run the tests for a specified group @@ -59,18 +59,18 @@ This command cleans up the previously generated tests; generates and runs tests ### Generate and run particular tests ```bash -vendor/bin/mftf run:test AdminLoginTest StorefrontPersistedCustomerLoginTest -r +vendor/bin/mftf run:test AdminLoginSuccessfulTest StorefrontPersistedCustomerLoginTest -r ``` -This command cleans up the previously generated tests; generates and runs the `LoginAsAdminTest` and `LoginAsCustomerTest` tests. +This command cleans up the previously generated tests; generates and runs the `AdminLoginSuccessfulTest` and `StorefrontPersistedCustomerLoginTest` tests. ### Generate and run particular test in a specific suite's context ```bash -vendor/bin/mftf run:test LoginSuite:AdminLoginTest -r +vendor/bin/mftf run:test WYSIWYGDisabledSuite:AdminCMSPageCreatePageTest -r ``` -This command cleans up previously generated tests; generates and run `AdminLoginTest` within the context of the `LoginSuite`. +This command cleans up previously generated tests; generates and run `AdminCMSPageCreatePageTest` within the context of the `WYSIWYGDisabledSuite`. ### Generate and run a testManifest.txt file @@ -362,7 +362,7 @@ vendor/bin/mftf run:manifest path/to/your/testManifest.txt Each line should contain either: one test path or one group (-g) reference. ``` -tests/functional/tests/MFTF/_generated/default/AdminLoginTestCest.php +tests/functional/tests/MFTF/_generated/default/AdminLoginSuccessfulTestCest.php -g PaypalTestSuite tests/functional/tests/MFTF/_generated/default/SomeOtherTestCest.php tests/functional/tests/MFTF/_generated/default/ThirdTestCest.php diff --git a/docs/merging.md b/docs/merging.md index fc187d576..d8436492e 100644 --- a/docs/merging.md +++ b/docs/merging.md @@ -1,6 +1,6 @@ # Merging -The MFTF allows you to merge test components defined in XML files, such as: +MFTF allows you to merge test components defined in XML files, such as: - [``][] - [``][] @@ -20,9 +20,9 @@ For example: - All tests with `` will be merged into one. -- All sections with `
` will be merged into one. +- All sections with `
` will be merged into one. - All data entities with `` will be merged into one. -- All action groups with `` will be merged into one. +- All action groups with `` will be merged into one. Although a file name does not influence merging, we recommend using the same file names in merging updates. This makes it easier to search later on. @@ -39,7 +39,7 @@ This makes it easier to search later on. ## Add a test You cannot add another [``][tests] using merging functionality. -To add a ``, create a new `*Test.xml` file or add a `` node to an existing `*Test.xml` file. +To add a ``, create a new `*Test.xml` file. ## Remove a test @@ -50,38 +50,35 @@ Learn more about running tests with different options using [`mftf`] or [`codece ### Example -Skip the `AdminLoginTest` test in the `.../Backend/Test/AdminLoginTest.xml` file while merging with the `.../Foo/Test/AdminLoginTest.xml` file: +Skip the `AdminLoginSuccessfulTest` test in the `.../Backend/Test/AdminLoginSuccessfulTest.xml` file while merging with the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml - + - + - - <description value="You should be able to log into the Magento Admin backend."/> + <title value="Admin should be able to log into the Magento Admin backend successfully"/> + <description value="Admin should be able to log into the Magento Admin backend successfully"/> <severity value="CRITICAL"/> <testCaseId value="MAGETWO-71572"/> <group value="example"/> <group value="login"/> </annotations> - <amOnPage url="{{AdminLoginPage.url}}" stepKey="amOnAdminLoginPage"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="{{_ENV.MAGENTO_ADMIN_USERNAME}}" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="{{_ENV.MAGENTO_ADMIN_PASSWORD}}" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickOnSignIn"/> - <closeAdminNotification stepKey="closeAdminNotification"/> - <seeInCurrentUrl url="{{AdminLoginPage.url}}" stepKey="seeAdminLoginUrl"/> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> </tests> ``` -Create the `.../Foo/Test/AdminLoginTest.xml` file: +Create the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="AdminLoginTest"> + <test name="AdminLoginSuccessfulTest"> <annotations> <skip> <issueId value="Issue#"/> @@ -91,29 +88,26 @@ Create the `.../Foo/Test/AdminLoginTest.xml` file: </tests> ``` -The `AdminLoginTest` result corresponds to: +The `AdminLoginSuccessfulTest` result corresponds to: ```xml -<test name="AdminLoginTest"> +<test name="AdminLoginSuccessfulTest"> <annotations> - <features value="Admin Login"/> + <features value="Backend"/> <stories value="Login on the Admin Login page"/> - <title value="You should be able to log into the Magento Admin backend."/> - <description value="You should be able to log into the Magento Admin backend."/> + <title value="Admin should be able to log into the Magento Admin backend successfully"/> + <description value="Admin should be able to log into the Magento Admin backend successfully"/> <severity value="CRITICAL"/> <testCaseId value="MAGETWO-71572"/> <group value="example"/> <group value="login"/> - <skip> - <issueId value="Issue#"/> - </skip> + <skip> + <issueId value="Issue#"/> + </skip> </annotations> - <amOnPage url="{{AdminLoginPage.url}}" stepKey="amOnAdminLoginPage"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="{{_ENV.MAGENTO_ADMIN_USERNAME}}" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="{{_ENV.MAGENTO_ADMIN_PASSWORD}}" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickOnSignIn"/> - <closeAdminNotification stepKey="closeAdminNotification"/> - <seeInCurrentUrl url="{{AdminLoginPage.url}}" stepKey="seeAdminLoginUrl"/> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> ``` @@ -126,159 +120,152 @@ See the previous examples. ### Add a test step -**Use case**: Add `checkOption` before `click` (`stepKey="clickLogin"`) and add `seeInCurrentUrl` after the `click` in the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) while merging with the `.../Foo/Test/LogInAsAdminTest.xml` file: +**Use case**: Add `AdminCheckOptionSalesActionGroup` before `AssertAdminSuccessLoginActionGroup` (`stepKey="assertLoggedIn"`) and add `AdminAssertSalesOnDashboardActionGroup` after the `AssertAdminSuccessLoginActionGroup` in the `AdminLoginSuccessfulTest` test (in the `.../Backend/Test/AdminLoginSuccessfulTest.xml` file) while merging with the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> + <test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> </tests> ``` -Create the `.../Foo/Test/LogInAsAdminTest.xml` file: +Create the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest"> - <checkOption selector="{{AdminLoginFormSection.rememberMe}}" stepKey="checkRememberMe" before="clickLogin"/> - <seeInCurrentUrl url="admin/admin/dashboard/" stepKey="seeAdminUrl" after="clickLogin"/> + <test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales" before="assertLoggedIn"/> + <actionGroup ref="AdminAssertSalesOnDashboardActionGroup" stepKey="assertSalesOnDashboard" after="assertLoggedIn"/> </test> </tests> ``` -The `LogInAsAdminTest` result corresponds to: +The `AdminLoginSuccessfulTest` result corresponds to: ```xml -<test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <checkOption selector="{{AdminLoginFormSection.rememberMe}}" stepKey="checkRememberMe"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <seeInCurrentUrl url="admin/admin/dashboard/" stepKey="seeAdminUrl"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> +<test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales" before="assertLoggedIn"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminAssertSalesOnDashboardActionGroup" stepKey="assertSalesOnDashboard" after="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> ``` ### Remove a test step -**Use case**: Remove `see` (`stepKey="seeLifetimeSales"`) from the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) while merging with the `.../Foo/Test/LogInAsAdminTest.xml` file: +**Use case**: Remove `AdminCheckOptionSalesActionGroup` (`stepKey="checkOptionSales"`) from the `LogInAsAdminTest` test (in the `.../Backend/Test/AdminLoginSuccessfulTest.xml` file) while merging with the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> + <test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> </tests> ``` -Create the `.../Foo/Test/LogInAsAdminTest.xml` file: +Create the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest"> - <remove keyForRemoval="seeLifetimeSales"/> + <test name="AdminLoginSuccessfulTest"> + <remove keyForRemoval="checkOptionSales"/> </test> </tests> ``` -The `LogInAsAdminTest` result corresponds to: +The `AdminLoginSuccessfulTest` result corresponds to: ```xml -<test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> +<test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> ``` ### Update a test step -**Use case**: Change selector in `fillField` (`stepKey="fillPassword"`) of the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) while merging with the `.../Foo/Test/LogInAsAdminTest.xml` file: +**Use case**: Change argument in `AdminCheckOptionSalesActionGroup` (`stepKey="checkOptionSales"`) of the `AdminLoginSuccessfulTest` test (in the `.../Backend/Test/AdminLoginSuccessfulTest.xml` file) while merging with the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> + <test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"> + <argument name="salesOption" value="{{AdminSalesData.lifeTimeSales}}"/> + </actionGroup> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> </tests> ``` -Create the `.../Foo/Test/LogInAsAdminTest.xml` file: +Create the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml -<tests ...> - <test name="LogInAsAdminTest"> - <fillField selector="{{AdminLoginFormSection.wrong-password}}" userInput="password" stepKey="fillPassword"/> - </test> -</tests> +<test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"> + <argument name="salesOption" value="{{AdminSalesData.annualSales}}"/> + </actionGroup> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> +</test> ``` -The `LogInAsAdminTest` result corresponds to: +The `AdminLoginSuccessfulTest` result corresponds to: ```xml -<test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.wrong-password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> +<test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"> + <argument name="salesOption" value="{{AdminSalesData.annualSales}}"/> + </actionGroup> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> ``` ### Add several test steps {#insert-after} -**Use case**: Add several actions after the `click` (`stepKey="clickLogin"`) in the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) while merging with the `.../Foo/Test/LogInAsAdminTest.xml` file: +**Use case**: Add several action groups after the `AdminLoginActionGroup` (`stepKey="loginAsAdmin"`) in the `AdminLoginSuccessfulTest` test (in the `.../Backend/Test/AdminLoginSuccessfulTest.xml` file) while merging with the `.../Foo/Test/LogInAsAdminTest.xml` file: ```xml -<tests ...> - <test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> - </test> -</tests> +<test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> +</test> ``` -Create the `.../Foo/Test/LogInAsAdminTest.xml` file: +Create the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest" insertAfter="clickLogin"> - <checkOption selector="{{AdminLoginFormSection.rememberMe}}" stepKey="checkRememberMe"/> - <seeInCurrentUrl url="admin/admin/dashboard/" stepKey="seeAdminUrl"/> + <test name="AdminLoginSuccessfulTest" insertAfter="loginAsAdmin"> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> </tests> ``` -The `LogInAsAdminTest` result corresponds to: +The `AdminLoginSuccessfulTest` result corresponds to: ```xml -<test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <checkOption selector="{{AdminLoginFormSection.rememberMe}}" stepKey="checkRememberMe"/> - <seeInCurrentUrl url="admin/admin/dashboard/" stepKey="seeAdminUrl"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> -</test> +<tests ...> + <test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> + </test> +</tests> ``` ## Merge action groups @@ -293,7 +280,7 @@ The controls change drastically in the B2B version, so it was abstracted to an a > Action group for selecting `customerGroup` in the `Cart Price Rules` section: ```xml -<actionGroup name="selectNotLoggedInCustomerGroup"> +<actionGroup name="SelectNotLoggedInCustomerGroupActionGroup"> <selectOption selector="{{AdminCartPriceRulesFormSection.customerGroups}}" userInput="NOT LOGGED IN" stepKey="selectCustomerGroup"/> </actionGroup> ``` @@ -302,7 +289,7 @@ The controls change drastically in the B2B version, so it was abstracted to an a ```xml <!-- name matches --> -<actionGroup name="selectNotLoggedInCustomerGroup"> +<actionGroup name="SelectNotLoggedInCustomerGroupActionGroup"> <!-- removes the original action --> <remove keyForRemoval="selectCustomerGroup"/> <!-- adds in sequence of actions to be performed instead--> @@ -323,62 +310,62 @@ To merge [pages][page], the `page name` must be the same as in the base module. ### Add a section **Use case**: The `FooBackend` module extends the `Backend` module and requires use of two new sections that must be covered with tests. -Add `BaseBackendSection` and `AnotherBackendSection` to the `BaseBackendPage` (`.../Backend/Page/BaseBackendPage.xml` file): +Add `AdminBaseBackendSection` and `AdminAnotherBackendSection` to the `AdminBaseBackendPage` (`.../Backend/Page/AdminBaseBackendPage.xml` file): ```xml <pages ...> - <page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="BaseBackendSection"/> - <section name="AnotherBackendSection"/> + <page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> + <section name="AdminBaseBackendSection"/> + <section name="AdminAnotherBackendSection"/> </page> </pages> ``` -Create the `.../FooBackend/Page/BaseBackendPage.xml` file: +Create the `.../FooBackend/Page/AdminBaseBackendPage.xml` file: ```xml <pages ...> - <page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="NewExtensionSection"/> + <page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> + <section name="AdminNewExtensionSection"/> </page> </pages> ``` -The `BaseBackendPage` result corresponds to: +The `AdminBaseBackendPage` result corresponds to: ```xml -<page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> +<page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="BaseBackendSection"/> - <section name="AnotherBackendSection"/> - <section name="NewExtensionSection"/> + <section name="AdminBaseBackendSection"/> + <section name="AdminAnotherBackendSection"/> + <section name="AdminNewExtensionSection"/> </page> ``` ### Remove a section -**Use case**: The `FooBackend` module extends the `Backend` module and requires deletion of the `AnotherBackendSection` section (the `.../Backend/Page/BaseBackendPage.xml` file): +**Use case**: The `FooBackend` module extends the `Backend` module and requires deletion of the `AdminAnotherBackendSection` section (the `.../Backend/Page/AdminBaseBackendPage.xml` file): ```xml -<page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="BaseBackendSection"/> - <section name="AnotherBackendSection"/> +<page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> + <section name="AdminBaseBackendSection"/> + <section name="AdminAnotherBackendSection"/> </page> ``` -Create the `.../FooBackend/Page/BaseBackendPage.xml` file: +Create the `.../FooBackend/Page/AdminBaseBackendPage.xml` file: ```xml -<page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="AnotherBackendSection" remove="true"/> +<page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> + <section name="AdminAnotherBackendSection" remove="true"/> </page> ``` -The `BaseBackendPage` result corresponds to: +The `AdminBaseBackendPage` result corresponds to: ```xml -<page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="BaseBackendSection"/> +<page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> + <section name="AdminBaseBackendSection"/> </page> ``` diff --git a/docs/metadata.md b/docs/metadata.md index 3f6032b57..414f4bc04 100644 --- a/docs/metadata.md +++ b/docs/metadata.md @@ -1,6 +1,6 @@ # Metadata -In this topic we talk about handling entities that you need in your tests (such as categories, products, wish lists, and similar) using the MFTF. +In this topic we talk about handling entities that you need in your tests (such as categories, products, wish lists, and similar) using MFTF. Using data handling actions like [`createData`], [`deleteData`], [`updateData`], and [`getData`], you are able to create, delete, update, and read entities for your tests. The framework enables you to send HTTP requests with these statically defined data entities: @@ -29,7 +29,7 @@ Each [operation] includes: - The type of body content encoding in [contentType]. - The body of the request represented as a tree of objects, arrays, and fields. -When a test step requires handling the specified data entity, the MFTF performs the following steps: +When a test step requires handling the specified data entity, MFTF performs the following steps: - Reads input data (`<data/>`) and the type (the `type` attribute) of the specified [entity]. - Searches the metadata operation for the `dataType` that matches the entity's `type`. For example, `<entity type="product">` matches `<operation dataType="product"`. @@ -94,8 +94,8 @@ Example: ### Sending a REST API request -The MFTF allows you to handle basic CRUD operations with an object using [Magento REST API][api reference] requests. -To convert a request to the MFTF format, wrap the corresponding REST API request into XML tags according to the [Reference documentation][reference]. +MFTF allows you to handle basic CRUD operations with an object using [Magento REST API][api reference] requests. +To convert a request to MFTF format, wrap the corresponding REST API request into XML tags according to the [Reference documentation][reference]. - GET is used for retrieving data from objects. - POST is used for creating new objects. @@ -123,7 +123,7 @@ Let's see what happens when you create a category: <createData entity="_defaultCategory" stepKey="createPreReqCategory"/> ``` -The MFTF searches in the _Data_ directory an entity with `<entity name="_defaultCategory">` and reads `type` of the entity. +MFTF searches in the _Data_ directory an entity with `<entity name="_defaultCategory">` and reads `type` of the entity. If there are more than one entity with the same name, all of the entities are merged. _Catalog/Data/CategoryData.xml_: @@ -136,8 +136,8 @@ _Catalog/Data/CategoryData.xml_: </entity> ``` -Here, `type` is equal to `"category"`, which instructs the MFTF to search an operation with `dataType="category"`. -Since the action is __to create__ a category, the MFTF will also search for operation with `type="create"` in _Metadata_ for `dataType="category"`. +Here, `type` is equal to `"category"`, which instructs MFTF to search an operation with `dataType="category"`. +Since the action is __to create__ a category, MFTF will also search for operation with `type="create"` in _Metadata_ for `dataType="category"`. _Catalog/Metadata/CategoryMeta.xml_: @@ -236,26 +236,26 @@ The corresponding test step is: <createData entity="guestCart" stepKey="createGuestCart"/> ``` -The MFTF searches in the _Data_ directory an entity with `<entity name="guestCart">` and reads `type`. +MFTF searches in the _Data_ directory an entity with `<entity name="GuestCart">` and reads `type`. _Quote/Data/GuestCartData.xml_: ```xml -<entity name="guestCart" type="guestCart"> +<entity name="GuestCart" type="GuestCart"> </entity> ``` -`type="guestCart"` points to the operation with `dataType=guestCart"` and `type="create"` in the _Metadata_ directory. +`type="guestCart"` points to the operation with `dataType=GuestCart"` and `type="create"` in the _Metadata_ directory. _Catalog/Data/CategoryData.xml_: ```xml -<operation name="CreateGuestCart" dataType="guestCart" type="create" auth="anonymous" url="/V1/guest-carts" method="POST"> +<operation name="CreateGuestCart" dataType="GuestCart" type="create" auth="anonymous" url="/V1/guest-carts" method="POST"> <contentType>application/json</contentType> </operation> ``` -As a result, the MFTF sends an unauthorized POST request with an empty body to the `https://example.com/rest/V1/guest-carts` and stores the single string response that the endpoint returns. +As a result, MFTF sends an unauthorized POST request with an empty body to the `https://example.com/rest/V1/guest-carts` and stores the single string response that the endpoint returns. ### Handling a REST API response {#rest-response} @@ -267,7 +267,7 @@ Let's see how to handle data after you created a category with custom attributes <createData entity="customizedCategory" stepKey="createPreReqCategory"/> ``` -The MFTF receives the corresponding JSON response and enables you to reference its data using a variable of format: +MFTF receives the corresponding JSON response and enables you to reference its data using a variable of format: **$** _stepKey_ **.** _JsonKey_ **$** @@ -377,7 +377,7 @@ The operation enables you to assign the following form fields: ### Create an object in storefront {#create-object-as-customerFormKey} -The MFTF uses the `CreateWishlist` operation to create a wish list on storefront: +MFTF uses the `CreateWishlist` operation to create a wish list on storefront: Source file is _Wishlist/Metadata/WishlistMeta.xml_ @@ -558,7 +558,7 @@ Example: <!-- LINK DEFINITIONS --> [actions]: test/actions.md -[api reference]: https://devdocs.magento.com/guides/v2.3/get-started/bk-get-started-api.html +[api reference]: https://devdocs.magento.com/guides/v2.4/get-started/bk-get-started-api.html [application/x-www-form-urlencoded]: https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.1 {:target="_blank"} [catalogCategoryRepositoryV1 image]: img/catalogCategoryRepository-operations.png @@ -573,7 +573,7 @@ Example: [getData]: test/actions.md#getdata [HTML forms]: https://www.w3.org/TR/html401/interact/forms.html {:target="\_blank"} -[oauth]: https://devdocs.magento.com/guides/v2.3/get-started/authentication/gs-authentication-oauth.html +[oauth]: https://devdocs.magento.com/guides/v2.4/get-started/authentication/gs-authentication-oauth.html {:target="\_blank"} [operation]: #operation-tag [reference]: #reference diff --git a/docs/page.md b/docs/page.md index 41afd274f..fd03cfd58 100644 --- a/docs/page.md +++ b/docs/page.md @@ -1,6 +1,6 @@ # Page structure -The MFTF uses a modified concept of [PageObjects], which models the testing areas of your page as objects within the code. +MFTF uses a modified concept of [PageObjects], which models the testing areas of your page as objects within the code. This reduces occurrences of duplicated code and enables you to fix things quickly, in one place, when things change. You define the contents of a page, for reference in a [`<test>`], at both the [`<page>`] and [`<section>`] level. @@ -61,7 +61,7 @@ Example (_Catalog/Page/AdminCategoryPage.xml_ file): <?xml version="1.0" encoding="UTF-8"?> <pages xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" - xsi:noNamespaceSchemaLocation="../../../../../..dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Page/etc/PageObject.xsd"> + xsi:noNamespaceSchemaLocation="urn:magento:mftf:Page/etc/PageObject.xsd"> <page name="AdminCategoryPage" url="catalog/category/" module="Magento_Catalog" area="admin"> <section name="AdminCategorySidebarActionSection"/> <section name="AdminCategorySidebarTreeSection"/> @@ -97,7 +97,7 @@ Example (_Catalog/Page/StorefrontCategoryPage.xml_ file): <?xml version="1.0" encoding="UTF-8"?> <pages xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" - xsi:noNamespaceSchemaLocation="../../../../../..dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Page/etc/PageObject.xsd"> + xsi:noNamespaceSchemaLocation="urn:magento:mftf:Page/etc/PageObject.xsd"> <page name="StorefrontCategoryPage" url="/{{var1}}.html" module="Magento_Catalog" parameterized="true" area="storefront"> <section name="StorefrontCategoryMainSection"/> </page> @@ -107,15 +107,15 @@ Example (_Catalog/Page/StorefrontCategoryPage.xml_ file): This example shows the page with the name `StorefrontCategoryPage`. It will be merged with the other `StorefrontCategoryPage` pages from other modules. -The following is an example of a call in test: +The following is an example of a call in a test: ```xml -<amOnPage url="{{StorefrontCategoryPage.url($$createPreReqCategory.name$$)}}" stepKey="navigateToCategoryPage"/> +<amOnPage url="{{StorefrontCategoryPage.url($createPreReqCategory.name$)}}" stepKey="navigateToCategoryPage"/> ``` The `StorefrontCategoryPage` page is declared as parameterized, where the `url` contains a `{{var1}}` parameter. -The corresponding web page is generated by the Magento Catalog module and is called by the `baseUrl`+`/$$createPreReqCategory.name$$.html` URl. +The corresponding web page is generated by the Magento Catalog module and is called by the `baseUrl`+`/$createPreReqCategory.name$.html` URl. `{{var1}}` is substituted with the `name` of the previously created category in the `createPreReqCategory` action. @@ -132,7 +132,7 @@ There are several XML elements that are used in `<page>` in the MFTF. ### pages {#pages-tag} `<pages>` are elements that point to the corresponding XML Schema location. -It contains one or more `<page>` elements. +It contains only one `<page>` element. ### page {#page-tag} diff --git a/docs/reporting.md b/docs/reporting.md index 59a8617ba..629c3cfd8 100644 --- a/docs/reporting.md +++ b/docs/reporting.md @@ -57,42 +57,49 @@ The general information can be useful for MFTF contributors, but can be ignored Let's consider the general part of the following test execution report: ```terminal -==== Redirecting to Composer-installed version in vendor/codeception ==== -Codeception PHP Testing Framework v2.3.9 -Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors. +Generate Tests Command Run +Codeception PHP Testing Framework v4.1.4 +Powered by PHPUnit 9.1.3 by Sebastian Bergmann and contributors. +Running with seed: Magento\FunctionalTestingFramework.functional Tests (2) ------------------------ -Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Helper\Acceptance, \Magento\FunctionalTestingFramework\Helper\MagentoFakerData, \Magento\FunctionalTestingFramework\Module\MagentoRestDriver, PhpBrowser, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTes +Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, \Magento\FunctionalTestingFramework\Module\MagentoActionProxies, Asserts, \Magento\FunctionalTestingFramework\Helper\HelperContainer ``` -After the test generation command (mentioned in the previous section), MFTF delegates control to the `vendor/codeception` tool, which is the `Codeception PHP Testing Framework` of version `2.3.9` that uses `PHPUnit` of version `6.5.13`. +After the test generation command (mentioned in the previous section), MFTF delegates control to the `vendor/codeception` tool, which is the `Codeception PHP Testing Framework` of version `4.1.4` that uses `PHPUnit` of version `9.1.3`. The tool runs `2 Tests` using the configuration defined in the `functional` suite under the `Magento\FunctionalTestingFramework` namespace. The corresponding configuration file is `acceptance/tests/functional.suite.yml`. -It enables `Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Helper\Acceptance, \Magento\FunctionalTestingFramework\Helper\MagentoFakerData, \Magento\FunctionalTestingFramework\Module\MagentoRestDriver, PhpBrowser, \Magento\FunctionalTestingFramework\Module\MagentoSequence, ...` +It enables `Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, \Magento\FunctionalTestingFramework\Module\MagentoActionProxies, Asserts, \Magento\FunctionalTestingFramework\Helper\HelperContainer,..` #### Passed tests The next chunk of the log reports about test execution of the first test: ```terminal -AdminLoginTestCest: Admin login test -Signature: Magento\AcceptanceTest\_default\Backend\AdminLoginTestCest:AdminLoginTest -Test: tests/functional/Magento/FunctionalTest/_generated/default/AdminLoginTestCest.php:AdminLoginTest +AdminLoginSuccessfulTestCest: Admin login successful test +Signature: Magento\AcceptanceTest\_default\Backend\AdminLoginSuccessfulTestCest:AdminLoginSuccessfulTest +Test: tests/functional/Magento/_generated/default/AdminLoginSuccessfulTestCest.php:AdminLoginSuccessfulTest Scenario -- -I am on page "/admin/admin" -I fill field "#username","admin" -I fill field "#login","123123q" -I click ".actions .action-primary" -I wait for page load 30 -I close admin notification -I see in current url "/admin/admin" -PASSED +[loginAsAdmin] AdminLoginActionGroup + [navigateToAdmin] am on page "/admin/admin" + [fillUsername] fill field "#username","admin" + [fillPassword] fill field "#login","123123q" + [clickLogin] click ".actions .action-primary" + [clickLoginWaitForPageLoad] wait for page load 30 + [clickDontAllowButtonIfVisible] conditional click ".modal-popup .action-secondary",".modal-popup .action-secondary",true + [closeAdminNotification] close admin notification +[assertLoggedIn] AssertAdminSuccessLoginActionGroup + [waitForAdminAccountTextVisible] wait for element visible ".page-header .admin-user-account-text",60 + [assertAdminAccountTextElement] see element ".page-header .admin-user-account-text" +[logoutFromAdmin] AdminLogoutActionGroup + [amOnLogoutPage] am on page "/admin/admin/auth/logout/" + PASSED ``` -The running test is `AdminLoginTestCest`, which is `Admin login test` (this text is generated from the test name but with the `Cest` part excluded). -Its test signature is `Magento\AcceptanceTest\_default\Backend\AdminLoginTestCest:AdminLoginTest` that matches a `className:methodName` format using namespaces. -A path to the corresponding `Test` is `tests/functional/Magento/FunctionalTest/_generated/default/AdminLoginTestCest.php:AdminLoginTest` (relative to the `acceptance/` directory). +The running test is `AdminLoginSuccessfulTestCest`, which is `Admin login successful test` (this text is generated from the test name but with the `Cest` part excluded). +Its test signature is `Magento\AcceptanceTest\_default\Backend\AdminLoginSuccessfulTestCest:AdminLoginSuccessfulTest` that matches a `className:methodName` format using namespaces. +A path to the corresponding `Test` is `tests/functional/Magento/_generated/default/AdminLoginSuccessfulTestCest.php:AdminLoginSuccessfulTest` (relative to the `acceptance/` directory). `Scenario` lists the tests steps as they run during test execution, ending with the successful test verdict `PASSED`. It means that all test steps were processed as expected. @@ -104,51 +111,54 @@ The second test fails with the following report: ```terminal AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test Signature: Magento\AcceptanceTest\_default\Backend\AdminMenuNavigationWithSecretKeysTestCest:AdminMenuNavigationWithSecretKeysTest -Test: tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest +Test: tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest Scenario -- -I magento cli "config:set admin/security/use_form_key 1" +[enableUrlSecretKeys] magento cli "config:set admin/security/use_form_key 1",60 Value was saved. -I magento cli "cache:clean config full_page" + +[cleanInvalidatedCaches1] magento cli "cache:clean config full_page",60 Cleaned cache types: config full_page -I am on page "/admin/admin" -I wait for page load -I fill field "#username","admin" -I fill field "#login","123123q" -I click ".actions .action-primary" -I wait for page load 30 -I close admin notification -I click "//li[@id='menu-magento-backend-stores']" -I wait for loading mask to disappear -I click "#nav li[data-ui-id='menu-magento-config-system-config']" -I wait for page load -I see current url matches "~\/admin\/system_config\/~" -I see "#something" -I save screenshot -FAIL - -I magento cli "config:set admin/security/use_form_key 0" + +[loginAsAdmin] AdminLoginActionGroup + [navigateToAdmin] am on page "/admin/admin" + [fillUsername] fill field "#username","admin" + [fillPassword] fill field "#login","123123q" + [clickLogin] click ".actions .action-primary" + [clickLoginWaitForPageLoad] wait for page load 30 + [clickDontAllowButtonIfVisible] conditional click ".modal-popup .action-secondary",".modal-popup .action-secondary",true + [closeAdminNotification] close admin notification +[clickStoresMenuOption1] click "#menu-magento-backend-stores" +[waitForStoresMenu1] wait for loading mask to disappear +[clickStoresConfigurationMenuOption1] click "#nav li[data-ui-id='menu-magento-config-system-config']" +[waitForConfigurationPageLoad1] wait for page load 60 +[seeCurrentUrlMatchesConfigPath1] see current url matches "~\/admin\/system_config\/~" +[clickCatalogMenuOption] click "#something" +[saveScreenshot] save screenshot +[disableUrlSecretKeys] magento cli "config:set admin/security/use_form_key 0",60 Value was saved. -I magento cli "cache:clean config full_page" + +[cleanInvalidatedCaches2] magento cli "cache:clean config full_page",60 Cleaned cache types: config full_page -I am on page "/admin/admin/auth/logout/" + +[logout] AdminLogoutActionGroup + [amOnPage] am on page "/admin/admin/auth/logout/" + FAIL -------------------------------------------------------------------------------- ``` The general test details and scenario has the same format as in the Passed test. -The interesting part starts near the `FAIL` line. ```terminal -I see "#something" -I save screenshot -FAIL +[clickCatalogMenuOption] click "#something" +[saveScreenshot] save screenshot ``` When a test step fails, MFTF always saves a screenshot of the web page with the failing state immediately after the failure occurs. -`I save screenshot` follows the failing test step `I see "#something"` in our case. +`[saveScreenshot] save screenshot` follows the failing test step `[clickCatalogMenuOption] click "#something"` in our case. A screenshot of the fail goes at the `acceptance/tests/_output` directory in both PNG and HTML formats: @@ -165,27 +175,19 @@ The file name encodes: - with the `AdminMenuNavigationWithSecretKeysTest` test name - and execution status `fail` -Actions after `FAIL` are run as a part of the [`after`][] hook of the test. +Actions after `saveScreenshot` are run as a part of the [`after`][] hook of the test. ### Test result report After MFTF completed test execution, it generates a general report about test results along with detailed information about each fail. ```terminal --------------------------------------------------------------------------------- -DEPRECATION: Calling the "Symfony\Component\BrowserKit\Client::getInternalResponse()" method before the "request()" one is deprecated since Symfony 4.1 and will throw an exception in 5.0. /Users/.../magento2ce/vendor/symfony/browser-kit/Client.php:208 - -Time: 52.43 seconds, Memory: 16.00MB +Time: 02:07.534, Memory: 150.50 MB There was 1 failure: --------- ``` - -First you see warnings and deprecations. -The `DEPRECATION` here is thrown by an MFTF dependency (Symfony) that is out of the scope for test writers and should be considered by MFTF contributors. -If you encounter this type of reporting, [report an issue][]. - -Then, MFTF reports that the test run took 52.43 seconds using 16 MB of system RAM. +MFTF reports that the test run took 02:07.534 using 150.50 MB of system RAM. And, finally, that there was `1 failure`. Next, the report provides details about the test failure. @@ -193,75 +195,38 @@ Next, the report provides details about the test failure. ```terminal --------- 1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test -Test tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest -Step See "#something" -Fail Failed asserting that on page /admin/admin/system_config/index/key/678b7ba922c.../ ---> DASHBOARD -SALES -CATALOG -CUSTOMERS -MARKETING -CONTENT -REPORTS -STORES -SYSTEM -FIND PARTNERS & EXTENSIONS -Configuration -admin -1 -Store View: Default Config -What is this? -Save Config -Country Options -State Options -Locale Options -Store Information -Store Name -Store Phone Number -Store Hours of Operation -Countr -[Content too long to display. See complete response in '/Users/dmytroshevtsov/Projects/vagrant/vagrant-magento/magento2ce/dev/tests/acceptance/tests/_output/' directory] ---> contains "#something". + Test tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest + Step Click "#something" + Fail CSS or XPath element with '#something' was not found. Scenario Steps: -23. $I->amOnPage("/admin/admin/auth/logout/") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:54 -22. // Cleaned cache types: + 27. // Exiting Action Group [logout] AdminLogoutActionGroup + 26. $I->amOnPage("/admin/admin/auth/logout/") at tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:55 + 25. // Entering Action Group [logout] AdminLogoutActionGroup + 24. // Cleaned cache types: config full_page -21. $I->magentoCLI("cache:clean config full_page") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:52 -20. // Value was saved. -19. $I->magentoCLI("config:set admin/security/use_form_key 0") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:50 -18. $I->saveScreenshot() at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:63 + + 23. $I->magentoCLI("cache:clean config full_page",60) at tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:52 + 22. // Value was saved. ``` - `1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test` - the failed Codeception test is *AdminMenuNavigationWithSecretKeysTestCest*. It references to the PHP class that implemented the failed test. -- `Test tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest` - the test is implemented in the *AdminMenuNavigationWithSecretKeysTest* test method of the *tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php* file under `<magento root>/dev/tests/acceptance/`. +- `Test tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest` - the test is implemented in the *AdminMenuNavigationWithSecretKeysTest* test method of the *tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php* file under `<magento root>/dev/tests/acceptance/`. It matches the corresponding test defined in XML that is *AdminMenuNavigationWithSecretKeysTest* defined in `<test name="AdminMenuNavigationWithSecretKeysTest">...</test>` -- `Step See "#something"` - the failing test step is the *see* action with the *#something* selector. It would correspond the `<see selector="#something" ... />` test step in the XML defined tests. - -- `Fail Failed asserting that on page /admin/admin/system_config/index/key/678b7ba922c.../` - the fail occurred on the web page `<MAGENTO_BASE_URL>/admin/admin/system_config/index/key/678b7ba922c.../`. - -```terminal ---> ... -[Content too long to display. See complete response in '/../../magento2/dev/tests/acceptance/tests/_output/' directory] ---> contains "#something". -``` - -The web page is too long to be reported in the CLI, and it is stored at *'/../../magento2/dev/tests/acceptance/tests/_output/'*. -Search the web page by test name *AdminMenuNavigationWithSecretKeysTest*. -The failing test assertion is that the web page contains *contains* a CSS locator *#something*. +- `Step Click "#something"` - the failing test step is the *click* action with the *#something* selector. It would correspond the `<click selector="#something" ... />` test step in the XML defined tests. Finally, the report finishes with fairly self-descriptive lines. ```terminal FAILURES! -Tests: 2, Assertions: 3, Failures: 1. +Tests: 2, Assertions: 2, Failures: 1. ``` -MFTF encountered failures due to the last test run, that included *2* tests with *3* assertions. +MFTF encountered failures due to the last test run, that included *2* tests with *2* assertions. *1* assertion fails. ## Allure @@ -324,8 +289,7 @@ And if you run the `open` command with no arguments while you are in the same di allure open ``` -Allure would attempt to open a generated report at the `magento2/allure-report/` directory.' -%} +Allure would attempt to open a generated report at the `magento2/allure-report/` directory. To clean up existing reports before generation (for example after getting new results), use the `--clean` flag: