From a9b6bbe43ba983692b1b9ac79c58af3ddfc693c0 Mon Sep 17 00:00:00 2001 From: Kevin Kozan Date: Fri, 11 Oct 2019 11:52:11 -0500 Subject: [PATCH 1/5] MQE-1776: CICD System Documentation - Initial Draft --- docs/guides/cicd.md | 101 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 docs/guides/cicd.md diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md new file mode 100644 index 000000000..a37392955 --- /dev/null +++ b/docs/guides/cicd.md @@ -0,0 +1,101 @@ +# How to use MFTF in CICD + +If you want to integrate MFTF tests into your CICD pipeline, then it's best to start with the conceptual flow of the pipeline code. + +## Concept + +The overall workflow that tests should follow is thus: + +* Obtain Magento instance + install pre-requisites +* Generate tests for running + * Options for single/parallel running +* Delegate and run tests + gather test run artifacts + * Re-run options +* Generate Allure report from aggregate + +### Obtain Magento instance + +To start, we need a Magento instance to operate against for generation and execution. + +``` +$ git clone https://github.com/magento/magento2 +or +$ composer create-project --repository=https://repo.magento.com/ magento/project-community-edition magento2ce +``` + +For more information on installing magento see [Install Magento using Composer]. + +After installing the Magento instance, you need to set a couple of configurations to the magento instance: + +``` +$ bin/magento config:set general/locale/timezone America/Los_Angeles +$ bin/magento config:set admin/security/admin_account_sharing 1 +$ bin/magento config:set admin/security/use_form_key 0 +$ bin/magento config:set cms/wysiwyg/enabled disabled +``` + +These help set the state for the `default` state of the Magento instance. If you are wanting to change the default state of the application (and have merged into the tests sufficiently to account for it), this is the step in which you would do it. + +#### Install allure + +This will be required to generate the report after your test runs. See [Allure] for details. + + +### Generate tests + +#### Single execution + +Simply generate tests based on what you want to run: + +``` +$ vendor/bin/mftf generate:tests +``` + +This will generate all tests, and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt` + +#### Parallel execution + +To generate all tests for use in parallel nodes: + +``` +$ vendor/bin/mftf generate:tests --config parallel +``` + +This will generate a folder under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. + +### Delegate and run tests + +#### Single execution +If you are running on a single node, this step is simply to call: + +``` +$ vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt +``` + +#### Parallel execution +To run MFTF tests in parallel, you will need to clone the contents of the current node and duplicate them depending on how many nodes you have available for use. + +* Clone contents of current node as a baseline +* For each `groups/group#.txt` file: + * Set a node's contents to the baseline + * Run `vendor/bin/mftf run:manifest ` + * Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to master + +#### Rerun options +In either single or parallel execution, to re-run failed tests simply add a `run:failed` command after executing a manifest: + +``` +$ vendor/bin/mftf run:failed +``` + +### Generate Allure report + +In the master node, simply generate using your `` into a desired output path + +``` +$ allure generate -c -o +``` + + +[Install Magento using Composer]: https://devdocs.magento.com/guides/v2.3/install-gde/composer.html +[Allure]: https://docs.qameta.io/allure/ From 4e94af42edac1f22bb4ac96e91c3cd109151b393 Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Tue, 29 Oct 2019 11:09:02 -0500 Subject: [PATCH 2/5] Editorial pass --- docs/guides/cicd.md | 99 ++++++++++++++++++++++++--------------------- 1 file changed, 53 insertions(+), 46 deletions(-) diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md index a37392955..425f743bf 100644 --- a/docs/guides/cicd.md +++ b/docs/guides/cicd.md @@ -1,99 +1,106 @@ # How to use MFTF in CICD -If you want to integrate MFTF tests into your CICD pipeline, then it's best to start with the conceptual flow of the pipeline code. +To integrate MFTF tests into your CICD pipeline, then it is best to start with the conceptual flow of the pipeline code. ## Concept -The overall workflow that tests should follow is thus: +The overall workflow that tests should follow is: -* Obtain Magento instance + install pre-requisites -* Generate tests for running - * Options for single/parallel running -* Delegate and run tests + gather test run artifacts - * Re-run options -* Generate Allure report from aggregate +- Obtain a Magento instance + install pre-requisites. +- Generate the tests. + - Set options for single or parallel running. +- Delegate and run tests and gather test-run artifacts. + - Re-run options. +- Generate the Allure reports from results. -### Obtain Magento instance +## Obtain a Magento instance -To start, we need a Magento instance to operate against for generation and execution. +To start, we need a Magento instance to operate against for test generation and execution. +```bash +git clone https://github.com/magento/magento2 ``` -$ git clone https://github.com/magento/magento2 + or -$ composer create-project --repository=https://repo.magento.com/ magento/project-community-edition magento2ce + +```bash +composer create-project --repository=https://repo.magento.com/ magento/project-community-edition magento2ce ``` For more information on installing magento see [Install Magento using Composer]. After installing the Magento instance, you need to set a couple of configurations to the magento instance: -``` -$ bin/magento config:set general/locale/timezone America/Los_Angeles -$ bin/magento config:set admin/security/admin_account_sharing 1 -$ bin/magento config:set admin/security/use_form_key 0 -$ bin/magento config:set cms/wysiwyg/enabled disabled +```bash +bin/magento config:set general/locale/timezone America/Los_Angeles +bin/magento config:set admin/security/admin_account_sharing 1 +bin/magento config:set admin/security/use_form_key 0 +bin/magento config:set cms/wysiwyg/enabled disabled ``` -These help set the state for the `default` state of the Magento instance. If you are wanting to change the default state of the application (and have merged into the tests sufficiently to account for it), this is the step in which you would do it. +These help set the `default` state of the Magento instance. If you wish to change the default state of the application (and have updated your tests sufficiently to account for it), this is the step in which you would do it. -#### Install allure +## Install allure This will be required to generate the report after your test runs. See [Allure] for details. +## Generate tests -### Generate tests - -#### Single execution +### Single execution -Simply generate tests based on what you want to run: +Generate tests based on what you want to run: -``` -$ vendor/bin/mftf generate:tests +```bash +vendor/bin/mftf generate:tests ``` -This will generate all tests, and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt` +This will generate all tests and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt` -#### Parallel execution +### Parallel execution To generate all tests for use in parallel nodes: -``` -$ vendor/bin/mftf generate:tests --config parallel +```bash +vendor/bin/mftf generate:tests --config parallel ``` This will generate a folder under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. -### Delegate and run tests +## Delegate and run tests -#### Single execution -If you are running on a single node, this step is simply to call: +### Single execution +If you are running on a single node, call: + +```bash +vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt ``` -$ vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt -``` -#### Parallel execution -To run MFTF tests in parallel, you will need to clone the contents of the current node and duplicate them depending on how many nodes you have available for use. +### Parallel execution + +To run MFTF tests in parallel, clone the contents of the current node and duplicate them depending on how many nodes you have. + +- Clone contents of current node as a baseline. +- For each `groups/group#.txt` file: -* Clone contents of current node as a baseline -* For each `groups/group#.txt` file: - * Set a node's contents to the baseline - * Run `vendor/bin/mftf run:manifest ` - * Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to master + - Set a node's contents to the baseline. + - Run `vendor/bin/mftf run:manifest `. + - Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to master. #### Rerun options -In either single or parallel execution, to re-run failed tests simply add a `run:failed` command after executing a manifest: -``` -$ vendor/bin/mftf run:failed +In either single or parallel execution, to re-run failed tests simply add the `run:failed` command after executing a manifest: + +```bash +vendor/bin/mftf run:failed ``` ### Generate Allure report In the master node, simply generate using your `` into a desired output path -``` -$ allure generate -c -o +```bash +allure generate -c -o ``` From 66ff24dc36e85b6164a2679ff77c1a005211ee50 Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Tue, 29 Oct 2019 11:10:29 -0500 Subject: [PATCH 3/5] Fixed links --- docs/guides/cicd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md index 425f743bf..de54080ad 100644 --- a/docs/guides/cicd.md +++ b/docs/guides/cicd.md @@ -27,7 +27,7 @@ or composer create-project --repository=https://repo.magento.com/ magento/project-community-edition magento2ce ``` -For more information on installing magento see [Install Magento using Composer]. +For more information on installing magento see [Install Magento using Composer][]. After installing the Magento instance, you need to set a couple of configurations to the magento instance: @@ -42,7 +42,7 @@ These help set the `default` state of the Magento instance. If you wish to chang ## Install allure -This will be required to generate the report after your test runs. See [Allure] for details. +This will be required to generate the report after your test runs. See [Allure][] for details. ## Generate tests From 87f246a088aa7f1850c05a2b9cadd583b1457ec9 Mon Sep 17 00:00:00 2001 From: Soumya Unnikrishnan Date: Thu, 23 Jul 2020 16:41:05 -0500 Subject: [PATCH 4/5] MQE-2223: Part 7 Docs Review "MFTF Ease Of Use" PR #483 --- docs/guides/cicd.md | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md index de54080ad..689749319 100644 --- a/docs/guides/cicd.md +++ b/docs/guides/cicd.md @@ -1,6 +1,6 @@ # How to use MFTF in CICD -To integrate MFTF tests into your CICD pipeline, then it is best to start with the conceptual flow of the pipeline code. +To integrate MFTF tests into your CICD pipeline, it is best to start with the conceptual flow of the pipeline code. ## Concept @@ -40,6 +40,8 @@ bin/magento config:set cms/wysiwyg/enabled disabled These help set the `default` state of the Magento instance. If you wish to change the default state of the application (and have updated your tests sufficiently to account for it), this is the step in which you would do it. +If your magento instance has Two-Factor Authentication enabled, see [Configure 2FA][] to configure MFTF tests. + ## Install allure This will be required to generate the report after your test runs. See [Allure][] for details. @@ -54,7 +56,7 @@ Generate tests based on what you want to run: vendor/bin/mftf generate:tests ``` -This will generate all tests and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt` +This will generate all tests and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/_generated/testManifest.txt` ### Parallel execution @@ -64,7 +66,7 @@ To generate all tests for use in parallel nodes: vendor/bin/mftf generate:tests --config parallel ``` -This will generate a folder under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. +This will generate a folder under `dev/tests/acceptance/tests/functional/Magento/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. ## Delegate and run tests @@ -73,21 +75,24 @@ This will generate a folder under `dev/tests/acceptance/tests/functional/Magento If you are running on a single node, call: ```bash -vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt +vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/_generated/testManifest.txt ``` ### Parallel execution -To run MFTF tests in parallel, clone the contents of the current node and duplicate them depending on how many nodes you have. +You can optimize your pipeline by running tests in parallel across multiple nodes. + +Tests can be split up into roughly equal running groups using `--config parallel`. + +You don't want perform installation on each node again and build it. So, to save time, stash pre-made artifacts from earlier steps and un-stash on the nodes. -- Clone contents of current node as a baseline. -- For each `groups/group#.txt` file: +The groups can be then distributed on each of the nodes and run separately in an isolated environment. - - Set a node's contents to the baseline. - - Run `vendor/bin/mftf run:manifest `. - - Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to master. + - Stash artifacts from main node and un-stash on current node. + - Run `vendor/bin/mftf run:manifest ` on current node. + - Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to main node. -#### Rerun options +### Rerun options In either single or parallel execution, to re-run failed tests simply add the `run:failed` command after executing a manifest: @@ -97,12 +102,13 @@ vendor/bin/mftf run:failed ### Generate Allure report -In the master node, simply generate using your `` into a desired output path +In the main node, simply generate using your `` into a desired output path ```bash allure generate -c -o ``` -[Install Magento using Composer]: https://devdocs.magento.com/guides/v2.3/install-gde/composer.html +[Install Magento using Composer]: https://devdocs.magento.com/guides/v2.4/install-gde/composer.html +[Configure 2FA]: ../configure-2fa.md [Allure]: https://docs.qameta.io/allure/ From cf297c44529976818aa9dac9d46ac5b140159b28 Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Thu, 23 Jul 2020 17:10:55 -0500 Subject: [PATCH 5/5] Grammar and formatting --- docs/guides/cicd.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md index 689749319..80cf134d8 100644 --- a/docs/guides/cicd.md +++ b/docs/guides/cicd.md @@ -11,7 +11,7 @@ The overall workflow that tests should follow is: - Set options for single or parallel running. - Delegate and run tests and gather test-run artifacts. - Re-run options. -- Generate the Allure reports from results. +- Generate the Allure reports from the results. ## Obtain a Magento instance @@ -29,7 +29,7 @@ composer create-project --repository=https://repo.magento.com/ magento/project-c For more information on installing magento see [Install Magento using Composer][]. -After installing the Magento instance, you need to set a couple of configurations to the magento instance: +After installing the Magento instance, set a couple of configurations to the Magento instance: ```bash bin/magento config:set general/locale/timezone America/Los_Angeles @@ -38,13 +38,13 @@ bin/magento config:set admin/security/use_form_key 0 bin/magento config:set cms/wysiwyg/enabled disabled ``` -These help set the `default` state of the Magento instance. If you wish to change the default state of the application (and have updated your tests sufficiently to account for it), this is the step in which you would do it. +These set the default state of the Magento instance. If you wish to change the default state of the application (and have updated your tests sufficiently to account for it), this is the step to do it. If your magento instance has Two-Factor Authentication enabled, see [Configure 2FA][] to configure MFTF tests. -## Install allure +## Install Allure -This will be required to generate the report after your test runs. See [Allure][] for details. +This is required for generating the report after your test runs. See [Allure][] for details. ## Generate tests @@ -56,7 +56,7 @@ Generate tests based on what you want to run: vendor/bin/mftf generate:tests ``` -This will generate all tests and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/_generated/testManifest.txt` +This will generate all tests and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/_generated/testManifest.txt`. ### Parallel execution @@ -66,7 +66,7 @@ To generate all tests for use in parallel nodes: vendor/bin/mftf generate:tests --config parallel ``` -This will generate a folder under `dev/tests/acceptance/tests/functional/Magento/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. +This generates a folder under `dev/tests/acceptance/tests/functional/Magento/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. ## Delegate and run tests @@ -84,17 +84,17 @@ You can optimize your pipeline by running tests in parallel across multiple node Tests can be split up into roughly equal running groups using `--config parallel`. -You don't want perform installation on each node again and build it. So, to save time, stash pre-made artifacts from earlier steps and un-stash on the nodes. +You do not want to perform installations on each node again and build it. So, to save time, stash pre-made artifacts from earlier steps and un-stash on the nodes. The groups can be then distributed on each of the nodes and run separately in an isolated environment. - - Stash artifacts from main node and un-stash on current node. - - Run `vendor/bin/mftf run:manifest ` on current node. - - Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to main node. +- Stash artifacts from main node and un-stash on current node. +- Run `vendor/bin/mftf run:manifest ` on current node. +- Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to main node. ### Rerun options -In either single or parallel execution, to re-run failed tests simply add the `run:failed` command after executing a manifest: +In either single or parallel execution, to re-run failed tests, simply add the `run:failed` command after executing a manifest: ```bash vendor/bin/mftf run:failed @@ -102,7 +102,7 @@ vendor/bin/mftf run:failed ### Generate Allure report -In the main node, simply generate using your `` into a desired output path +In the main node, generate reports using your `` into a desired output path: ```bash allure generate -c -o