[Discussion] Core bundled tests - access to test credentials

[fooman]

Magento ships with core bundled extensions. These extensions also ship with their own MFTF tests. Conceivably a lot of them will require test credentials to run when interacting with outside services (vertex, klarna come to mind but I am sure most have an outside service component).

It is currently not clear how this should be handled.

• Are there common credentials that everyone can use?
• Is every one required to sign up on their own just so that the full test suite can pass?
• Something else?

Alternatively shipping tests which can't be run or fail is also not an ideal road to travel on.

[okolesnyk]

Hi @fooman

Currently we have .credentials file which contains all creds to 3rd party systems. In our case this is mostly payments and shipment integrations.
This file initially located in MFTF https://github.com/magento/magento2-functional-testing-framework/blob/develop/etc/config/.credentials.example
but when you run vendor/bin/mftf build:project we copy this file to dev/tests/acceptance/.credentials.
On out CICD we fill this file with our own creadentials needed for tests.
If you want to run our 3rd party integration tests you need to fill this file with own sandbox credentials.

_CREDS - is an object which gives you access to .creadentials
In tests you can access this values via this notation {{_CREDS.payment_authorizenet_login}}

Are there common credentials that everyone can use?

No, we do not share any Sandbox credentials because our builds rely on them and also tests rely on how sandbox was configured. If someone will make adjustments to it tests will possibly fail and block delivery teams.

Is every one required to sign up on their own just so that the full test suite can pass?

Yes. To ignore some tests which requires 3rd party integration we going to introduce separate test suite called 3rdPartyIntegrationTests or something similar, so that you will be able simply identify them.
Currently we only have test suite called PaypalTestSuite.

Something else?

I have several open questions on my mind:

• How to share configuration of our sandboxes, so that everyone can simply create own sandbox and configure it fast ?
• Should we introduce separate config file which will be located in module, where everyone can provide keys for needed credentials ? something like this:

<creadentials>
    <field path="carriers/fedex/account">{{PUT_YOUR_ACCOUNT_HERE}}</field>
    <field path="carriers/fedex/password">{{PUT_YOUR_PASSWORD_HERE}}</field>
</credentials>

Values in such files will always be empty, but it will allow everyone understand what should be provided to run your 3rdParty integration test.

[fooman]

I like the idea of 3rdPartyIntegrationTests. Ideally we get to a stage where we can

1. install Magento via composer
2. install Mftf
3. install tests
4. then run the tests and it passes.

Please note that these steps do not include disabling/deleting present but tests that don't run without further steps (ie not everyone will use authorizenet).

Especially as an extension developer I am looking to be able to run Magento with the out of the box configuration (ie all modules that are enabled by default).

Should we introduce separate config file which will be located in module, where everyone can provide keys for needed credentials ?

I think keeping the current mechanism with the .credentials.xml file works as we we would already get an error when {{_CREDS.payment_authorizenet_login}} is empty. Keeping all credentials in one place makes this easier to administer in a CI/CD environment.

[jilu1]

@fooman
This is a old post. As you may already know, we now support 3 different credential storage, local credentials file, AWS Secrets Manager, and Vault. Integration of these technology into CI/CD is well documented and already used internally. We are testing core and vendor bundle extensions in composer based package for coming Magento 2.4.2 release.
Please feel free to reach out to us if you have any issues.