Travis CI does not check API functional tests

[dverkade]

Summary

Currently Travis CI does not execute the API functional tests which are included in Magento. This way developers submitting a PR are not sure of all tests will pass. This will cause a delay in the processing of the actual PR and disappointment for the developer because he thinks he passed all tests, but actually didn't.

With the headless architecture, API's are getting more and more important. So it would be a logical step to include the API functional tests in Travis CI as well so that developers know that there code will pass the tests.

Examples

See this PR where all tests in Travis passed, but a API functional test was failing

• [Bugfix] #7333 Unable to set negative custom option fixed price in admin view. #15267

Proposed solution

Extend Travis CI to run the API functional tests as well.

[magento-engcom-team]

Hi @dverkade. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me {$VERSION} instance

where {$VERSION} is version tags (starting from 2.2.0+) or develop branches (2.2-develop +).
For more details, please, review the Magento Contributor Assistant documentation.

@dverkade do you confirm that you was able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

[ishakhsuvarov]

Hi @dverkade

Thank you for raising this topic.

Unfortunately Travis CI has limited capacity regarding number and execution time of tests. The ones currently executed are considered to be the most critical and picked to be executable by Travis in a reasonable amount of time.

There are more tests missing currently, like performance tests, full functional tests suite (there are ~1000 variations bundled with magento), sample data tests, health check tests, etc.

To combat this issue we are currently working on a solution, which would allow to execute Pull Requests using Magento's internal CICD, capable of running full test suite in ~2 hours, and publish build results and artifacts back to GitHub. This implementation process takes some time, but work is already in progress.

Considering information above, I don't think we are going to invest additional time in extending existing Travis CI solution and spend it to speed up artifacts publication development instead.

I am going to close this ticket with won't fix resolution for now, if you don't mind. Should you have any more questions on the subject please feel free to continue this thread or reach us out directly.

Thanks