Automation to Run CLI e2e tests on various different OS #25955
Description
Related command
azdev test azext_connectedk8s.{function/command_to_test} --discover --live
Is your feature request related to a problem? Please describe.
Our team owns the connectedk8s CLI extension.
There are two problems which we want to highlight regarding cli e2e testing framework:
- One of our customers recently reported an issue when running the az connectedk8s commands on a Cent OS 7 VM. The 'az connectedk8s connect' command was failing with 'Import Error' since we had introduced circular imports in the 1.3.9 version of the connectedk8s CLI. However, this issue did not repro on Ubuntu & Windows and hence we could not catch this during e2e tests
- The CLI extension testing pipeline runs all of the tests only for Linux (not sure which one). We claim that the connectedk8s CLI will work on any Linux, Windows & Mac distros. However, we end up doing manual testing for on windows and mac before releasing the connectedk8s CLI code changes.
Describe the solution you'd like
CLI e2e test pipeline provides a way to run automation tests on various OS. We are specifically looking out for options to run the e2e tests on Windows, Ubuntu, Cent OS 7 & Mac OS for now. This would help us save testing time/cost(solve problem #2) and also prevent bugs from being released for certain types of OS distros (as explained in problem #1 above)
Describe alternatives you've considered
The only alternative is to do manual testing locally with Windows, Mac, Cent OS 7, and the testing pipeline can be used to test on Linux. This is neither a wholesome experience to the developer and nor the adoption of testing pipeline that easy when manual steps are required
Additional context