[Proposal] Introducing new flags to run tests

[Smit-create]

I and @czgdp1807 had a discussion on adding some new features for running the tests like running all the tests with --show-c or any other such flags. We can also add a few more features to efficiently run a variety of tests. @czgdp1807 Can you please provide some demo CLIs that you would like to have?

[czgdp1807]

A simple example is,

python script_to_run_tests_with_given_flag.py --flag path/to/test/folder

A concrete example,

python script_to_run_tests_with_given_flag.py --show-c integration_tests/

Now script_to_run_tests_with_given_flag.py will execute lpython --show-c command for all the test files in integration_tests/.

@certik What do you think?

[certik]

We could do that, or we could implement an equivalent of what we do in LFortran: https://gitlab.com/lfortran/lfortran/-/blob/391ff8562350a9942bb02adb83a191219c42704f/integration_tests/CMakeLists.txt#L85 where we mark each test what backends it is supposed to compile with. As not all tests work with all backends, and at the same time, you want to ensure that those tests that work with say LLVM and C backends continue working.

[Smit-create]

I have the following suggestion to create a single run_tests.py that can do the following:

1. To run the integration tests for some folder test_me(by default llvm backend):

python run_tests.py test_me

2. To run the integration tests with some backend (say C) for some folder test_me:

python run_tests.py  --c-backend test_me

3. To show the asr of all the test files in some folder:

python run_tests.py --show-asr test_me

4. To update the tests (this is explicitly for dev purposes):

python run_tests.py -u test_me

What are your views on this? @certik @czgdp1807

[certik]

I would like to move the test infrastructure to libasr, and share it with LFortran as much as possible. See also this MR that we need to finish that makes it run in parallel: https://gitlab.com/lfortran/lfortran/-/merge_requests/1749.

For C integration tests, I think we should just use cmake, because we need to link in other C files etc.

I would not couple integration tests with tests.

But to be able to run a subset of tests, I think we can do that. Any change must be very carefully tested, to ensure the test runner will fail when it has to fail.

[czgdp1807]

I think we can close this because gh-565 has been merged.

[Smit-create]

This can be closed now.