Testing

[cmichelenstrofer]

Unit tests, integration tests, regression tests... this is a discussion for what is the best practice in scientific code and what we will do in WecOptTool.

• This is a great discussion on the topic and good arguments for using unit tests and for test-driven development.
• discussion on testing special cases and trends in scientific code.

[cmichelenstrofer]

Thoughts on the type of tests and WecOptTool:

• unit tests for each function. This test the specific function and only that function. It should include cases for which we know the answer (e.g. special cases such as when the inputs are zero, or synthetic cases, or trends). We haven't been doing this and I think we should. It would make debugging so much easier (ideally follow a test-driven development where the test for the function is written before the function).
• Integration tests: these call multiple functions and solve a bigger problem. This is what we have mostly been doing and have lots of examples in v1. We have many theoretical cases for which we know the true solution. But.. we have too many and it is slow. Need to consolidate/streamline these.
• Regression tests: only if there are no special or theoretical cases to test something against. E.g. more complex integration tests for which we don't have theoretical answers (real use of the code). Check that at the very least the results do not change between different versions/commits of the code.

Suggestion:

• one test file per module for unit tests, this mirrors the source folder structure.
• one file for integration tests (if this gets large, split into multiple files in some logical/intutitive way)
• No separate file/effort for regression tests. Use regression tests in the unit or integration tests if needed. There should be integration tests based on regression that represent more realistic test cases users would do (functionality for which there is no theoretical solution and for which WecOptTool would actually be used, such as non-linear constraints)

[cmichelenstrofer]

Pytest module:

• documentation
• another tutorial

some points:

• fixtures are modular: can call other fixtures.
• Can put fixtures in separate conftest.py so they can be used by different test files.
• You can mark tests with categories and only run certain categories. good to mark slow tests.
• You can parametrize a test to run with many different inputs
• Can design tests meant to fail
• more advanced topics (e.g. fixture scope) in documentation