From a8e734f4e61250b97c735009bc4581de19965a01 Mon Sep 17 00:00:00 2001 From: Max Moroz Date: Wed, 15 Mar 2017 23:35:06 -0700 Subject: [PATCH 1/4] Add test README --- README.md | 6 +++ test-data/unit/README.md | 95 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) create mode 100644 test-data/unit/README.md diff --git a/README.md b/README.md index a1905879c785..b90c0cb44c2e 100644 --- a/README.md +++ b/README.md @@ -261,6 +261,12 @@ may provide the necessary dependencies. To use the feature, pass e.g. `--txt-report "$(mktemp -d)"`. +Writing tests +------------- + +See [Unit Test README.md](test-data/unit/README.md) + + Development status ------------------ diff --git a/test-data/unit/README.md b/test-data/unit/README.md new file mode 100644 index 000000000000..d63e983bb49b --- /dev/null +++ b/test-data/unit/README.md @@ -0,0 +1,95 @@ +Tests +===== + + +Unit tests +---------- + +To add a simple unit test for a new feature you developed, open or create a +`test-data/unit/check-*.test` file with a name that roughly relates to the +feature you added. + +Add the test in this format anywhere in the file: + + [case testNewSyntaxBasics] + # flags: --python-version 3.6 + x: int + x = 5 + y: int = 5 + + a: str + a = 5 # E: Incompatible types in assignment (expression has type "int", variable has type "str") + b: str = 5 # E: Incompatible types in assignment (expression has type "int", variable has type "str") + + zzz: int + zzz: str # E: Name 'zzz' already defined + +- no code here is executed, just type checked +- `# flags: ` indicates which flags to use for this unit test +- `# E: abc...` indicates that this line should result in type check error +with text "abc..." +- note a space after `E:` and `flags:` +- lines without `# E: ` should cause no type check errors +- optional `[builtins]` tells the type checker to use stubs from the indicated +file (see Fixtures section below) +- optional `[out]` is an alternative to the "# E:" notation: it indicates that +any text after it contains the expected type checking error messages. +usually, "E: " is preferred because it makes it easier to associate the +errors with the code generating them at a glance, and to change the code of +the test without having to change line numbers in `[out]` +- an empty `[out]` section has no effect + + +Fixtures +-------- + +The unit tests use minimal stubs for builtins, so a lot of operations are not +possible. You should generally define any needed classes within the test case +instead of relying on builtins, though clearly this is not always an option +(see below for more about stubs in test cases). This way tests run much +faster and don't break if the stubs change. If your test crashes mysteriously +even though the code works when run manually, you should make sure you have +all the stubs you need for your test case, including built-in classes such as +`list` or `dict`, as these are not included by default. + + - The builtins used by default in unit tests live in + `test-data/unit/lib-stub`. + + - Individual test cases can override the stubs by using `[builtins + fixtures/foo.py]`; this targets files in `test-data/unit/fixtures`; feel + free to modify existing files there or create new ones as you deem fit. + + - You are also free to add additional stubs to this directory, but + generally don't update files in `lib-stub` without first discussing the + addition with other mypy developers, as additions could slow down the test + suite. + + +Test suites +----------- + +The file `runtests.py` runs all the tests. Test suites for individual +components are in the files `mypy/test/test*.py`. You can run many of these +individually by doing `runtests.py testfoobar`. Type checker test cases +(`testcheck.py`) have been migrated to pytest and you can run them using +`py.test mypy`. (We are planning to migrate all test cases to pytest +eventually -- any help is appreciated here.) + +Many test suites store test case descriptions in text files +(`test-data/unit/*.test`). The module `mypy.test.data` parses these +descriptions. The package `mypy.myunit` contains the test framework used for +the non-checker test cases. + +Notes about test cases: + + - Python evaluation test cases are a little different from unit tests + (`mypy/test/testpythoneval.py`, `test-data/unit/pythoneval.test`). These + type check programs and run them. Unlike the unit tests, these use the + full builtins and library stubs instead of minimal ones. Run them using + `runtests.py testpythoneval`. + +More things (this should be expanded): + + - You can run just a subset of tests by giving a wildcard argument, e.g. + `runtests.py unit-test -a "*Overload*"` to run all unit tests with + `Overload` as a substring. From 01b9a4a4dd5239aca579e852f86d5240035f5420 Mon Sep 17 00:00:00 2001 From: Max Moroz Date: Thu, 16 Mar 2017 14:15:30 -0700 Subject: [PATCH 2/4] Moved test running instructions from main README to test README --- README.md | 80 ++---------------------------- test-data/unit/README.md | 102 ++++++++++++++++++++++++++++++--------- 2 files changed, 84 insertions(+), 98 deletions(-) diff --git a/README.md b/README.md index b90c0cb44c2e..943a82e29ecb 100644 --- a/README.md +++ b/README.md @@ -173,6 +173,7 @@ The mypy wiki contains some useful information for contributors: https://github.com/python/mypy/wiki/Developer-Guides + Working with the git version of mypy ------------------------------------ @@ -187,84 +188,11 @@ whenever you change branches, merge, rebase, or pull. (It's possible to automate this: Search Google for "git hook update submodule") -Running tests and linting -------------------------- - -First install any additional dependencies needed for testing: - - $ python3 -m pip install -U -r test-requirements.txt - -To run all tests, run the script `runtests.py` in the mypy repository: - - $ ./runtests.py - -Note that some tests will be disabled for older python versions. - -This will run all tests, including integration and regression tests, -and will type check mypy and verify that all stubs are valid. - -You can run a subset of test suites by passing positive or negative -filters: - - $ ./runtests.py lex parse -x lint -x stub - -For example, to run unit tests only, which run pretty quickly: - - $ ./runtests.py unit-test pytest - -The unit test suites are driven by a mixture of test frameworks: -mypy's own `myunit` framework, and `pytest`, which we're in the -process of migrating to. For finer control over which unit tests are -run and how, you can run `py.test` or `scripts/myunit` directly, or -pass inferior arguments via `-a`: - - $ py.test mypy/test/testcheck.py -v -k MethodCall - $ ./runtests.py -v 'pytest mypy/test/testcheck' -a -v -a -k -a MethodCall - - $ PYTHONPATH=$PWD scripts/myunit -m mypy.test.testlex -v '*backslash*' - $ ./runtests.py mypy.test.testlex -a -v -a '*backslash*' -You can also run the type checker for manual testing without -installing anything by setting up the Python module search path -suitably (the lib-typing/3.2 path entry is not needed for Python 3.5 -or when you have manually installed the `typing` module): - - $ export PYTHONPATH=$PWD:$PWD/lib-typing/3.2 - $ python -m mypy PROGRAM.py - -You can add the entry scripts to PATH for a single python3 version: - - $ export PATH=$PWD/scripts - $ mypy PROGRAM.py - -You can check a module or string instead of a file: - - $ mypy PROGRAM.py - $ mypy -m MODULE - $ mypy -c 'import MODULE' - -To run the linter: - - $ ./runtests.py lint - - -Coverage reports ----------------- - -There is an experimental feature to generate coverage reports. To use -this feature, you need to `pip install -U lxml`. This is an extension -module and requires various library headers to install; on a -Debian-derived system the command - `apt-get install python3-dev libxml2-dev libxslt1-dev` -may provide the necessary dependencies. - -To use the feature, pass e.g. `--txt-report "$(mktemp -d)"`. - - -Writing tests -------------- +Tests +----- -See [Unit Test README.md](test-data/unit/README.md) +See [Test README.md](test-data/unit/README.md) Development status diff --git a/test-data/unit/README.md b/test-data/unit/README.md index d63e983bb49b..f1df8a6f5696 100644 --- a/test-data/unit/README.md +++ b/test-data/unit/README.md @@ -2,8 +2,8 @@ Tests ===== -Unit tests ----------- +Quick Start +----------- To add a simple unit test for a new feature you developed, open or create a `test-data/unit/check-*.test` file with a name that roughly relates to the @@ -16,11 +16,11 @@ Add the test in this format anywhere in the file: x: int x = 5 y: int = 5 - + a: str a = 5 # E: Incompatible types in assignment (expression has type "int", variable has type "str") b: str = 5 # E: Incompatible types in assignment (expression has type "int", variable has type "str") - + zzz: int zzz: str # E: Name 'zzz' already defined @@ -65,31 +65,89 @@ all the stubs you need for your test case, including built-in classes such as suite. -Test suites ------------ +Running tests and linting +------------------------- + +First install any additional dependencies needed for testing: + + $ python3 -m pip install -U -r test-requirements.txt + +To run all tests, run the script `runtests.py` in the mypy repository: + + $ ./runtests.py + +Note that some tests will be disabled for older python versions. + +This will run all tests, including integration and regression tests, +and will type check mypy and verify that all stubs are valid. + +You can run a subset of test suites by passing positive or negative +filters: + + $ ./runtests.py lex parse -x lint -x stub + +For example, to run unit tests only, which run pretty quickly: + + $ ./runtests.py unit-test pytest -The file `runtests.py` runs all the tests. Test suites for individual -components are in the files `mypy/test/test*.py`. You can run many of these -individually by doing `runtests.py testfoobar`. Type checker test cases -(`testcheck.py`) have been migrated to pytest and you can run them using -`py.test mypy`. (We are planning to migrate all test cases to pytest -eventually -- any help is appreciated here.) +The unit test suites are driven by a mixture of test frameworks: mypy's own +`myunit` framework, and `pytest`, which we're in the process of migrating to. +Test suites for individual components are in the files `mypy/test/test*.py`. +You can run many of these individually by doing `runtests.py testfoobar`. For +finer control over which unit tests are run and how, you can run `py.test` or +`scripts/myunit` directly, or pass inferior arguments via `-a`: + + $ py.test mypy/test/testcheck.py -v -k MethodCall + $ ./runtests.py -v 'pytest mypy/test/testcheck' -a -v -a -k -a MethodCall + + $ PYTHONPATH=$PWD scripts/myunit -m mypy.test.testlex -v '*backslash*' + $ ./runtests.py mypy.test.testlex -a -v -a '*backslash*' + +You can also run the type checker for manual testing without +installing anything by setting up the Python module search path +suitably (the lib-typing/3.2 path entry is not needed for Python 3.5 +or when you have manually installed the `typing` module): + + $ export PYTHONPATH=$PWD:$PWD/lib-typing/3.2 + $ python -m mypy PROGRAM.py + +You can add the entry scripts to PATH for a single python3 version: + + $ export PATH=$PWD/scripts + $ mypy PROGRAM.py + +You can check a module or string instead of a file: + + $ mypy PROGRAM.py + $ mypy -m MODULE + $ mypy -c 'import MODULE' + +To run the linter: + + $ ./runtests.py lint Many test suites store test case descriptions in text files (`test-data/unit/*.test`). The module `mypy.test.data` parses these descriptions. The package `mypy.myunit` contains the test framework used for the non-checker test cases. -Notes about test cases: +Python evaluation test cases are a little different from unit tests +(`mypy/test/testpythoneval.py`, `test-data/unit/pythoneval.test`). These +type check programs and run them. Unlike the unit tests, these use the +full builtins and library stubs instead of minimal ones. Run them using +`runtests.py testpythoneval`. + + +Coverage reports +---------------- + +There is an experimental feature to generate coverage reports. To use +this feature, you need to `pip install -U lxml`. This is an extension +module and requires various library headers to install; on a +Debian-derived system the command + `apt-get install python3-dev libxml2-dev libxslt1-dev` +may provide the necessary dependencies. - - Python evaluation test cases are a little different from unit tests - (`mypy/test/testpythoneval.py`, `test-data/unit/pythoneval.test`). These - type check programs and run them. Unlike the unit tests, these use the - full builtins and library stubs instead of minimal ones. Run them using - `runtests.py testpythoneval`. +To use the feature, pass e.g. `--txt-report "$(mktemp -d)"`. -More things (this should be expanded): - - You can run just a subset of tests by giving a wildcard argument, e.g. - `runtests.py unit-test -a "*Overload*"` to run all unit tests with - `Overload` as a substring. From 8a98dce30b8fc43a158e1004a01e2747ae73ea87 Mon Sep 17 00:00:00 2001 From: Max Moroz Date: Thu, 16 Mar 2017 16:47:34 -0700 Subject: [PATCH 3/4] Add parallelization notes --- test-data/unit/README.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/test-data/unit/README.md b/test-data/unit/README.md index f1df8a6f5696..2d82094ca9ac 100644 --- a/test-data/unit/README.md +++ b/test-data/unit/README.md @@ -25,13 +25,13 @@ Add the test in this format anywhere in the file: zzz: str # E: Name 'zzz' already defined - no code here is executed, just type checked -- `# flags: ` indicates which flags to use for this unit test +- optional `# flags: ` indicates which flags to use for this unit test - `# E: abc...` indicates that this line should result in type check error with text "abc..." - note a space after `E:` and `flags:` - lines without `# E: ` should cause no type check errors -- optional `[builtins]` tells the type checker to use stubs from the indicated -file (see Fixtures section below) +- optional `[builtins fixtures/...]` tells the type checker to use +stubs from the indicated file (see Fixtures section below) - optional `[out]` is an alternative to the "# E:" notation: it indicates that any text after it contains the expected type checking error messages. usually, "E: " is preferred because it makes it easier to associate the @@ -56,7 +56,7 @@ all the stubs you need for your test case, including built-in classes such as `test-data/unit/lib-stub`. - Individual test cases can override the stubs by using `[builtins - fixtures/foo.py]`; this targets files in `test-data/unit/fixtures`; feel + fixtures/foo.pyi]`; this targets files in `test-data/unit/fixtures`; feel free to modify existing files there or create new ones as you deem fit. - You are also free to add additional stubs to this directory, but @@ -137,6 +137,20 @@ type check programs and run them. Unlike the unit tests, these use the full builtins and library stubs instead of minimal ones. Run them using `runtests.py testpythoneval`. +`runtests.py` by default runs tests in parallel using as many processes as +there are logical cores the `runtests.py` process is allowed to use (on +Windows this information isn't available, so 2 processes are used by +default). You can change the number of workers using `-j` option. + +All pytest tests run as a single test from the perspective of `runtests.py`, +and so `-j` option has no effect on them. Instead, `pytest` itself determines +the number of processes to use. The default (set in `./pytest.ini`) is the +number of logical cores; this can be overridden using `-n` option. + +Note that running more processes than logical cores is likely to +significantly decrease performance; the relevant count is the number of +processes used by `runtests.py` plus those used by `pytest`. + Coverage reports ---------------- From 94d7fa92339a4f201ed10cbbb7145afe349488ab Mon Sep 17 00:00:00 2001 From: Max Moroz Date: Sat, 18 Mar 2017 13:46:53 -0700 Subject: [PATCH 4/4] Address CR --- test-data/unit/README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/test-data/unit/README.md b/test-data/unit/README.md index 2d82094ca9ac..61ccd16cb0f0 100644 --- a/test-data/unit/README.md +++ b/test-data/unit/README.md @@ -139,7 +139,7 @@ full builtins and library stubs instead of minimal ones. Run them using `runtests.py` by default runs tests in parallel using as many processes as there are logical cores the `runtests.py` process is allowed to use (on -Windows this information isn't available, so 2 processes are used by +some platforms this information isn't available, so 2 processes are used by default). You can change the number of workers using `-j` option. All pytest tests run as a single test from the perspective of `runtests.py`, @@ -162,6 +162,4 @@ Debian-derived system the command `apt-get install python3-dev libxml2-dev libxslt1-dev` may provide the necessary dependencies. -To use the feature, pass e.g. `--txt-report "$(mktemp -d)"`. - - +To use the feature, pass e.g. `--txt-report "$(mktemp -d)"`. \ No newline at end of file