diff --git a/README.md b/README.md index a1905879c785..943a82e29ecb 100644 --- a/README.md +++ b/README.md @@ -173,6 +173,7 @@ The mypy wiki contains some useful information for contributors: https://github.com/python/mypy/wiki/Developer-Guides + Working with the git version of mypy ------------------------------------ @@ -187,78 +188,11 @@ whenever you change branches, merge, rebase, or pull. (It's possible to automate this: Search Google for "git hook update submodule") -Running tests and linting -------------------------- - -First install any additional dependencies needed for testing: - - $ python3 -m pip install -U -r test-requirements.txt - -To run all tests, run the script `runtests.py` in the mypy repository: - - $ ./runtests.py - -Note that some tests will be disabled for older python versions. - -This will run all tests, including integration and regression tests, -and will type check mypy and verify that all stubs are valid. - -You can run a subset of test suites by passing positive or negative -filters: - - $ ./runtests.py lex parse -x lint -x stub - -For example, to run unit tests only, which run pretty quickly: - - $ ./runtests.py unit-test pytest - -The unit test suites are driven by a mixture of test frameworks: -mypy's own `myunit` framework, and `pytest`, which we're in the -process of migrating to. For finer control over which unit tests are -run and how, you can run `py.test` or `scripts/myunit` directly, or -pass inferior arguments via `-a`: - - $ py.test mypy/test/testcheck.py -v -k MethodCall - $ ./runtests.py -v 'pytest mypy/test/testcheck' -a -v -a -k -a MethodCall - - $ PYTHONPATH=$PWD scripts/myunit -m mypy.test.testlex -v '*backslash*' - $ ./runtests.py mypy.test.testlex -a -v -a '*backslash*' - -You can also run the type checker for manual testing without -installing anything by setting up the Python module search path -suitably (the lib-typing/3.2 path entry is not needed for Python 3.5 -or when you have manually installed the `typing` module): - - $ export PYTHONPATH=$PWD:$PWD/lib-typing/3.2 - $ python -m mypy PROGRAM.py - -You can add the entry scripts to PATH for a single python3 version: - - $ export PATH=$PWD/scripts - $ mypy PROGRAM.py - -You can check a module or string instead of a file: - - $ mypy PROGRAM.py - $ mypy -m MODULE - $ mypy -c 'import MODULE' - -To run the linter: - - $ ./runtests.py lint - - -Coverage reports ----------------- -There is an experimental feature to generate coverage reports. To use -this feature, you need to `pip install -U lxml`. This is an extension -module and requires various library headers to install; on a -Debian-derived system the command - `apt-get install python3-dev libxml2-dev libxslt1-dev` -may provide the necessary dependencies. +Tests +----- -To use the feature, pass e.g. `--txt-report "$(mktemp -d)"`. +See [Test README.md](test-data/unit/README.md) Development status diff --git a/test-data/unit/README.md b/test-data/unit/README.md new file mode 100644 index 000000000000..61ccd16cb0f0 --- /dev/null +++ b/test-data/unit/README.md @@ -0,0 +1,165 @@ +Tests +===== + + +Quick Start +----------- + +To add a simple unit test for a new feature you developed, open or create a +`test-data/unit/check-*.test` file with a name that roughly relates to the +feature you added. + +Add the test in this format anywhere in the file: + + [case testNewSyntaxBasics] + # flags: --python-version 3.6 + x: int + x = 5 + y: int = 5 + + a: str + a = 5 # E: Incompatible types in assignment (expression has type "int", variable has type "str") + b: str = 5 # E: Incompatible types in assignment (expression has type "int", variable has type "str") + + zzz: int + zzz: str # E: Name 'zzz' already defined + +- no code here is executed, just type checked +- optional `# flags: ` indicates which flags to use for this unit test +- `# E: abc...` indicates that this line should result in type check error +with text "abc..." +- note a space after `E:` and `flags:` +- lines without `# E: ` should cause no type check errors +- optional `[builtins fixtures/...]` tells the type checker to use +stubs from the indicated file (see Fixtures section below) +- optional `[out]` is an alternative to the "# E:" notation: it indicates that +any text after it contains the expected type checking error messages. +usually, "E: " is preferred because it makes it easier to associate the +errors with the code generating them at a glance, and to change the code of +the test without having to change line numbers in `[out]` +- an empty `[out]` section has no effect + + +Fixtures +-------- + +The unit tests use minimal stubs for builtins, so a lot of operations are not +possible. You should generally define any needed classes within the test case +instead of relying on builtins, though clearly this is not always an option +(see below for more about stubs in test cases). This way tests run much +faster and don't break if the stubs change. If your test crashes mysteriously +even though the code works when run manually, you should make sure you have +all the stubs you need for your test case, including built-in classes such as +`list` or `dict`, as these are not included by default. + + - The builtins used by default in unit tests live in + `test-data/unit/lib-stub`. + + - Individual test cases can override the stubs by using `[builtins + fixtures/foo.pyi]`; this targets files in `test-data/unit/fixtures`; feel + free to modify existing files there or create new ones as you deem fit. + + - You are also free to add additional stubs to this directory, but + generally don't update files in `lib-stub` without first discussing the + addition with other mypy developers, as additions could slow down the test + suite. + + +Running tests and linting +------------------------- + +First install any additional dependencies needed for testing: + + $ python3 -m pip install -U -r test-requirements.txt + +To run all tests, run the script `runtests.py` in the mypy repository: + + $ ./runtests.py + +Note that some tests will be disabled for older python versions. + +This will run all tests, including integration and regression tests, +and will type check mypy and verify that all stubs are valid. + +You can run a subset of test suites by passing positive or negative +filters: + + $ ./runtests.py lex parse -x lint -x stub + +For example, to run unit tests only, which run pretty quickly: + + $ ./runtests.py unit-test pytest + +The unit test suites are driven by a mixture of test frameworks: mypy's own +`myunit` framework, and `pytest`, which we're in the process of migrating to. +Test suites for individual components are in the files `mypy/test/test*.py`. +You can run many of these individually by doing `runtests.py testfoobar`. For +finer control over which unit tests are run and how, you can run `py.test` or +`scripts/myunit` directly, or pass inferior arguments via `-a`: + + $ py.test mypy/test/testcheck.py -v -k MethodCall + $ ./runtests.py -v 'pytest mypy/test/testcheck' -a -v -a -k -a MethodCall + + $ PYTHONPATH=$PWD scripts/myunit -m mypy.test.testlex -v '*backslash*' + $ ./runtests.py mypy.test.testlex -a -v -a '*backslash*' + +You can also run the type checker for manual testing without +installing anything by setting up the Python module search path +suitably (the lib-typing/3.2 path entry is not needed for Python 3.5 +or when you have manually installed the `typing` module): + + $ export PYTHONPATH=$PWD:$PWD/lib-typing/3.2 + $ python -m mypy PROGRAM.py + +You can add the entry scripts to PATH for a single python3 version: + + $ export PATH=$PWD/scripts + $ mypy PROGRAM.py + +You can check a module or string instead of a file: + + $ mypy PROGRAM.py + $ mypy -m MODULE + $ mypy -c 'import MODULE' + +To run the linter: + + $ ./runtests.py lint + +Many test suites store test case descriptions in text files +(`test-data/unit/*.test`). The module `mypy.test.data` parses these +descriptions. The package `mypy.myunit` contains the test framework used for +the non-checker test cases. + +Python evaluation test cases are a little different from unit tests +(`mypy/test/testpythoneval.py`, `test-data/unit/pythoneval.test`). These +type check programs and run them. Unlike the unit tests, these use the +full builtins and library stubs instead of minimal ones. Run them using +`runtests.py testpythoneval`. + +`runtests.py` by default runs tests in parallel using as many processes as +there are logical cores the `runtests.py` process is allowed to use (on +some platforms this information isn't available, so 2 processes are used by +default). You can change the number of workers using `-j` option. + +All pytest tests run as a single test from the perspective of `runtests.py`, +and so `-j` option has no effect on them. Instead, `pytest` itself determines +the number of processes to use. The default (set in `./pytest.ini`) is the +number of logical cores; this can be overridden using `-n` option. + +Note that running more processes than logical cores is likely to +significantly decrease performance; the relevant count is the number of +processes used by `runtests.py` plus those used by `pytest`. + + +Coverage reports +---------------- + +There is an experimental feature to generate coverage reports. To use +this feature, you need to `pip install -U lxml`. This is an extension +module and requires various library headers to install; on a +Debian-derived system the command + `apt-get install python3-dev libxml2-dev libxslt1-dev` +may provide the necessary dependencies. + +To use the feature, pass e.g. `--txt-report "$(mktemp -d)"`. \ No newline at end of file