You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
*[Beta backporting](beta-backporting.html). The mystery of the oft-misinterpreted `beta-nominated` / `beta-accepted` tags finally revealed.
36
+
*[Bots, websites and infrastructure](infrastructure.html). A catalog of the IRC bots, websites and other infrastructure used by the project, what they do, and who maintains them (i.e. who to contact when they malfunction and go on a bot rampage).
Copy file name to clipboardExpand all lines: test-suite.md
+1-227Lines changed: 1 addition & 227 deletions
Original file line number
Diff line number
Diff line change
@@ -5,231 +5,5 @@ title: The Rust Test Suite · The Rust Programming Language
5
5
6
6
# The Rust test suite
7
7
8
-
## THIS DOCUMENT IS DEPRECATED. PLEASE READ src/test/COMPILER_TESTS.md IN THE COMPILER REPO INSTEAD
8
+
**THIS DOCUMENT IS DEPRECATED. PLEASE READ src/test/COMPILER_TESTS.md IN THE COMPILER REPO INSTEAD**
9
9
10
-
The rust test suite has several sets of tests for different purposes. As the compiler is built over multiple stages, and with varying host and target combinations, debugging and profiling settings, the tests can be run in many different ways.
11
-
12
-
## Recipes
13
-
14
-
* Run the test suite: `make check`. This runs all stage2 tests. This is the criterion for a successful build.
15
-
* Run the test suite with benchmarks: `make check PLEASE_BENCH=1`
16
-
* Run only ignored (broken) tests: `make check CHECK_IGNORED=1`
17
-
* Run with verbose output: `make check VERBOSE=1`. This will print useful information like the exact commands being run.
18
-
* Run with valgrind: `make check CFG_ENABLE_VALGRIND=1`
19
-
* Run a specific test: `make check TESTNAME=[...]`
20
-
* The pattern `[...]` can be a complete path to a test, such as
21
-
`run-pass/foobar.rs`; it can also be any substring of a path.
22
-
For instance, `make check TESTNAME=foo` will run all tests that
23
-
have `foo` in some part of their filename.
24
-
* Note that while this will run only tests matching the given
25
-
pattern, it will still execute all test runners - most of them
26
-
will just not execute any tests. For more precise control, call
27
-
`make` on one of the targets below.
28
-
* You may need to `touch` the test source file to ensure that `make`
29
-
runs the necessary test runners.
30
-
* Run without parallelism: `make check RUST_TEST_TASKS=1` - This can make it easier to understand failure output.
31
-
* Build and test std without re-bootstrapping: `make check-stage1-std NO_REBUILD=1` - This makes the build/test cycle **much** faster. (Note: `NO_REBUILD` can also prevent compile tests from being rebuilt. If you want to rebuild those but not the compiler, look for files with the `.ok` extension in the `tmp` subdirectory and remove the appropriate ones.)
32
-
33
-
These options can be combined. For instance, `make check CHECK_IGNORED=1 TESTNAME=test/run-pass/foobar.rs` runs the ignored test `foobar.rs` in the `run-pass` directory.
34
-
35
-
## Language and compiler tests
36
-
37
-
These are tests of the compiler against Rust source code. They typically have a `main` function that takes no arguments and may have directives that instruct the test runner how to run the test. These tests may be compiled and executed, pretty-printed, jitted, etc. depending on the test configuration.
38
-
39
-
The test runner for these tests is at src/test/compiletest and is compiled to test/compiletest.stage[N].
There are seven different modes for compile tests. Each test is run under one or more modes:
53
-
54
-
* compile-fail - The test should fail to compile. Must include at least one expected error.
55
-
* run-fail - The test should compile but fail at runtime. Must include at least one error-pattern directive.
56
-
* run-pass - The test should compile and run successfully
57
-
* run-pass-valgrind - The test should compile and run (under Valgrind, where possible) successfully
58
-
* pretty - The test should round-trip through the pretty-printer and then compile successfully
59
-
* debug_info - The test will be compiled with debuginfo and an embedded `gdb` command script will be run to query the debuginfo
60
-
* codegen - The test will be compiled to LLVM bitcode, and a companion test written in C++ will be compiled to LLVM bitcode with `clang`, and the number of instructions will be compared
61
-
62
-
Valid directives include:
63
-
64
-
*`compile-flags:[...]` - Additional arguments to pass to the compiler
65
-
*`pp-exact` - The test should pretty-print exactly as written
66
-
*`pp-exact:[filename]` - The pretty-printed test should match the example in `filename`
67
-
*`ignore-test` - Test is broken, don't run it
68
-
*`ignore-pretty` - Test doesn't pretty-print correctly
69
-
*`error-pattern:[...]` - A message that should be expected on standard out. If multiple patterns are provided then they must all be matched, in order (**Note:** error-patterns are somewhat deprecated, see the section on Expected Errors below).
70
-
*`no-reformat` - Don't reformat the code when running the test through the pretty-printer
71
-
*`aux-build:foo.rs` - Compiles an auxiliary library for use in multi-crate tests. See the section on multi-crate testing below.
72
-
*`exec-env:NAME` or `exec-env:NAME=VALUE` - Sets an environment variable during test execution
73
-
*`debugger:CMD` - Issues a command to the debugger in `debuginfo` mode
74
-
*`check:RESULT` - Checks the result of a previous `debugger:` directive in `debuginfo` mode
75
-
76
-
There are eight directories containing compile tests, living in the src/tests directory:
77
-
78
-
* run-pass - Tests that are expected to compile and run successfully. Also used for pretty-print testing.
79
-
* run-pass-valgrind - Tests that are expected to compile and run successfully under Valgrind. Also used for pretty-print testing.
80
-
* run-fail - Tests that are expected compile but fail when run. Also used for pretty-print testing.
81
-
* compile-fail - Tests that are expected not to compile
82
-
* bench - Benchmarks and miscellaneous snippets of code that are expected to compile and run successfully. Also used for pretty-print testing.
83
-
* pretty - Pretty-print tests
84
-
* debug-info - Debuginfo tests
85
-
* codegen - Codegen tests
86
-
* auxiliary - libraries used in multi-crate testing. See the section on this topic below.
87
-
88
-
And finally, build targets:
89
-
90
-
*`check-stage[N]-rpass` - The tests in the run-pass directory, in run-pass mode
91
-
*`check-stage[N]-rpass-valgrind` - The tests in the run-pass-valgrind directory, in run-pass-valgrind mode
92
-
*`check-stage[N]-rfail` - The tests in the run-fail-directory, in run-fail mode
93
-
*`check-stage[N]-cfail` - The tests in the compile-fail directory, in compile-fail mode
94
-
*`check-stage[N]-bench` - The tests in the bench directory, in run-pass mode
95
-
*`check-stage[N]-pretty` - All the pretty-print tests
96
-
*`check-stage[N]-pretty-rpass` - The tests in the run-pass directory, in pretty mode
97
-
*`check-stage[N]-pretty-rpass-valgrind` - The tests in the run-pass-valgrind directory, in pretty mode
98
-
*`check-stage[N]-pretty-rfail` - The tests in the run-fail directory, in pretty mode
99
-
*`check-stage[N]-pretty-bench` - The tests in the bench directory, in pretty mode
100
-
*`check-stage[N]-pretty-pretty` - The tests in the pretty directory, in pretty mode
101
-
*`check-stage[N]-debuginfo` - The tests in the debuginfo directory
102
-
*`check-stage[N]-codegen` - The tests in the codegen directory
103
-
104
-
### Specifying the expected errors and warnings
105
-
106
-
When writing a compile-fail test, you must specify at least one
107
-
expected error or warning message. The preferred way to do this is to
108
-
place a comment with the form `//~ ERROR msg` or `//~ WARNING msg` on
109
-
the line where the error or warning is expected to occur. You may
110
-
have as many of these comments as you like. The test harness will
111
-
verify that the compiler reports precisely the errors/warnings that are
112
-
specified, no more and no less. An example of using the error/warning
113
-
messages is:
114
-
115
-
```rust
116
-
// Regression test for issue #XXX
117
-
118
-
fnmain() {
119
-
leta:bool=10; //~ ERROR mismatched types
120
-
log (debug, b);
121
-
}
122
-
```
123
-
124
-
In fact, this test would fail, because there are two errors: the type
125
-
mismatch and the undefined variable `b`.
126
-
127
-
Sometimes it is not possible or not convenient to place the `//~`
128
-
comment on precisely the line where the error occurs. For those cases,
129
-
you may make a comment of the form `//~^` where the caret `^`
130
-
indicates that the error is expected to appear on the line above. You
131
-
may have as many caret as you like, so `//~^^^ ERROR foo` indicates
132
-
that the error message `foo` is expected to be reported 3 lines above
133
-
the comment. We could therefore correct the above test like so:
134
-
135
-
```rust
136
-
// Regression test for issue #XXX
137
-
138
-
fnmain() {
139
-
leta:bool=10; //~ ERROR mismatched types
140
-
log (debug, b);
141
-
//~^ ERROR undefined variable `b`
142
-
}
143
-
```
144
-
145
-
The older technique for specifying error messages was to use an
146
-
`error-pattern` directive. These directives are placed at the top of
147
-
the file and each message found in an `error-pattern` directive must
148
-
appear in the output.
149
-
150
-
Using error comments is preferred, however, because it is a more
151
-
thorough test:
152
-
153
-
- It verifies that the error is reported on the expected line number.
154
-
- It verifies that no additional errors or warnings are reported.
155
-
156
-
### Multi-crate testing
157
-
158
-
Sometimes it is useful to write tests that make use of more than one crate. We have limited support for this scenario. Basically, you can write and add modules into the `src/test/auxiliary` directory. These files are not built nor tested directly. Instead, you write a main test in one of the other directories (`run-pass`, `compile-fail`, etc) and add a `aux-build` directive at the head of the main test. When running the main test, the test framework will build the files it is directed to build from the auxiliary directory. These builds *must* succeed or the test will fail. You can then include `use` and `import` commands to make use of the byproducts from these builds as you wish.
159
-
160
-
An example consisting of two files:
161
-
162
-
```rust
163
-
auxiliary/cci_iter_lib.rs:
164
-
#[inline]
165
-
fniter<T>(v: [T], f:fn(T)) {...}
166
-
167
-
run-pass/cci_iter_exe.rs:
168
-
// aux-build:cci_iter_lib.rs
169
-
externcrate cci_iter_lib;
170
-
fnmain() {
171
-
cci_iter_lib::iter([1, 2, 3]) {|i|... }
172
-
}
173
-
```
174
-
175
-
### Recipes
176
-
177
-
* Running the run-pass tests for stage1: `make check-stage1-rpass`
178
-
* Running a specific compile-fail test: `make check-stage2-cfail TESTNAME=type-mismatch`
179
-
* Finding the command to compile a failing test: `make check-stage1-rpass TESTNAME=hello VERBOSE=1`
Most crates include <ahref="https://github.com/mozilla/rust/wiki/Doc-unit-testing">unit tests</a> which are part of the crate they test. These crates are built with the `--test` flag and run as part of `make check`.
185
-
186
-
`libcore` has its tests in a separate crate, named `libcoretest`.
187
-
188
-
All tests in a module should go in an inner module named `test`, with the attribute `#[cfg(test)]`. Placing tests in their own module is a practical issue - because test cases are not included in normal builds, building with `--test` require a different set of imports than without, and that causes 'unused import' errors.
189
-
190
-
```rust
191
-
usestd::option;
192
-
193
-
fndo_something() { ... }
194
-
195
-
#[cfg(test)]
196
-
modtest {
197
-
usestd::vec;
198
-
199
-
fnhelper_fn() { ... }
200
-
201
-
#[test]
202
-
fnresolve_fn_types() {
203
-
...
204
-
}
205
-
}
206
-
```
207
-
208
-
### Build targets
209
-
210
-
*`make check-stage[N]-crates`
211
-
*`make check-stage[N]-[crate name]test`
212
-
213
-
## Documentation tests
214
-
215
-
The build system is able to extract Rust code snippets from documentation and run them using the compiletest driver. Currently the tutorial and reference manual are tested this way. The targets are `make check-stage[N]-doc-tutorial` and `make check-stage[N]-doc-rust`, respectively. There are also several auxiliary guides; to run the tests extracted from them, do:
216
-
217
-
*`make check-stage[N]-doc-guide-borrowed-ptr`
218
-
*`make check-stage[N]-doc-guide-tasks`
219
-
*`make check-stage[N]-doc-guide-ffi`
220
-
*`make check-stage[N]-doc-guide-macros`
221
-
*`make check-stage[N]-doc-guide-testing`
222
-
223
-
Crate API docs are tested as well:
224
-
225
-
*`make check-stage[N]-doc-crate-std`
226
-
227
-
To run all doc tests use `make check-stage[N]-doc`.
228
-
229
-
## Minimal (but faster) checking on windows
230
-
231
-
Because Windows has slow process spawning running `make check` on that platform can take a long time. For this reason we have a `make check-lite` target that the Windows build servers run to keep the cycle time down. This is a stripped-down target which only checks run-pass, run-fail, compile-fail, run-make and target libraries.
232
-
233
-
## Benchmarks, saved metrics and ratchets
234
-
235
-
All benchmark metrics are saved by default. Depending on configuration, some benchmark metrics are ratcheted. The `codegen` compile tests are always ratcheted if they run, since they are deterministic / low-noise. See [[Doc unit testing]] for details on metrics and ratchets.
0 commit comments