Skip to content

Commit 0ff8e47

Browse files
adpaco-awstedinski
authored andcommitted
Add section on "fixme" tests (rust-lang#731)
1 parent b5e904a commit 0ff8e47

File tree

1 file changed

+29
-0
lines changed

1 file changed

+29
-0
lines changed

rmc-docs/src/rmc-testing.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,10 @@ In particular, the RMC testing suites are composed of:
6161
[Prusti](https://github.com/viperproject/prusti-dev) tool.
6262
* `smack`: Works like `rmc` but contains tests from the
6363
[SMACK](https://github.com/smackers/smack) tool.
64+
* `rmc-fixme`: Similar to `rmc`, but runs ignored tests from the `rmc` testing
65+
suite (i.e., tests with `fixme` or `ignore` in their name).
66+
Allows us to detect when a previously not supported test becomes
67+
supported. More details in ["Fixme" tests](#"fixme"-tests).
6468
* `expected`: Similar to `rmc` but with an additional check which ensures that
6569
lines appearing in `*.expected` files appear in the output
6670
generated by `rmc`.
@@ -135,6 +139,31 @@ Alternatively, CBMC flags can also be passed using `cbmc-flags`:
135139
For `cargo-rmc` tests, the preferred way to pass command-line options is adding
136140
them to `Cargo.toml` below the `[rmc.flags]` marker.
137141

142+
### "Fixme" tests
143+
144+
Any test containing `fixme` or `ignore` in its name is considered a test not
145+
supported for some reason. Generally, these tests are not run because they will
146+
either return a wrong result (since they are not supported) or take a long time
147+
to finish.
148+
149+
However, "fixme" tests included in the `rmc` folder are run via the `rmc-fixme`
150+
testing suite. `rmc-fixme` works on test files from `rmc` but:
151+
1. Only runs tests whose name contains `fixme` or `ignore` (ignoring the rest).
152+
2. The expected outcome is failure. In other words, a test is successful if it
153+
fails.
154+
155+
We welcome contributions with "fixme" tests which demonstrate a bug or
156+
unsupported feature in RMC. Ideally, the test should include some comments
157+
regarding:
158+
* The expected result of the test.
159+
* The actual result of the test (e.g., interesting parts of the output).
160+
* Links to related issues.
161+
162+
To include a new "fixme" test in `rmc` you only need to ensure its name contains
163+
`fixme` or `ignore`. If your changes to RMC cause a "fixme" test to become
164+
supported, you only need to rename it so the name does not contain `fixme` nor
165+
`ignore`.
166+
138167
## Rust unit tests
139168

140169
These tests follow the

0 commit comments

Comments
 (0)