Skip to content

Commit 73a513c

Browse files
committed
tests: Require run-fail ui tests to have an exit code (SIGABRT not ok)
And introduce two new directives for ui tests: * `run-crash` * `run-fail-or-crash` Normally a `run-fail` ui test like tests that panic shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal/crash however. Introduce and use `run-crash` for those tests. Note that Windows crashes are not handled by signals but by certain high bits set on the process exit code. Example exit code for crash on Windows: `0xc000001d`. Because of this, we define "crash" on all platforms as "not exit with success and not exit with a regular failure code in the range 1..=127". Some tests behave differently on different targets: * Targets without unwind support will abort (crash) instead of exit with failure code 101 after panicking. As a special case, allow crashes for `run-fail` tests for such targets. * Different sanitizer implementations handle detected memory problems differently. Some abort (crash) the process while others exit with failure code 1. Introduce and use `run-fail-or-crash` for such tests.
1 parent 98dafe7 commit 73a513c

File tree

2 files changed

+19
-7
lines changed

2 files changed

+19
-7
lines changed

src/tests/directives.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,10 @@ expectations](ui.md#controlling-passfail-expectations).
7575
| `check-fail` | Building (no codegen) should fail | `ui`, `crashes` | N/A |
7676
| `build-pass` | Building should pass | `ui`, `crashes`, `codegen`, `incremental` | N/A |
7777
| `build-fail` | Building should fail | `ui`, `crashes` | N/A |
78-
| `run-pass` | Running the test binary should pass | `ui`, `crashes`, `incremental` | N/A |
79-
| `run-fail` | Running the test binary should fail | `ui`, `crashes` | N/A |
78+
| `run-pass` | Program must exit with code `0` | `ui`, `crashes`, `incremental` | N/A |
79+
| `run-fail` | Program must exit with code `1..=127` | `ui`, `crashes` | N/A |
80+
| `run-crash` | Program must crash | `ui` | N/A |
81+
| `run-fail-or-crash` | Program must `run-fail` or `run-crash` | `ui` | N/A |
8082
| `ignore-pass` | Ignore `--pass` flag | `ui`, `crashes`, `codegen`, `incremental` | N/A |
8183
| `dont-check-failure-status` | Don't check exact failure status (i.e. `1`) | `ui`, `incremental` | N/A |
8284
| `failure-status` | Check | `ui`, `crashes` | Any `u16` |

src/tests/ui.md

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -448,7 +448,7 @@ even run the resulting program. Just add one of the following
448448
- `//@ build-pass` — compilation and linking should succeed but do
449449
not run the resulting binary.
450450
- `//@ run-pass` — compilation should succeed and running the resulting
451-
binary should also succeed.
451+
binary should make it exit with code 0 which indicates success.
452452
- Fail directives:
453453
- `//@ check-fail` — compilation should fail (the codegen phase is skipped).
454454
This is the default for UI tests.
@@ -457,10 +457,20 @@ even run the resulting program. Just add one of the following
457457
- First time is to ensure that the compile succeeds without the codegen phase
458458
- Second time is to ensure that the full compile fails
459459
- `//@ run-fail` — compilation should succeed, but running the resulting
460-
binary should fail.
461-
462-
For `run-pass` and `run-fail` tests, by default the output of the program itself
463-
is not checked.
460+
binary should make it exit with a code in the range `1..=127` which
461+
indicates regular failure. On targets without unwind support, crashes
462+
are also accepted.
463+
- `//@ run-crash` — compilation should succeed, but running the resulting
464+
binary should fail with a crash. Crashing is defined as "not exiting with
465+
a code in the range `0..=127`". Example on Linux: Termination by `SIGABRT`
466+
or `SIGSEGV`. Example on Windows: Exiting with the code for
467+
`STATUS_ILLEGAL_INSTRUCTION` (`0xC000001D`).
468+
- `//@ run-fail-or-crash` — compilation should succeed, but running the
469+
resulting binary should either `run-fail` or `run-crash`. Useful if a test
470+
crashes on some targets but just fails on others.
471+
472+
For `run-pass`. `run-fail`, `run-crash` and `run-fail-or-crash` tests, by
473+
default the output of the program itself is not checked.
464474

465475
If you want to check the output of running the program, include the
466476
`check-run-results` directive. This will check for a `.run.stderr` and

0 commit comments

Comments
 (0)