Skip to content

Commit 806f313

Browse files
jieyouxujieyouxu
committed
Propose compiletest directive handling rework project goal for 2025H1
1 parent 1b1c674 commit 806f313

File tree

1 file changed

+151
-0
lines changed

1 file changed

+151
-0
lines changed

‎src/2025h1/compiletest-directive-rework.md

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# (WIP) Making compiletest more maintainable: reworking directive handling
2+
3+
> **TODO**: speak to @Zalathar and @tgross35 for potential project goal co-owners and/or dedicated
4+
> reviewers.
5+
6+
| Metadata | |
7+
|----------|-------------------------|
8+
| Owner(s) | [@jieyouxu] |
9+
| Teams | [bootstrap], [compiler] |
10+
| Status | Proposed |
11+
12+
## Summary
13+
14+
*Short description of what you will do over the next 6 months.*
15+
16+
Rework [`compiletest`]'s directive handling to make it more maintainable, have better UX for
17+
compiler contributors, and fix some long-standing issues.
18+
19+
## Motivation
20+
21+
`rustc` relies on the test infrastructure implemented by the test harness [`compiletest`] (supported
22+
by bootstrap) to run the test suites under `tests/` (e.g. `ui` tests, `mir-opt` tests, `run-make`
23+
tests, etc.). However, [`compiletest`] is currently very [undertested] and [undermaintained], which
24+
is not ideal because we rely on the test suites to check `rustc`'s behavior. The current
25+
implementation in [`compiletest`] is also such that it's very hard and unpleasant to make changes
26+
(e.g. adding new directives) to provide up-to-date test infrastructure support for the needs of
27+
compiler (and rustdoc) contributors. The UX is not great either because of poor error handling and
28+
error reporting.
29+
30+
[undertested]: https://github.com/rust-lang/rust/issues/47606
31+
[undermaintained]: https://github.com/orgs/rust-lang/projects/53
32+
33+
### The status quo
34+
35+
The current status quo is that [`compiletest`] imposes significant friction for compiler (and
36+
rustdoc) contributors who want to run tests and diagnose test failures. [`compiletest`] error
37+
messages are opaque, terse and hard to read. We had to include a separate allow-list of known
38+
directives to detect unknown directives. We still sometimes let malformed directives through and
39+
silently do nothing. Argument splitting is naive and inconsistent. The implementation is very
40+
convoluted. Also there's still insufficient documentation.
41+
42+
See the [tracking issue of various directive handling related bugs][directive-bugs-tracking-issue].
43+
44+
[directive-bugs-tracking-issue]: https://github.com/rust-lang/rust/issues/131425
45+
46+
### The next 6 months
47+
48+
The key changes I want to achieve:
49+
50+
1. Directive handling is **testable** (at all) and in addition have strong test coverage.
51+
2. Directives have **stricter syntax** to reduce ambiguity and enable invalid directive detection or
52+
make invalid directive detection easier.
53+
3. Directives are **well-documented**. Move directive documentation close to directives themselves
54+
and make it possible to be generated alongside tool docs for `compiletest`, so it's less likely
55+
to become outdated and to enable documentation coverage enforcement.
56+
- Also, make sure that we have robust *self* documentation so it's not only one or two
57+
contributors who understands how things work inside `compiletest`...
58+
4. Generally improve directive handling **robustness**. Examples: fixing argument splitting in
59+
`compile-flags`, fix paths related to `aux-build`, etc.
60+
5. Test writers and reviewers can receive **better diagnostics**, for things like a directive is not
61+
accepted in a given test suite or *why* something in `compiletest` failed.
62+
63+
### The "shiny future" we are working towards
64+
65+
My long-term goal for [`compiletest`] is that I want it to make it significantly easier to
66+
maintain. Concretely, this means significantly better test coverage, easier to extend, better
67+
documentation. Hopefully, by being more maintainable, we are able to attract more active maintainers
68+
from both bootstrap and compiler teams and make the code base significantly more pleasant to work
69+
on.
70+
71+
For directive handling *specifically*, it should mean that:
72+
73+
- It's relatively straightforward and low friction to implement new directives, including test
74+
coverage and documentation. It should be easy to do the right thing.
75+
- [`compiletest`] should produce error messages that are easy to read and understand, possibly even
76+
making suggestions.
77+
- Directives should be documented (and enforced to be documented) via rustdoc which are made
78+
available on nightly-rustc docs so we can back-link from dev-guide and not have to maintain two
79+
sets of docs that are mutually inconsistent.
80+
81+
## Ownership and team asks
82+
83+
**Owner:** [@jieyouxu]
84+
85+
<!--
86+
*This section defines the specific work items that are planned and who is expected to do them. It
87+
should also include what will be needed from Rust teams. The table below shows some common sets of
88+
asks and work, but feel free to adjust it as needed. Every row in the table should either correspond
89+
to something done by a contributor or something asked of a team. For items done by a contributor,
90+
list the contributor, or ![Help wanted][] if you don't yet know who will do it. For things asked of
91+
teams, list ![Team][] and the name of the team. The things typically asked of teams are defined in
92+
the [Definitions](#definitions) section below.*
93+
-->
94+
95+
Note that [`compiletest`] is (in theory) currently co-maintained by both t-bootstrap and t-compiler,
96+
but AFAIK is (in practice) currently not really actively maintained by anyone else. The following
97+
team asks are probably mostly [compiler] for feedback on their use cases (as a test infra consumer)
98+
and [bootstrap] for implementation review.
99+
100+
| Subgoal | Owner(s) or team(s) | Notes |
101+
|------------------------------------------------------|------------------------------------------------|------------------------------------------------------------------------------------------------|
102+
| General discussion and moral support | ![Team][] [bootstrap] and ![Team][] [compiler] | |
103+
| Consultations for desired test behaviors | ![Team][] [compiler] and ![Team][] [rustdoc] | Test infra consumers |
104+
| Experimental prototype[^1] | | To see how approaches look like and gain experience/feedback |
105+
| ↳ Discussion and moral support | ![Team][] [bootstrap] and ![Team][] [compiler] | |
106+
| ↳ Implementation | [@jieyouxu] | |
107+
| ↳ Standard reviews | ![Team][] [bootstrap] and ![Team][] [compiler] | Probably mostly [bootstrap] or whoever is more interested in reviewing [`compiletest`] changes |
108+
| [`compiletest`] changes w/ experience from prototype | | |
109+
| ↳ Discussion and moral support | ![Team][] [bootstrap] and ![Team][] [compiler] | |
110+
| ↳ Implementation | [@jieyouxu] | |
111+
| ↳ Standard reviews | ![Team][] [bootstrap] and ![Team][] [compiler] | Probably mostly [bootstrap] or whoever is more interested in reviewing [`compiletest`] changes |
112+
| Inside Rust blog post for project outcome | ![Team][] [bootstrap] and ![Team][] [compiler] | |
113+
114+
[^1]: I want to start with an out-of-tree experimental prototype to see how the pieces are fit
115+
together to make it easier to rapidly iterate and receive feedback without having to mess with
116+
the "live" [`compiletest`] that does not have sufficient test coverage.
117+
118+
### Definitions
119+
120+
Definitions for terms used above:
121+
122+
* *Discussion and moral support* is the lowest level offering, basically committing the team to nothing but good vibes and general support for this endeavor.
123+
* *Author RFC* and *Implementation* means actually writing the code, document, whatever.
124+
* *Design meeting* means holding a synchronous meeting to review a proposal and provide feedback (no decision expected).
125+
* *RFC decisions* means reviewing an RFC and deciding whether to accept.
126+
* *Org decisions* means reaching a decision on an organizational or policy matter.
127+
* *Secondary review* of an RFC means that the team is "tangentially" involved in the RFC and should be expected to briefly review.
128+
* *Stabilizations* means reviewing a stabilization and report and deciding whether to stabilize.
129+
* *Standard reviews* refers to reviews for PRs against the repository; these PRs are not expected to be unduly large or complicated.
130+
* *Prioritized nominations* refers to prioritized lang-team response to nominated issues, with the expectation that there will be *some* response from the next weekly triage meeting.
131+
* *Dedicated review* means identifying an individual (or group of individuals) who will review the changes, as they're expected to require significant context.
132+
* Other kinds of decisions:
133+
* [Lang team experiments](https://lang-team.rust-lang.org/how_to/experiment.html) are used to add nightly features that do not yet have an RFC. They are limited to trusted contributors and are used to resolve design details such that an RFC can be written.
134+
* Compiler [Major Change Proposal (MCP)](https://forge.rust-lang.org/compiler/mcp.html) is used to propose a 'larger than average' change and get feedback from the compiler team.
135+
* Library [API Change Proposal (ACP)](https://std-dev-guide.rust-lang.org/development/feature-lifecycle.html) describes a change to the standard library.
136+
137+
## Frequently asked questions
138+
139+
TODO: pending during project discussions
140+
141+
<!--
142+
### What do I do with this space?
143+
144+
*This is a good place to elaborate on your reasoning above -- for example, why did you put the
145+
design axioms in the order that you did? It's also a good place to put the answers to any questions
146+
that come up during discussion. The expectation is that this FAQ section will grow as the goal is
147+
discussed and eventually should contain a complete summary of the points raised along the way.*
148+
-->
149+
150+
[@jieyouxu]: https://github.com/jieyouxu
151+
[`compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest

0 commit comments

Comments
 (0)