Skip to content

Commit e5c1c3b

Browse files
committed
Add guidance for fixing spec test failures in dependabot PRs
1 parent c8c971e commit e5c1c3b

File tree

1 file changed

+29
-0
lines changed

1 file changed

+29
-0
lines changed

CONTRIBUTING.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,35 @@ The following environment variables are used for [CSFLE testing](https://github.
101101
* `KMS_TLS_CA_FILE`
102102
* `KMS_TLS_CERTIFICATE_KEY_FILE`
103103

104+
### Updating spec tests
105+
106+
Tests from the MongoDB Specifications repository are included through a
107+
submodule and updated automatically through Dependabot. To update tests
108+
manually, switch to the `tests/specifications` directory and update the
109+
repository to the appropriate commit. Remember to commit this change to the
110+
library repository.
111+
112+
#### Handling test failures on updates
113+
114+
Failures on updates can occur for multiple reasons, and the remedy to this will
115+
depend on the type of failure. Note that only tests for implemented
116+
specifications are run in the test runner.
117+
118+
* If a specification is not fully implemented (e.g. a recent change to the spec
119+
has not been applied yet), skip the test in question with a reference to the
120+
ticket that covers the change
121+
* If a test fails because it uses features not yet implemented in the unified
122+
test runner, skip the corresponding test with a reference to the ticket that
123+
covers implementing the new features
124+
* If the test failure points to a bug in the spec, consider the effort required
125+
to fix the failure. If it's a small change, commit and push the fix directly
126+
to the pull request. Otherwise, skip the test with a reference to a ticket to
127+
fix the failing test.
128+
129+
The goal is that the library passes tests with the latest spec version at all
130+
times, either by implementing small changes quickly, or by skipping tests as
131+
necessary.
132+
104133
## Code quality
105134

106135
Before submitting a pull request, please ensure that your code adheres to the

0 commit comments

Comments
 (0)