Skip to content

Commit e4808b8

Browse files
jmikolajmikola
authored
DRIVERS-709: Unified Test Format (#846)
* DRIVERS-709 wip * consolidate test format, simplify assert syntax, docs for matching * bulkWrite errors, schemaVersion, and configureFailPoint APM exclusion * Revise schema version and move allowMultipleMongoses to top-level * Define terms and start test execution steps * Readability improvements and TODO items * Suggest YAML nodes and anchors for collection/database names * Fix link text and version formatting * indent Server Fail Points and describe process for sharded-replicaset * Test implementation and various other clarifications * Language edits from PR feedback * Require explicit database and collection names and add extra notes * observeEvents and useMultipleMongoses client entity options * Example insertOne test * special operations for sessions spec tests * open questions * open question for representing options in operation.arguments * Rename schema and test file to convey they are outdated * clarify schema version comparisons and compatibility * lsid assertions require observed commands with lsid fields * Rename topologies link related spec tickets * top-level description field * change stream iteration, open questions, and errorCode assertion * Future Work, createchangeStream operation, and review fixes * Update last modified and advisors * Fix typos * Changes for events, GridFS, and error assertions Close existing Open Questions and add new Future Work sections. * runOnRequirements, expectedEventsForClient, refer to YAML objects * Sync JSON file with YAML to add description field * Require arrays, rename "expect" prefix, clarify "sharded" requirement * Remove note about clearing state between test files * Note that server versions are compared numerically * Fix typo * Clarify type of observeEvents * Require BSON types be expressed as strings for $$type operator * Clarify test.outcome and comparison rules * Make formatting of "primary" RP consistent with WC and RC * Add Goals, clarify schemaVersion, and Future Work entries * Elaborate on schema version * Clarify (un)supported entity types * Allow equivalent language types for BSON types in entity map * Raise errors for unsupported operations and arguments * Fix link syntax * Feedback from Oleg * $$sessionLsid error handling and heartbeatFrequencyMS tweaks * Clarify BSON type support and advise against using deprecated types * Clarify server version string comparison rules * Revise version docs, MUST reset state, explain SHOULD in Design Rationale * Improve Evaluating Matches docs * Test runners MUST raise an error for incompatible files * Future work for operations/arguments and schema version * Prohibit $$unsetOrMatches for array elements * Optional arrays must be non-empty, revise expectedEvent and $$matchesHexBytes * Update schema for current spec syntax * Update Makefile for new schema * expectedError.isError can only be true * Add invalid test files for testing the schema itself * Prohibit additionalProperties in test objects * Requirements for model objects and iterables and clarify failPoint Also restructures Entity Test Operations sections * Fix RST syntax and typos * Specify descriptions for invalid schema tests * Initial valid-fail and valid-pass tests * Clarify iteration rules * Don't mix testRunner ops with error/result expectations * Additional POC tests * Clarifications for CRUD operations * Advise using $$unsetOrMatches for InsertOneResult * POC for session tests * POC for transaction tests * Fix insertOne syntax * Use complete fullDocument assertions in change stream tests * Clarify rules for enabling event listeners * Fix event assertions for ChangeStream tests * Fix syntax of callback argument for withTransaction tests * Remove stream entity and add proxy operations for GridFS * Reorganized entity operation content and advise not using watch * Fix retryable reads test and add additional find test * Fix retryable writes test to use correct collection object * Fix syntax errors in session tests * Fix typos in transaction tests * Specify useMultipleMongoses:false for failPoint tests * Fix outcome assertion in session pinning test * Add related issue and iterateUntilDocumentOrError clarification * Rename returnDocument test and fix syntax error * Require sharded-replicaset for retryable write session test * Fix typo * Remove "Related Issues" section and clear "Change Log" * Fix schema validation error in returnDocument-enum-invalid.yml * Generate JSON for YAML tests * Add schema checks to Travis CI
1 parent c378d48 commit e4808b8

File tree

328 files changed

+14381
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

328 files changed

+14381
-0
lines changed

‎.travis.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,13 @@ matrix:
2020
script:
2121
- python3 ./source/server-discovery-and-monitoring/tests/errors/generate-error-tests.py
2222
- cd source && make && git diff --exit-code
23+
24+
- name: "Unifed Test Format schema checks"
25+
dist: xenial
26+
language: python
27+
python:
28+
- 3.7
29+
install:
30+
- npm install -g ajv-cli
31+
script:
32+
- cd source/unified-test-format/tests && make

‎source/unified-test-format/schema-1.0.json

Lines changed: 286 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,286 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
4+
"title": "Unified Test Format",
5+
"type": "object",
6+
"additionalProperties": false,
7+
"required": ["description", "schemaVersion", "tests"],
8+
"properties": {
9+
"description": { "type": "string" },
10+
"schemaVersion": { "$ref": "#/definitions/version" },
11+
"runOnRequirements": {
12+
"type": "array",
13+
"minItems": 1,
14+
"items": { "$ref": "#/definitions/runOnRequirement" }
15+
},
16+
"createEntities": {
17+
"type": "array",
18+
"minItems": 1,
19+
"items": { "$ref": "#/definitions/entity" }
20+
},
21+
"initialData": {
22+
"type": "array",
23+
"minItems": 1,
24+
"items": { "$ref": "#/definitions/collectionData" }
25+
},
26+
"tests": {
27+
"type": "array",
28+
"minItems": 1,
29+
"items": { "$ref": "#/definitions/test" }
30+
}
31+
},
32+
33+
"definitions": {
34+
"version": {
35+
"type": "string",
36+
"pattern": "^[0-9]+(\\.[0-9]+){1,2}$"
37+
},
38+
39+
"runOnRequirement": {
40+
"type": "object",
41+
"additionalProperties": false,
42+
"minProperties": 1,
43+
"properties": {
44+
"maxServerVersion": { "$ref": "#/definitions/version" },
45+
"minServerVersion": { "$ref": "#/definitions/version" },
46+
"topologies": {
47+
"type": "array",
48+
"minItems": 1,
49+
"items": {
50+
"type": "string",
51+
"enum": ["single", "replicaset", "sharded", "sharded-replicaset"]
52+
}
53+
}
54+
}
55+
},
56+
57+
"entity": {
58+
"type": "object",
59+
"additionalProperties": false,
60+
"maxProperties": 1,
61+
"minProperties": 1,
62+
"properties": {
63+
"client": {
64+
"type": "object",
65+
"additionalProperties": false,
66+
"required": ["id"],
67+
"properties": {
68+
"id": { "type": "string" },
69+
"uriOptions": { "type": "object" },
70+
"useMultipleMongoses": { "type": "boolean" },
71+
"observeEvents": {
72+
"type": "array",
73+
"minItems": 1,
74+
"items": {
75+
"type": "string",
76+
"enum": ["commandStartedEvent", "commandSucceededEvent", "commandFailedEvent"]
77+
}
78+
},
79+
"ignoreCommandMonitoringEvents": {
80+
"type": "array",
81+
"minItems": 1,
82+
"items": { "type": "string" }
83+
}
84+
}
85+
},
86+
"database": {
87+
"type": "object",
88+
"additionalProperties": false,
89+
"required": ["id", "client", "databaseName"],
90+
"properties": {
91+
"id": { "type": "string" },
92+
"client": { "type": "string" },
93+
"databaseName": { "type": "string" },
94+
"databaseOptions": { "$ref": "#/definitions/collectionOrDatabaseOptions" }
95+
}
96+
},
97+
"collection": {
98+
"type": "object",
99+
"additionalProperties": false,
100+
"required": ["id", "database", "collectionName"],
101+
"properties": {
102+
"id": { "type": "string" },
103+
"database": { "type": "string" },
104+
"collectionName": { "type": "string" },
105+
"collectionOptions": { "$ref": "#/definitions/collectionOrDatabaseOptions" }
106+
}
107+
},
108+
"session": {
109+
"type": "object",
110+
"additionalProperties": false,
111+
"required": ["id", "client"],
112+
"properties": {
113+
"id": { "type": "string" },
114+
"client": { "type": "string" },
115+
"sessionOptions": { "type": "object" }
116+
}
117+
},
118+
"bucket": {
119+
"type": "object",
120+
"additionalProperties": false,
121+
"required": ["id", "database"],
122+
"properties": {
123+
"id": { "type": "string" },
124+
"database": { "type": "string" },
125+
"bucketOptions": { "type": "object" }
126+
}
127+
},
128+
"stream": {
129+
"type": "object",
130+
"additionalProperties": false,
131+
"required": ["id", "hexBytes"],
132+
"properties": {
133+
"id": { "type": "string" },
134+
"hexBytes": {
135+
"type": "string",
136+
"pattern": "^([0-9A-Fa-f]{2})*$"
137+
}
138+
}
139+
}
140+
}
141+
},
142+
143+
"collectionData": {
144+
"type": "object",
145+
"additionalProperties": false,
146+
"required": ["collectionName", "databaseName", "documents"],
147+
"properties": {
148+
"collectionName": { "type": "string" },
149+
"databaseName": { "type": "string" },
150+
"documents": {
151+
"type": "array",
152+
"items": { "type": "object" }
153+
}
154+
}
155+
},
156+
157+
"expectedEventsForClient": {
158+
"type": "object",
159+
"additionalProperties": false,
160+
"required": ["client", "events"],
161+
"properties": {
162+
"client": { "type": "string" },
163+
"events": {
164+
"type": "array",
165+
"items": { "$ref": "#/definitions/expectedEvent" }
166+
}
167+
}
168+
},
169+
170+
"expectedEvent": {
171+
"type": "object",
172+
"additionalProperties": false,
173+
"maxProperties": 1,
174+
"minProperties": 1,
175+
"properties": {
176+
"commandStartedEvent": {
177+
"type": "object",
178+
"additionalProperties": false,
179+
"properties": {
180+
"command": { "type": "object" },
181+
"commandName": { "type": "string" },
182+
"databaseName": { "type": "string" }
183+
}
184+
},
185+
"commandSucceededEvent": {
186+
"type": "object",
187+
"additionalProperties": false,
188+
"properties": {
189+
"reply": { "type": "object" },
190+
"commandName": { "type": "string" }
191+
}
192+
},
193+
"commandFailedEvent": {
194+
"type": "object",
195+
"additionalProperties": false,
196+
"properties": {
197+
"commandName": { "type": "string" }
198+
}
199+
}
200+
}
201+
},
202+
203+
"collectionOrDatabaseOptions": {
204+
"type": "object",
205+
"additionalProperties": false,
206+
"properties": {
207+
"readConcern": { "type": "object" },
208+
"readPreference": { "type": "object" },
209+
"writeConcern": { "type": "object" }
210+
}
211+
},
212+
213+
"operation": {
214+
"type": "object",
215+
"additionalProperties": false,
216+
"required": ["name", "object"],
217+
"properties": {
218+
"name": { "type": "string" },
219+
"object": { "type": "string" },
220+
"arguments": { "type": "object" },
221+
"expectError": { "$ref": "#/definitions/expectedError" },
222+
"expectResult": {},
223+
"saveResultAsEntity": { "type": "string" }
224+
},
225+
"allOf": [
226+
{ "not": { "required": ["expectError", "expectResult"] } },
227+
{ "not": { "required": ["expectError", "saveResultAsEntity"] } }
228+
]
229+
},
230+
231+
"expectedError": {
232+
"type": "object",
233+
"additionalProperties": false,
234+
"minProperties": 1,
235+
"properties": {
236+
"isError": {
237+
"type": "boolean",
238+
"const": true
239+
},
240+
"isClientError": { "type": "boolean" },
241+
"errorContains": { "type": "string" },
242+
"errorCode": { "type": "integer" },
243+
"errorCodeName": { "type": "string" },
244+
"errorLabelsContain": {
245+
"type": "array",
246+
"minItems": 1,
247+
"items": { "type": "string" }
248+
},
249+
"errorLabelsOmit": {
250+
"type": "array",
251+
"minItems": 1,
252+
"items": { "type": "string" }
253+
},
254+
"expectResult": {}
255+
}
256+
},
257+
258+
"test": {
259+
"type": "object",
260+
"additionalProperties": false,
261+
"required": ["description", "operations"],
262+
"properties": {
263+
"description": { "type": "string" },
264+
"runOnRequirements": {
265+
"type": "array",
266+
"minItems": 1,
267+
"items": { "$ref": "#/definitions/runOnRequirement" }
268+
},
269+
"skipReason": { "type": "string" },
270+
"operations": {
271+
"type": "array",
272+
"items": { "$ref": "#/definitions/operation" }
273+
},
274+
"expectEvents": {
275+
"type": "array",
276+
"items": { "$ref": "#/definitions/expectedEventsForClient" }
277+
},
278+
"outcome": {
279+
"type": "array",
280+
"minItems": 1,
281+
"items": { "$ref": "#/definitions/collectionData" }
282+
}
283+
}
284+
}
285+
}
286+
}

‎source/unified-test-format/schema-testing-example.json

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
{
2+
"tests": [
3+
{
4+
"description": "valid because error:true and result has only expected fields",
5+
"operations": [
6+
{
7+
"object": "collection",
8+
"name": "runCommand",
9+
"error": true,
10+
"result": {
11+
"errorContains": "string"
12+
}
13+
}
14+
]
15+
},
16+
{
17+
"description": "valid because error == false. result can be any type (non-object)",
18+
"operations": [
19+
{
20+
"object": "collection",
21+
"name": "operation",
22+
"error": false,
23+
"result": 1
24+
}
25+
]
26+
},
27+
{
28+
"description": "valid because error == false. result can be any type (object)",
29+
"operations": [
30+
{
31+
"object": "collection",
32+
"name": "operation",
33+
"error": false,
34+
"result": {
35+
"foo": "bar"
36+
}
37+
}
38+
]
39+
},
40+
{
41+
"description": "valid because error != true. result can be any type (non-object)",
42+
"operations": [
43+
{
44+
"object": "collection",
45+
"name": "operation",
46+
"result": 1
47+
}
48+
]
49+
},
50+
{
51+
"description": "valid because error != true. result can be any type (object)",
52+
"operations": [
53+
{
54+
"object": "collection",
55+
"name": "operation",
56+
"result": {
57+
"foo": "bar"
58+
}
59+
}
60+
]
61+
}
62+
]
63+
}

‎source/unified-test-format/tests/Makefile

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
SCHEMA=../schema-1.0.json
2+
3+
.PHONY: all invalid valid-fail valid-pass HAS_AJV
4+
5+
all: invalid valid-fail valid-pass
6+
7+
invalid: HAS_AJV
8+
@# Redirect stdout to hide expected validation errors
9+
@ajv test -s $(SCHEMA) -d "invalid/*.yml" --invalid > /dev/null && echo "invalid/*.yml passed test"
10+
11+
valid-fail: HAS_AJV
12+
@ajv test -s $(SCHEMA) -d "valid-fail/*.yml" --valid
13+
14+
valid-pass: HAS_AJV
15+
@ajv test -s $(SCHEMA) -d "valid-pass/*.yml" --valid
16+
17+
HAS_AJV:
18+
@if ! command -v ajv > /dev/null; then \
19+
echo 'Error: need "npm install -g ajv-cli"' 1>&2; \
20+
exit 1; \
21+
fi

0 commit comments

Comments
 (0)