-
-
Notifications
You must be signed in to change notification settings - Fork 218
Add annotation tests from @hyperjump/json-schema #770
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
0aef5f8
55b9fad
3426e76
9f90e2d
2779c99
35ccff6
265103e
ec05504
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
name: Validate annotation tests | ||
|
||
on: | ||
pull_request: | ||
paths: | ||
- "annotations/**" | ||
|
||
jobs: | ||
annotate: | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- uses: actions/checkout@v4 | ||
|
||
- name: Set up Deno | ||
uses: denoland/setup-deno@v2 | ||
with: | ||
deno-version: "2.x" | ||
|
||
- name: Validate annotation tests | ||
run: deno --node-modules-dir=auto --allow-read --no-prompt bin/annotation-tests.ts |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,116 @@ | ||
# Annotations Tests Suite | ||
|
||
The Annotations Test Suite tests which annotations should appear (or not appear) | ||
on which values of an instance. These tests are agnostic of any output format. | ||
|
||
## Supported Dialects | ||
|
||
Although the annotation terminology of didn't appear in the spec until 2019-09, | ||
the concept is compatible with every version of JSON Schema. Test Cases in this | ||
Test Suite are designed to be compatible with as many releases of JSON Schema as | ||
possible. They do not include `$schema` or `$id`/`id` keywords so | ||
implementations can run the same Test Suite for each dialect they support. | ||
|
||
Since this Test Suite can be used for a variety of dialects, there are a couple | ||
of options that can be used by Test Runners to filter out Test Cases that don't | ||
apply to the dialect under test. | ||
|
||
## Test Case Components | ||
|
||
### description | ||
|
||
A short description of what behavior the Test Case is covering. | ||
|
||
### compatibility | ||
|
||
The `compatibility` option allows you to set which dialects the Test Case is | ||
compatible with. Test Runners can use this value to filter out Test Cases that | ||
don't apply the to dialect currently under test. The terminology for annotations | ||
didn't appear in the spec until 2019-09, but the concept is compatible with | ||
older releases as well. When setting `compatibility`, test authors should take | ||
into account dialects before 2019-09 for implementations that chose to support | ||
annotations for older dialects. | ||
|
||
Dialects are indicated by the number corresponding to their release. Date-based | ||
releases use just the year. If this option isn't present, it means the Test Case | ||
is compatible with any dialect. | ||
|
||
If this option is present with a number, the number indicates the minimum | ||
release the Test Case is compatible with. This example indicates that the Test | ||
Case is compatible with draft-07 and up. | ||
|
||
**Example**: `"compatibility": "7"` | ||
|
||
You can use a `<=` operator to indicate that the Test Case is compatible with | ||
releases less then or equal to the given release. This example indicates that | ||
the Test Case is compatible with 2019-09 and under. | ||
|
||
**Example**: `"compatibility": "<=2019"` | ||
|
||
You can use comma-separated values to indicate multiple constraints if needed. | ||
This example indicates that the Test Case is compatible with releases between | ||
draft-06 and 2019-09. | ||
|
||
**Example**: `"compatibility": "6,<=2019"` | ||
|
||
For convenience, you can use the `=` operator to indicate a Test Case is only | ||
compatible with a single release. This example indicates that the Test Case is | ||
compatible only with 2020-12. | ||
|
||
**Example**: `"compatibility": "=2020"` | ||
|
||
### schema | ||
|
||
The schema that will serve as the subject for the tests. Whenever possible, this | ||
schema shouldn't include `$schema` or `id`/`$id` because Test Cases should be | ||
designed to work with as many releases as possible. | ||
|
||
### externalSchemas | ||
|
||
This allows you to define additional schemas that `schema` makes references to. | ||
The value is an object where the keys are retrieval URIs and values are schemas. | ||
Most external schemas aren't self identifying (using `id`/`$id`) and rely on the | ||
retrieval URI for identification. This is done to increase the number of | ||
dialects that the test is compatible with. Because `id` changed to `$id` in | ||
draft-06, if you use `$id`, the test becomes incompatible with draft-03/4 and in | ||
most cases, that's not necessary. | ||
|
||
### tests | ||
|
||
A collection of Tests to run to verify the Test Case. | ||
|
||
## Test Components | ||
|
||
### instance | ||
|
||
The JSON instance to be annotated. | ||
|
||
### assertions | ||
|
||
A collection of assertions that must be true for the test to pass. | ||
|
||
## Assertions Components | ||
|
||
### location | ||
|
||
The instance location. | ||
|
||
### keyword | ||
|
||
The annotating keyword. | ||
|
||
### expected | ||
|
||
A collection of `keyword` annotations expected on the instance at `location`. | ||
`expected` is an object where the keys are schema locations and the values are | ||
the annotation that schema location contributed for the given `keyword`. | ||
|
||
There can be more than one expected annotation because multiple schema locations | ||
could contribute annotations for a single keyword. | ||
|
||
An empty object is an assertion that the annotation must not appear at the | ||
`location` for the `keyword`. | ||
|
||
As a convention for this Test Suite, the `expected` array should be sorted such | ||
that the most recently encountered value for an annotation given top-down | ||
evaluation of the schema comes before previously encountered values. | ||
Comment on lines
+114
to
+116
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure this can be enforced. (I see you do have "as a convention".) If a schema uses `properties: {foo: ..., bar: ...}, who's to say which property is evaluated first? As a result, the annotations can be produced in any order. If this is just for the sake of test organization, then maybe the request needs some more clarity (e.g. "sorted alphabetically by location pointer" or something) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is about the Implementations can evaluate keywords in any order they want. If everyone who contributes tests with Maybe I'm the only one who thinks maintaining this consistency has value. The vast majority of tests will only have one expected value per assertion, so maybe it won't even come up enough to be an issue anyway. In any case, it will be up to us to enforce it or not in the review process. From the perspective of test harness implementations, the order shouldn't change the pass/fail result for a test. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
{ | ||
"$schema": "http://json-schema.org/draft-07/schema#", | ||
|
||
"type": "object", | ||
"properties": { | ||
"location": { | ||
"markdownDescription": "The instance location.", | ||
"type": "string", | ||
"format": "json-pointer" | ||
}, | ||
"keyword": { | ||
"markdownDescription": "The annotation keyword.", | ||
"type": "string" | ||
}, | ||
"expected": { | ||
"markdownDescription": "An object of schemaLocation/annotations pairs for `keyword` annotations expected on the instance at `location`.", | ||
"type": "object", | ||
"propertyNames": { | ||
"format": "uri" | ||
} | ||
} | ||
}, | ||
"required": ["location", "keyword", "expected"] | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
{ | ||
"$schema": "http://json-schema.org/draft-07/schema#", | ||
|
||
"type": "object", | ||
"properties": { | ||
"description": { | ||
"markdownDescription": "A short description of what behavior the Test Case is covering.", | ||
"type": "string" | ||
}, | ||
"compatibility": { | ||
"markdownDescription": "Set which dialects the Test Case is compatible with. Examples:\n- `\"7\"` -- draft-07 and above\n- `\"<=2019\"` -- 2019-09 and previous\n- `\"6,<=2019\"` -- Between draft-06 and 2019-09\n- `\"=2020\"` -- 2020-12 only", | ||
"type": "string", | ||
"pattern": "^(<=|=)?([123467]|2019|2020)(,(<=|=)?([123467]|2019|2020))*$" | ||
}, | ||
"schema": { | ||
"markdownDescription": "This schema shouldn't include `$schema` or `id`/`$id` unless necesary for the test because Test Cases should be designed to work with as many releases as possible.", | ||
"type": ["boolean", "object"] | ||
}, | ||
"externalSchemas": { | ||
"markdownDescription": "The keys are retrieval URIs and values are schemas.", | ||
"type": "object", | ||
"patternProperties": { | ||
"": { | ||
"type": ["boolean", "object"] | ||
} | ||
}, | ||
"propertyNames": { | ||
"format": "uri" | ||
} | ||
}, | ||
"tests": { | ||
"markdownDescription": "A collection of Tests to run to verify the Test Case.", | ||
"type": "array", | ||
"items": { "$ref": "./test.schema.json" } | ||
} | ||
}, | ||
"required": ["description", "schema", "tests"] | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
{ | ||
"$schema": "http://json-schema.org/draft-07/schema#", | ||
|
||
"type": "object", | ||
"properties": { | ||
"description": { | ||
"type": "string" | ||
}, | ||
"suite": { | ||
"type": "array", | ||
"items": { "$ref": "./test-case.schema.json" } | ||
} | ||
}, | ||
"required": ["description", "suite"] | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
{ | ||
"$schema": "http://json-schema.org/draft-07/schema#", | ||
|
||
"type": "object", | ||
"properties": { | ||
"instance": { | ||
"markdownDescription": "The JSON instance to be annotated." | ||
}, | ||
"assertions": { | ||
"markdownDescription": "A collection of assertions that must be true for the test to pass.", | ||
"type": "array", | ||
"items": { "$ref": "./assertion.schema.json" } | ||
} | ||
}, | ||
"required": ["instance", "assertions"] | ||
} |
Uh oh!
There was an error while loading. Please reload this page.