Skip to content

Updates tests from JSON Schema Test Suite #733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stevehu merged 2 commits into from
Apr 30, 2023
Merged

Updates tests from JSON Schema Test Suite #733

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6e05994
Updates tests from JSON Schema Test Suite
Apr 30, 2023
8eafa28
Corrects case-sensitivity issue
Apr 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions src/main/java/com/networknt/schema/uri/URITranslator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;

Expand Down Expand Up @@ -141,14 +140,14 @@ class PrefixReplacer implements URITranslator {
private final String tgt;

public PrefixReplacer(String src, String tgt) {
this.src = src.toLowerCase(Locale.US);
this.src = src;
this.tgt = tgt;
}

@Override
public URI translate(URI original) {
if (null != original) {
String o = original.toASCIIString().toLowerCase(Locale.US);
String o = original.toString();
if (o.startsWith(src)) {
o = tgt + o.substring(src.length());
return URI.create(o);
Expand Down
4 changes: 2 additions & 2 deletions src/test/java/com/networknt/schema/AbstractJsonSchemaTestSuite.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,8 @@ private JsonSchemaFactory buildValidatorFactory(VersionFlag defaultVersion, Test
.builder(base)
.objectMapper(mapper)
.addUriTranslator(URITranslator.combine(
URITranslator.prefix("http://json-schema.org", "resource:"),
URITranslator.prefix("https://json-schema.org", "resource:")
URITranslator.prefix("https://", "http://"),
URITranslator.prefix("http://json-schema.org", "resource:")
))
.build();
}
Expand Down
14 changes: 14 additions & 0 deletions src/test/suite/bin/jsonschema_suite
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -498,6 +498,13 @@ class SanityTests(unittest.TestCase):
"title",
"type",
"uniqueItems",

# Technically this is wrong, $comment doesn't exist in this
# draft, but the point of this test is to detect mistakes by,
# test authors, whereas the point of the $comment keyword is
# to just standardize a place for a comment, so it's not a
# mistake to use it in earlier drafts in tests per se.
"$comment",
},
"draft3": {
"$ref",
Expand Down Expand Up @@ -528,6 +535,13 @@ class SanityTests(unittest.TestCase):
"title",
"type",
"uniqueItems",

# Technically this is wrong, $comment doesn't exist in this
# draft, but the point of this test is to detect mistakes by,
# test authors, whereas the point of the $comment keyword is
# to just standardize a place for a comment, so it's not a
# mistake to use it in earlier drafts in tests per se.
"$comment",
},
}

Expand Down
8 changes: 4 additions & 4 deletions src/test/suite/output-tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,21 @@ These tests are intended to validate that implementations are correctly generati

Output was initially specified with draft 2019-09. It remained largely unchanged for draft 2020-12, but will receive an update with the next release.

***NOTE** Although the formats didn't change between 2019-09 and 2020-12, the tests are replicated for 2020-12 because the `$schema` is different and implementations may (but shouldn't) produce different output.*
_**NOTE** Although the formats didn't change much between 2019-09 and 2020-12, the tests are copied for 2020-12 because the `$schema` is different and implementations may (but shouldn't) produce different output._

## Organization

The tests are organized by specification release and then into two categories: content and structure.

Content tests verify that the keywords are producing the correct annotations and/or error messages. Since there are no requirements on the content of error messages, there's not much that can be verified for them, but it is possible to identify when a error message _could_ be present. Primarily, these tests need to extensively cover the annotation behaviors of each keyword. The only output format needed for these tests is `basic` for 2019-09/2020/12 and `list` for later versions.
Content tests verify that the keywords are producing the correct annotations and/or error messages. Since there are no requirements on the content of error messages, there's not much that can be verified for them, but it is possible to identify when a error message _could_ be present. Primarily, these tests need to extensively cover the annotation behaviors of each keyword. The only output format needed for these tests is `basic` for 2019-09/2020-12 and `list` for later versions.

Structure tests verify that the structures of the various formats (i.e. `flag`, `basic`, `detailed`, `verbose` for 2019/2020 and `flag`, `list`, `hierarchical` for later versions) are correct. These tests don't need to cover each keyword; rather they need to sufficiently cover the various aspects of building the output structures by using whatever keywords are necessary to do so.
Structure tests verify that the structures of the various formats (i.e. `flag`, `basic`, `detailed`, `verbose` for 2019-09/2020-12 and `flag`, `list`, `hierarchical` for later versions) are correct. These tests don't need to cover each keyword; rather they need to sufficiently cover the various aspects of building the output structures by using whatever keywords are necessary to do so.

In each release folder, you'll also find an _output-schema.json_ file that contains the schema from the specification repo that describes output for that release. This schema will need to be loaded as the tests reference it.

## Test Files

The content of a test file is the same as the validation tests in `tests/`, however an `output` property has been added to each test case.
The content of a test file is similar to the validation tests in `tests/`: for each test case, the `valid` property has been removed, and an `output` property has been added.

The `output` property itself has a property for each of the output formats where the value is a schema that will successfully validate for compliant output. For the content tests, only `basic`/`list` needs to be present.

Expand Down
38 changes: 38 additions & 0 deletions src/test/suite/tests/draft-next/dependentSchemas.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,5 +128,43 @@
"valid": false
}
]
},
{
"description": "dependent subschema incompatible with root",
"schema": {
"properties": {
"foo": {}
},
"dependentSchemas": {
"foo": {
"properties": {
"bar": {}
},
"additionalProperties": false
}
}
},
"tests": [
{
"description": "matches root",
"data": {"foo": 1},
"valid": false
},
{
"description": "matches dependency",
"data": {"bar": 1},
"valid": true
},
{
"description": "matches both",
"data": {"foo": 1, "bar": 2},
"valid": false
},
{
"description": "no dependency",
"data": {"baz": 1},
"valid": true
}
]
}
]
95 changes: 95 additions & 0 deletions src/test/suite/tests/draft-next/format.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid email string is only an annotation by default",
"data": "2962",
"valid": true
}
]
},
Expand Down Expand Up @@ -74,6 +79,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid idn-email string is only an annotation by default",
"data": "2962",
"valid": true
}
]
},
Expand Down Expand Up @@ -113,6 +123,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid regex string is only an annotation by default",
"data": "^(abc]",
"valid": true
}
]
},
Expand Down Expand Up @@ -152,6 +167,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid ipv4 string is only an annotation by default",
"data": "127.0.0.0.1",
"valid": true
}
]
},
Expand Down Expand Up @@ -191,6 +211,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid ipv6 string is only an annotation by default",
"data": "12345::",
"valid": true
}
]
},
Expand Down Expand Up @@ -230,6 +255,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid idn-hostname string is only an annotation by default",
"data": "〮실례.테스트",
"valid": true
}
]
},
Expand Down Expand Up @@ -269,6 +299,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid hostname string is only an annotation by default",
"data": "-a-host-name-that-starts-with--",
"valid": true
}
]
},
Expand Down Expand Up @@ -308,6 +343,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid date string is only an annotation by default",
"data": "06/19/1963",
"valid": true
}
]
},
Expand Down Expand Up @@ -347,6 +387,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid date-time string is only an annotation by default",
"data": "1990-02-31T15:59:60.123-08:00",
"valid": true
}
]
},
Expand Down Expand Up @@ -386,6 +431,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid time string is only an annotation by default",
"data": "08:30:06 PST",
"valid": true
}
]
},
Expand Down Expand Up @@ -425,6 +475,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid json-pointer string is only an annotation by default",
"data": "/foo/bar~",
"valid": true
}
]
},
Expand Down Expand Up @@ -464,6 +519,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid relative-json-pointer string is only an annotation by default",
"data": "/foo/bar",
"valid": true
}
]
},
Expand Down Expand Up @@ -503,6 +563,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid iri string is only an annotation by default",
"data": "http://2001:0db8:85a3:0000:0000:8a2e:0370:7334",
"valid": true
}
]
},
Expand Down Expand Up @@ -542,6 +607,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid iri-reference string is only an annotation by default",
"data": "\\\\WINDOWS\\filëßåré",
"valid": true
}
]
},
Expand Down Expand Up @@ -581,6 +651,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid uri string is only an annotation by default",
"data": "//foo.bar/?baz=qux#quux",
"valid": true
}
]
},
Expand Down Expand Up @@ -620,6 +695,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid uri-reference string is only an annotation by default",
"data": "\\\\WINDOWS\\fileshare",
"valid": true
}
]
},
Expand Down Expand Up @@ -659,6 +739,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid uri-template string is only an annotation by default",
"data": "http://example.com/dictionary/{term:1}/{term",
"valid": true
}
]
},
Expand Down Expand Up @@ -698,6 +783,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid uuid string is only an annotation by default",
"data": "2eb8aa08-aa98-11ea-b4aa-73b441d1638",
"valid": true
}
]
},
Expand Down Expand Up @@ -737,6 +827,11 @@
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "invalid duration string is only an annotation by default",
"data": "PT1D",
"valid": true
}
]
}
Expand Down
10 changes: 5 additions & 5 deletions src/test/suite/tests/draft-next/id.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,32 +217,32 @@
"id_in_enum": {
"enum": [
{
"$id": "http://localhost:1234/draft-next/id/my_identifier.json",
"$id": "https://localhost:1234/draft-next/id/my_identifier.json",
"type": "null"
}
]
},
"real_id_in_schema": {
"$id": "http://localhost:1234/draft-next/id/my_identifier.json",
"$id": "https://localhost:1234/draft-next/id/my_identifier.json",
"type": "string"
},
"zzz_id_in_const": {
"const": {
"$id": "http://localhost:1234/draft-next/id/my_identifier.json",
"$id": "https://localhost:1234/draft-next/id/my_identifier.json",
"type": "null"
}
}
},
"anyOf": [
{ "$ref": "#/$defs/id_in_enum" },
{ "$ref": "http://localhost:1234/draft-next/id/my_identifier.json" }
{ "$ref": "https://localhost:1234/draft-next/id/my_identifier.json" }
]
},
"tests": [
{
"description": "exact match to enum, and type matches",
"data": {
"$id": "http://localhost:1234/draft-next/id/my_identifier.json",
"$id": "https://localhost:1234/draft-next/id/my_identifier.json",
"type": "null"
},
"valid": true
Expand Down
Loading