From 3b538fff3b2a8219a52ef6b19c9d425b0520b4ca Mon Sep 17 00:00:00 2001 From: Henry Andrews Date: Sun, 16 Sep 2018 17:51:57 -0700 Subject: [PATCH 1/2] $recursiveRef and $recursiveAnchor These keywords allow for recursive extension of schemas, in the way needed by meta-schemas. This is not Object-Oriented extension, rather it is a shortcut for the kind of "allOf" combination that is already allowed, but without the need to redeclare all recursive keywords in the extension. It works by both the original target of the reference and the dynamically chosen target of the reference opting in to the dynamic behavior with "$recursiveAnchor", and the source of the reference opting in by using "$recursiveRef". This ensures that regular "$ref" usage is unchanged and and predictable, and also that schemas not intended for recursive extension cannot be extended without being changed. --- jsonschema-core.xml | 245 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 222 insertions(+), 23 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index 6f0ae9bb..ffa325ce 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -1006,35 +1006,234 @@ -
+
- The "$ref" keyword can be used to reference a schema which is to be applied to the - current instance location. "$ref" is an applicator key word, applying the referred - schema to the instance. + Several keywords can be used to reference a schema which is to be applied to the + current instance location. "$ref" and "$recursiveRef" are an applicator + keywords, applying the referred schema to the instance. "$recursiveAnchor" + is a helper keyword that controls how the referred schema of "$recursiveRef" + is determined. - The value of the "$ref" property MUST be a string which is a URI Reference. - Resolved against the current URI base, it identifies the URI of a schema to use. + As the value of "$ref" and "$recursiveRef" are URI References, this allows + the possibility to externalise or divide a schema across multiple files, + and provides the ability to validate recursive structures through + self-reference. - As the value of "$ref" is a URI Reference, this allows the possibility to externalise or - divide a schema across multiple files, and provides the ability to validate recursive structures - through self-reference. - - - The URI is not a network locator, only an identifier. A schema need not be - downloadable from the address if it is a network-addressable URL, and - implementations SHOULD NOT assume they should perform a network operation when they - encounter a network-addressable URI. - - - A schema MUST NOT be run into an infinite loop against a schema. For example, if two - schemas "#alice" and "#bob" both have an "allOf" property that refers to the other, - a naive validator might get stuck in an infinite recursive loop trying to validate - the instance. - Schemas SHOULD NOT make use of infinite recursive nesting like this; the behavior is - undefined. + The resolved URI produced by these keywords is not necessarily a network + locator, only an identifier. A schema need not be downloadable from the + address if it is a network-addressable URL, and implementations SHOULD NOT + assume they should perform a network operation when they encounter + a network-addressable URI. + +
+ + The "$ref" keyword is used to reference a statically identified schema. + + + The value of the "$ref" property MUST be a string which is a URI Reference. + Resolved against the current URI base, it identifies the URI of a schema + to use. + +
+ +
+ + The "$recursiveRef" and "$recursiveAnchor" keywords are used to construct + extensible recursive schemas. + + + Intuitively, when using "$ref" or another + similar keyword to combine a recursive schema with another schema (recursive + or otherwise), the goal of the schema author is often to have the + recursion respect that combination. The recursive reference would + ideally always recurse to where the processing of the schema started. + + + But this is not possible with the static behavior of "$ref", wich can + only refer to the root schema of the current schema document. + More accurately, it can only refer to one location, and that location + is constrained by the static rules for resolving URI References. + + + This constraint leads to verbose and error-prone re-definitions of each + recursive element, as can be seen in the meta-scheme for JSON Hyper-Schema + in all prior drafts. + +
+ + Consider the following two schemas. The first (given the id "basic") + is an object with one string property and one reference property. + The reference is recursive, pointing to the root of the current + schema document. The second schema references the first, and + also describes a "things" property, which is an array of + recursive references. + + + + + + The problem is that the referred targets of the + "$ref": "#" + references are statically determined. Since the + "things" array is in the + combined schema, its referred schema is the combined + schema. But the "recursive" + property in the basic schema still points to the root + of that basic schema, and therefore will not see the + description of the "things" + property. What we want is for it to resolve + to the combined schema as well. + +
+
+ + Since the desired behavior can seem surprising, and unpredictable, + it is important to use keywords to explicitly control all aspects + of the behavior. In order to create a recursive reference, we + must do three things: + + + In our "basic" schema, indicate that the schema author + intends for it to be extensible recursively. + + + In our "extension" schema, indicate that it is intended + to be a recursive extension. + + + Use a reference keyword that explicitly activates the + recursive behavior at the point of reference. + + + These three things together ensure that all schema authors + are intentionally constructing a recursive extension, which in + turn gives all uses of the regular "$ref" keyword confidence + that it only behaves as it appears to, using lexical scoping. + + + The "$recursiveAnchor" keyword is how schema authors indicate + that a schema can be extended recursively, and be a recursive + schema. This keyword MAY appear in the root schema of a + schema document, and MUST NOT appear in a subschema. + + + The value of "$recursiveAnchor" MUST be of type boolean, and + MUST be true. The value false is reserved for possible future use. + + + The "$recursiveRef" keyword behaves identically to "$ref", except + that if the referred schema has "$recursiveAnchor" set to true, + then the implementation MUST check the dyanamic scope to see + if "$recursiveAnchor" had previously been set. If so, then the + referred schema is considered to be the outermost (in terms of + dynamic scope) schema object containing "$recursiveAnchor" set to true. + + + Note that if the schema to which "$recursiveRef" referrs does not + contain "$recursiveAnchor" set to true, or if there are no other + "$recursiveAnchor" keywords set to true anywhere further back in + the dynamic scope, then "$recursiveRef"'s behavior is identical + to that of "$ref". + +
+ + With this in mind, we can rewrite the previous example: + + + + + + Now lets consider the evaluation of the "extension" schema. + Note that the "$ref": "basic" was not changed, as it works + just fine as a normals static reference. And the + "$recursiveRef" in the "extended" schema does not behave at + all differently, because the "$recursiveAnchor" in its + referred schema is the outermost "$recursiveAnchor" in the + dynamic scope. However, the "$recursiveRef" in the "basic" + schema referrs to a "$recursiveAnchor" that is not the + outermost such keyword in the dynamic scope. That is still + the "$recursiveAnchor" in the "extension" schema. + Therefore, when processing starts with the extension + schema, the "$recursiveRef" in the basic schema actually + referrs to the "extension" schema's root schema. + +
+
+
+ +
+ + A schema MUST NOT be run into an infinite loop against an instance. For + example, if two schemas "#alice" and "#bob" both have an "allOf" property + that refers to the other, a naive validator might get stuck in an infinite + recursive loop trying to validate the instance. Schemas SHOULD NOT make + use of infinite recursive nesting like this; the behavior is undefined. + +
+
The use of URIs to identify remote schemas does not necessarily mean anything is downloaded, From 062b037973fb95ade51688c44c6d2adc075bd587 Mon Sep 17 00:00:00 2001 From: Henry Andrews Date: Mon, 5 Nov 2018 15:28:50 -0800 Subject: [PATCH 2/2] Rework $recursiveRef based on feedback. Hopefully this version of the description and example are easier to follow. Also replaced "reffered" and "referring" with "referenced" and "referencing", which seems to read a lot more smoothly. --- jsonschema-core.xml | 178 ++++++++++++++++++++++++-------------------- 1 file changed, 98 insertions(+), 80 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index ffa325ce..b9d1e042 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -205,11 +205,11 @@ The schemas to be applied may be present as subschemas comprising all or part of the keyword's value. Alternatively, an applicator may refer to a schema elsewhere in the same schema document, or in a different one. - The mechanism for identifying such referred schemas is defined by the + The mechanism for identifying such referenced schemas is defined by the keyword. - Applicator keywords also define how subschema or referred schema + Applicator keywords also define how subschema or referenced schema boolean assertion results are modified and/or combined to produce the boolean result of the applicator. Applicators may apply any boolean logic operation @@ -626,20 +626,19 @@ and used with caution when defining additional keywords.
-
+
As noted in , an applicator keyword may refer to a schema to be applied, rather than including it as a subschema in the applicator's value. In such situations, the - schema being applied is known as the referred (or referenced) schema, - while the schema containing the applicator keyword is the referring - (or referencing) schema. + schema being applied is known as the referenced schema, while + the schema containing the applicator keyword is the referencing schema. While root schemas and subschemas are static concepts based on a - schema's position within a schema document, referred and referring + schema's position within a schema document, referenced and referencing schemas are dynamic. Different pairs of schemas may find themselves - in various referred and referring arrangements during the evaluation + in various referenced and referencing arrangements during the evaluation of an instance against a schema. @@ -1010,8 +1009,8 @@ Several keywords can be used to reference a schema which is to be applied to the current instance location. "$ref" and "$recursiveRef" are an applicator - keywords, applying the referred schema to the instance. "$recursiveAnchor" - is a helper keyword that controls how the referred schema of "$recursiveRef" + keywords, applying the referenced schema to the instance. "$recursiveAnchor" + is a helper keyword that controls how the referenced schema of "$recursiveRef" is determined. @@ -1039,49 +1038,40 @@
-
+
The "$recursiveRef" and "$recursiveAnchor" keywords are used to construct - extensible recursive schemas. + extensible recursive schemas. A recursive schema is one that has + a reference to its own root, identified by the empty fragment + URI reference ("#"). - Intuitively, when using "$ref" or another - similar keyword to combine a recursive schema with another schema (recursive - or otherwise), the goal of the schema author is often to have the - recursion respect that combination. The recursive reference would - ideally always recurse to where the processing of the schema started. - - - But this is not possible with the static behavior of "$ref", wich can - only refer to the root schema of the current schema document. - More accurately, it can only refer to one location, and that location - is constrained by the static rules for resolving URI References. - - - This constraint leads to verbose and error-prone re-definitions of each - recursive element, as can be seen in the meta-scheme for JSON Hyper-Schema - in all prior drafts. + Extending a recursive schema with "$ref" alone involves redefining all + recursive references in the source schema to point to the root of the + extension. This produces the correct recursive behavior in the extension, + which is that all recursion should reference the root of the extension.
- Consider the following two schemas. The first (given the id "basic") - is an object with one string property and one reference property. - The reference is recursive, pointing to the root of the current - schema document. The second schema references the first, and - also describes a "things" property, which is an array of - recursive references. + Consider the following two schemas. The first schema, identified + as "original" as it is the schema to be extended, describes + an object with one string property and one recursive reference + property, "r". The second schema, identified as "extension", + references the first, and describes an additional things" property, + which is an array of recursive references. + It also repeats the description of "r" from the original schema. - The problem is that the referred targets of the - "$ref": "#" - references are statically determined. Since the - "things" array is in the - combined schema, its referred schema is the combined - schema. But the "recursive" - property in the basic schema still points to the root - of that basic schema, and therefore will not see the - description of the "things" - property. What we want is for it to resolve - to the combined schema as well. + This apparent duplication is important because + it resolves to "https://example.com/extension#", meaning that + for instance validated against the extension schema, the value + of "r" must be valid according to the extension, and not just the + original schema as "r" was described there.
+ + This approach is fine for a single recursive field, but the more + complicated the original schema, the more redefinitions are necessary + in the extension. This leads to a verbose and error-prone extension, + which must be kept synchronized with the original schema if the + original changes its recursive fields. + This approach can be seen in the meta-schema for JSON Hyper-Schema + in all prior drafts. +
- Since the desired behavior can seem surprising, and unpredictable, - it is important to use keywords to explicitly control all aspects - of the behavior. In order to create a recursive reference, we - must do three things: + The desired behavior is for the recursive reference, "r", in the + original schema to resolve to the original schema when that + is the only schema being used, but to resolve to the extension + schema when using the extension. Then there would be no need + to redefine the "r" property, or others like it, in the extension. + + + In order to create a recursive reference, we must do three things: - In our "basic" schema, indicate that the schema author + In our original schema, indicate that the schema author intends for it to be extensible recursively. - In our "extension" schema, indicate that it is intended + In our extension schema, indicate that it is intended to be a recursive extension. @@ -1146,22 +1146,25 @@ The "$recursiveAnchor" keyword is how schema authors indicate that a schema can be extended recursively, and be a recursive schema. This keyword MAY appear in the root schema of a - schema document, and MUST NOT appear in a subschema. + schema document, and MUST NOT appear in any subschema. The value of "$recursiveAnchor" MUST be of type boolean, and MUST be true. The value false is reserved for possible future use. +
+
The "$recursiveRef" keyword behaves identically to "$ref", except - that if the referred schema has "$recursiveAnchor" set to true, - then the implementation MUST check the dyanamic scope to see - if "$recursiveAnchor" had previously been set. If so, then the - referred schema is considered to be the outermost (in terms of - dynamic scope) schema object containing "$recursiveAnchor" set to true. + that if the referenced schema has "$recursiveAnchor" set to true, + then the implementation MUST examine the dynamic scope for the + outermost (first seen) schema document with "$recursiveAnchor" + set to true. If such a schema document exists, then the target + of the "$recursiveRef" MUST be set to that document's URI, in + place of the URI produced by the rules for "$ref". - Note that if the schema to which "$recursiveRef" referrs does not + Note that if the schema referenced by "$recursiveRef" does not contain "$recursiveAnchor" set to true, or if there are no other "$recursiveAnchor" keywords set to true anywhere further back in the dynamic scope, then "$recursiveRef"'s behavior is identical @@ -1175,14 +1178,14 @@ - Now lets consider the evaluation of the "extension" schema. - Note that the "$ref": "basic" was not changed, as it works - just fine as a normals static reference. And the - "$recursiveRef" in the "extended" schema does not behave at - all differently, because the "$recursiveAnchor" in its - referred schema is the outermost "$recursiveAnchor" in the - dynamic scope. However, the "$recursiveRef" in the "basic" - schema referrs to a "$recursiveAnchor" that is not the - outermost such keyword in the dynamic scope. That is still - the "$recursiveAnchor" in the "extension" schema. - Therefore, when processing starts with the extension - schema, the "$recursiveRef" in the basic schema actually - referrs to the "extension" schema's root schema. + Note that the "r" property no longer appears in the + extension schema. Instead, all "$ref"s have been changed + to "$recursiveRef"s, and both schemas have "$recursiveAnchor" + set to true in their root schema. + + When using the original schema on its own, there is no change + in behavior. The "$recursiveRef" does lead to a schema where + "$recursiveAnchor" is set to true, but since the original schema + is the only schema document in the dynamics scope (it references + itself, and does not reference any other schema documents), the + behavior is effectively the same as "$ref". + + + When using the extension schema, the "$recursiveRef" within + that schema (for the array items within "things") also effectively + behaves like "$ref". The extension schema is the outermost + dynamic scope, so the reference target is not changed. + + + In contrast, when using the extension schema, the "$recursiveRef" + for "r" in the original schema now behaves differently. Its + initial target is the root schema of the original schema document, + which has "$recursiveAnchor" set to true. In this case, the + outermost dynamic scope that also has "$recursiveAnchor" set to + true is the extension schema. So when using the extensions schema, + "r"'s reference in the original schema will resolve to + "https://example.com/extension#", not "https://example.com/original#". +
@@ -1512,7 +1530,7 @@ The application can use the schema location path to determine which values are which. The values in the feature's immediate "enabled" property schema are more specific, while the values under the re-usable - schema that is referred to with "$ref" are more generic. The schema + schema that is referenced to with "$ref" are more generic. The schema location path will show whether each value was found by crossing a "$ref" or not.