Resolving $refs defined in unknown keywords

[aravindanve]

How should a json-schema (draft7) implementation resolve $refs defined in unknown keywords?

JSON-Schema-Test-Suite defines schemas such as this, and I assume they are valid:

{
    "tilda~field": {"type": "integer"},
    "slash/field": {"type": "integer"},
    "percent%field": {"type": "integer"},
    "properties": {
        "tilda": {"$ref": "#/tilda~0field"},
        "slash": {"$ref": "#/slash~1field"},
        "percent": {"$ref": "#/percent%25field"}
    }
}

Take this example below:

{
    "$id": "http://example.com/root.json",
    "definitions": {
        "A": { "type": "integer" }
    },
    "properties": {
        "$id": {
            "type": "string"
        },
        "attributes": {
            "$ref": "#/tilda~0field/slash~1field/$id"
        }
    },
    "tilda~field": {
        "$id": "t/inner.json",
        "slash/field": {
            "$id": {
                "$id": "test/b",
                "$ref": "document.json"
            }
        }
    }
}

Which of the following is the $ref at #/tilda~0field/slash~1field/$id/$ref resolved to?

• http://example.com/root.json/document.json
• http://example.com/t/document.json
• http://example.com/t/test/document.json

Which of the $ids in #/tilda~0field must be considered as baseURI for the $ref in question and why.

PS I'm not sure if this is the right place to post this. If I need to clarify further or post this elsewhere, please let me know.

[gregsdennis]

Which of the following is the $ref at #/tilda~0field/slash~1field/$id/$ref resolved to?

I think Section 8.2 gives the answer: "A subschema's "$id" is resolved against the base URI of its parent schema." This means that it will be http://example.com/t/test/document.json.

Which of the $ids in #/tilda~0field must be considered as baseURI for the $ref in question and why.

The base URI for this is the $id at the root, re-routed by the $id in tilda~field.

1. Start with http://example.com/root.json.
2. Change folders to t and use file inner.json.
3. Change folders to test (inside t) and use file document.json.

I'm not sure if this is the right place to post this. If I need to clarify further or post this elsewhere, please let me know.

This is a perfect place to post questions. It's watched and it will be available for other to read. If you want to carry on in conversation "offline" about something, then feel free to join the Slack workspace.

[gregsdennis]

I think this answer is better than what I said, though the gist of it is the same.

[aravindanve]

Thanks for the prompt response @gregsdennis!

I think Section 8.2 gives the answer: "A subschema's "$id" is resolved against the base URI of its parent schema." This means that it will be http://example.com/t/test/document.json.

The problem is, I'm not clear on how to identify the last parent schema. The spec says:

A JSON Schema MAY contain properties which are not schema keywords. Unknown keywords SHOULD be ignored.

So if a schema defines an unknown keyword mappings like this:

{
    "$id": "http://example.com/root.json",
    "properties": {
        "required": ["$id"],
        "$id": { "type": "string" },
        "$name": { "type": "string" },
        "$email": { "type": "string", "format": "email" },
        "$title": {
            "$ref": "#/mappings/$title"
        },
        "$profile": {
            "$ref": "#/mappings/$profile"
        }
    },
    "mappings": {
        "$id": "_id",
        "$name": "name",
        "$title": {
            "$ref": "title.json"
        },
        "$profile": {
            "$ref": "profile.json"
        }
    }
}

Now, when my validator encounters "$ref": "#/mappings/$profile", should it try and resolve the $ref found at #/mappings/$profile further or throw?

Resolving the $ref

If it has to resolve the $ref further, then it has to figure out the current baseURI at the location #/mappings/$profile. Which will be done by looking at the parent, which has the $id "_id".

But #/mappings is not a valid schema, and $id here probably describes how a property called $id must be mapped to the underlying database or whatever. Another hint is that the keyword #/mappings/$title is being used in a custom sense here. Now how must the validator proceed?

Should it use http://example.com/_id to resolve profile.json against, even though the parent is clearly not a valid schema, or use the baseURI at the root to resolve the ref?

PS I understand in both cases the ref resolves to http://example.com/profile.json but that is just a coincidence here, it may not be the case always.

[jdesrosiers]

@aravindanve, that's a really good question. I doubt anyone considered a case like that when this was defined. I certainly haven't.

#/mappings/$profile and #/mappings/$id are not schemas because "#/mappings" is not a JSON Schema keyword. Therefore, you can not $ref those paths because $ref is only allowed to reference a schema.

If you could $ref a non-schema, the $id at #/mappings/$id wouldn't apply because #/mappings is not a schema and thus $id is just a property, not a keyword.

[jdesrosiers]

The interesting question here is, how does this affect custom keywords? How does it affect vocabularies?

[gregsdennis]

The key lies in the "SHOULD be ignored" portion. Just ignore it as miscellaneous data. It's not invalid to have it there, but neither should your implementation do anything with it, except maybe store it for serialization back into JSON.

Don't throw; don't resolve the references. It's just JSON data.

In regard to vocabularies, a draft-08 concept, supposing a vocabulary was declared that defines mappings, you would then process in accordance with the vocabulary documentation. This means that someone would need to provide implementation for those keywords via a plug-in or other means to your library so that the keywords can be consumed and interpreted. If no such implementation is provided, then your library can recognize the vocabulary and should throw, but only if the vocabulary is listed as required in the meta-schema. If the vocabulary is not required or no vocabulary defines that particular keyword, it gets ignored as before.

[aravindanve]

@jdesrosiers @gregsdennis This clears things up.

But you see, my first example:

{
    "tilda~field": {"type": "integer"},
    "slash/field": {"type": "integer"},
    "percent%field": {"type": "integer"},
    "properties": {
        "tilda": {"$ref": "#/tilda~0field"},
        "slash": {"$ref": "#/slash~1field"},
        "percent": {"$ref": "#/percent%25field"}
    }
}

Is lifted directly from JSON-Schema-Test-Suite. I think there are a bunch of tests that routinely reference schemas nested inside unknown keywords.

For now, in my implementation I'll disallow $refs pointing to unknown parts of the schema. But it means it will fail a bunch of tests defined by the suite.

[aravindanve]

@jdesrosiers One other thing

#/mappings/$profile and #/mappings/$id are not schemas because "#/mappings" is not a JSON Schema keyword. Therefore, you can not $ref those paths because $ref is only allowed to reference a schema.

Technically #/mappings may not be a valid schema, but #/mappings/$profile is.

So when you say "you can not $ref those paths because $ref is only allowed to reference a schema", it is referencing a schema :)

[aravindanve]

@gregsdennis What do you mean by:

Don't throw; don't resolve the references. It's just JSON data.

If i dont resolve the reference in #/properties/$profile, it automatically is treated as an empty schema by the validator. i.e. any data that is passed, will pass validation. Is that the expected behaviour?

[gregsdennis]

Every time I look at this my brain hurts sees different things. References are hard.

First, since the original reference (which is processed by the implementation as it's under the properties keyword) points to this location, the value at this location is interpreted as a schema (this part I got wrong before). However, the $id value at #/tilda~field is not interpreted as a schema because tilda~field is not a known keyword. It is just a container for JSON data.

Second, the schema at #/tilda~0field/slash~1field/$id has both an $id and a $ref. The draft-07 spec says that keywords alongside $ref should be ignored (this is changing in draft-08), which means that the $id of test/b should have no effect. That being the case, and given the first point, I would expect that http://example.com/document.json is resolved.

To confuse things even more, if you were to wrap the tilda~field inside a definitions keyword, then the $id would be interpreted and the resolution would be http://example.com/t/document.json. This is because definitions is a known keyword and its contents are interpreted specifically to be schemas prior to any $refs evaluating to those locations.

---

By "Don't throw; don't resolve the references," I was just saying that the implementation should ignore the contents of the data. On its own, the contents of unknown keywords shouldn't be validated and references contained within the data shouldn't be resolved. But in your case, the resolution journey begins within a known keyword, so continued resolution is expected.

[handrews]

@aravindanve @gregsdennis 's explanations are correct.

• If the $ref keyword itself is under an unknown keyword, it is ignored because unknown keywords and their entire contents are ignored
• If the target of a $ref is nested under a property name that is not a known keyword, that is fine, because it is just being used as a location. definitions (draft-07 and earlier) and $defs(drat-08) are the recommended locations to put schemas, but that just ensures that they aren't used for anything else. Technically, you can use any name you want to group schemas.

So that test suite example is fine. It would be nice to make it a bit less confusing but if you want to argue for that please file that on the test suite repository.

[aravindanve]

@gregsdennis @handrews I can see what the tests are getting at. I can see the utility in allowing such purely location based references. But my point is, it introduces a certain level of ambiguity as those unknown keyword object references go deeper and deeper.

Okay, so for now i'll stick to this:

First, since the original reference (which is processed by the implementation as it's under the properties keyword) points to this location, the value at this location is interpreted as a schema (this part I got wrong before). However, the $id value at #/tilda~field is not interpreted as a schema because tilda~field is not a known keyword. It is just a container for JSON data.

I'll resolve all $refs in nested schemas against the document root $id (Only for unknown keywords of course). Which is what I have done to get the tests in the suite to pass.

Because the alternative is to determine the nearest parent that "fits" the definition of a schema and use its $id which is non-deterministic at best and may produce random outcomes based on factors such as levels of nesting, names of the properties in those objects etc. Anyways, I'll try and come up with a better example that encompasses all the edge cases I'm considering.

Thanks & Cheers!

[awwright]

Which of the following is the $ref at #/tilda~0field/slash~1field/$id/$ref resolved to?

In my interpretation, you don't, because {$ref} is only parsed where a schema is expected, and that is not a place where a schema is expected.

[gregsdennis]

@awwright on its own, correct, this is not a place where a schema is expected. However, the example shows a $ref pointing to this location, so the data is interpreted as a schema upon resolution of this first one, at which point the secondary $ref (in the data) is then resolved.