Clarify that we are interested in the data model. #1194
Conversation
* Change URIs to IRIs where appropriate
…ons can start testing with them (json-schema-org#1148)
* Remove bookending requirement for dynamicRef * Add $dynamicRef changes to change log
Signed-off-by: Juan Cruz Viotti <[email protected]>
Looks like this one has been missed. Signed-off-by: Juan Cruz Viotti <[email protected]>
It is a best practice to allow callers to override the value of these program variables. For example, with this chance, you can do: ```sh make XML2RFC=/opt/bin/xml2rfc ``` See: https://www.gnu.org/software/make/manual/make.html#Command-Variables Signed-off-by: Juan Cruz Viotti <[email protected]>
* Since we have "JSON text" in notational conventions, we don't need to reference the application/json media type
ioggstream
changed the title
| interchangeable because of the data model it defines. | ||
| </t> | ||
| <t> | ||
| JSON Schema is only defined over JSON documents. However, any document or memory | ||
| structure that can be parsed into or processed according to the JSON Schema data | ||
| model can be interpreted against a JSON Schema, including media types like | ||
| model can be interpreted against a JSON Schema, including data formats like |
There was a problem hiding this comment.
CBOR is a data format. It can have different media types.
| A JSON document is an information resource (series of octets) described by the | ||
| application/json media type. |
There was a problem hiding this comment.
iiuc this means "JSON documents" := "JSON text", thus resulting in a redundant definition.
There was a problem hiding this comment.
I think you're right. I don't think there's much value in including that sentence.
|
The JSON Schema spec is being submitted as an Internet specification, or at least must be compatible with Internet standards. Saying "application/json media type" is intended in the first part, as it's a well-defined technical term, and using it implies something specific for interoperability on the Internet. It's saying more than simply "JSON document := JSON text." (Where "JSON text" is the ABNF production.) A JSON document is not merely any series of bytes that matches the JSON text grammar. It also has to be marked as being a JSON document. A Web browser cannot parse a "text/plain" document as JSON just because it matches the JSON grammar. As for using "media types" vs. "data formats", I don't care one way or the other, we don't actually need the well-defined term here. But CBOR is a media type, it's content-type registration is |
|
Hi @awwright, let me clarify + forgive the nitpicking
Ok
My understanding of "The media type for JSON text is application/json" from RFC8259 is that they imply the same thing.
Your point is clear now. Thinking twice, imho a
To progress with the draft, in my experience these are the kind of clarifications I was asked to resolve by the IETF community for a couple of specs I started working, so I'm just trying to share the lesson learnt :) While I try to do my best, I can still make mistakes. Moreover I understand that this whole discussion may seem pointless, but I think it is not from the perspective of a first time reader.
See above wrt "pointless nitpicking": my understanding from reading CBOR is that it is a data format. |
|
As far as I'm concerned, nit-picking is worthwhile and encouraged.
I think this is an unfortunate case where RFC 8259 is using "JSON text" to refer to both the ABNF production (the set of all strings that form valid JSON) and a JSON document (a string that's valid JSON that's known to be I'm not sure why RFC 8259 avoids the term "JSON document" since that's the terminology in other media types ("HTML document", "PDF document", etc). I didn't previously notice this.
I think it's fair to call CBOR either a data format or a media type; since media types I would describe as standardized data formats. |
Just to be clear, the only thing we plan to submit as an internet standard are the JSON Schema media types, which are just relatively minor extension of |
|
@jdesrosiers I would call JSON Schema a little more than a "relatively minor extension of |
I've said multiple times now that the meaning comes from the dialect declaration. Does this not make sense? Do you see this as insufficient? Why?
Sure, you can think of dialect declaration as an extension mechanism. The document has no intrinsic semantics. You extend those semantics by declaring a dialect. The media type defines how to declare the dialect of the schema using the But, yes, if the document doesn't declare a dialect, the document would have no meaning that a user-agent could act on. All meaning and behavior is pushed to user-space. Other than JSON Pointer fragment identification, it would be just like working with a JSON document. It wouldn't function as a schema, but that doesn't mean it's useless in some other context. You could also use this media type to declare a dialect that has nothing to do with schemas. It could have some other purpose entirely. Of course the media type isn't specifically designed for or intended to be used in these ways, but it's possible. |
I asked earlier: If I specify the schema {"type":[]} or {"not":{}}, how do I know what that means? More specifically, how do I determine what the "dialect" is? |
It sounds like what you're proposing is just JSON with Namespaces. There's similar projects that do this, like JSON-LD. I'm not necessarily against namespaces or a new media type for them, but this misses the point of JSON Schema, which is to have a single, standard vocabulary for describing assertions, annotations, and sets of JSON. And also, most people are not interested in declaring a meta-schema or dialect for day-to-day use of JSON Schema. Different implementations doing something different for even simple statements like And by the way, I see a good idea related to what you're proposing; what would it take to encode a meta-schema as a JSON-LD document, and write a validator that supports these JSON-LD schemas? |
Sorry, I didn't realize that was the question you were asking. Dialect identification works the way it always has. See here for a summary. If no dialect is declared, behavior is undefined. Implementations can assume a default dialect, or use heuristics to guess a dialect or, throw an error, or proceed with no dialect (all properties treated as unknown keywords). I'd rather it not be undefined, but it has to be compatible with older dialects and existing implementations.
If you think of a dialect as a namespace, I guess you could say that, but that's not really the point. The point is just to define something flexible enough to support any past or future JSON Schema dialect.
I disagree that that's the goal, but even if is, the reality is that we have many dialects of JSON Schema in use today and this media type needs to support all of the dialects people are using in the wild. Then you also have to take into account the third-party dialects out there for things like code-gen, form-gen, and databases. All of these should be able to use the media-type as well. That's why the media-type needs to be flexible. |
I'm not sure what you're referring to, "dialect" is new term to me. Usually we just called this the meta-schema.
Can you specifically name some of these dialects? JSON Schema is not (or should not be) any meaningfully different from HTML... over time new features (versions) have emerged, but there is only one HTML "dialect," and newer specifications are backward-compatible. (In HTML 4.01 you could declare usage of one of a few DTDs, but these have all been superseded.) And if these features are not sufficient, you can use a different XML namespace. |
The term was introduced in the 2019-09 spec.
Previously, this said "version" instead of "dialect", but the meaning is the same. The concept hasn't changed, only the word we use to describe it.
Some examples include draft-04, draft-07, 2020-12, Open API 2 JSON Schema, Open API 3.0 JSON Schema, Open API 3.1 JSON Schema, and Mongo DB JSON Schema.
I completely agree that we want to go that direction with JSON Schema, but we still have to support the current and older versions that don't work that way. I expect to end up in the same place as HTML is today. HTML no longer has versions, but it still has |
|
My understanding is that the schema identifier is provided as a convenience, and it doesn't allow someone to unilaterally redefine a keyword to be (significantly) backwards-incompatible. But it can be used as a hint to suggest that there's custom keywords in use, or that there's new keywords that the validator doesn't understand (which is effectively the same thing from the validator's point of view). There's a few reasons I understand it this way: First, the meta-schema is not authoritative, just informative. The spec defines the behavior and permitted values, the meta-schema tries to describe that as closely as possible; which is why we can issue revisions of a meta-schema. Or, at least it used to work this way. Second, the 8.1. Meta-Schemas and Vocabularies section begins by listing just two purposes, which makes the following section (8.1.1. The "$schema" Keyword) seem much less ambitious than I gather from your description. Third, version identifiers are frequently ignored. Maybe, if we absolutely had to, we could use $schema to backtrack on functionality that we decided was a bad idea. (Similar to User-Agent sniffing, how TLS uses its version identifiers, or "quirks mode" in HTML.) But using a version identifier for something more than this is generally a bad idea. Finally, as a general Internet principle, undefined and implementation-specific behavior should be avoided, and the idea that an (a) xmlns changes the semantics of an XML document. In XHTML, you had to specify the xmlns=, but this isn't quite the same thing because it's the same regardless of the HTML version, and if you omit the namespace declaration, the (b) Doctypes they don't change the semantics of the document, only its grammar. I think this is essentially the same as a JSON schema. HTML 4.01 had three DTDs, a version that was reverse compatible with older HTML, a "strict" version for new documents that provided stronger guarantees of cross-platform compatibility, and a frames version because mixing frame elements with non-frame elements is nonsensical. There's a lot of implementations out there that will never implement $vocabulary or even $schema because there's no need to. The core functionality can all be implemented, with reverse-compatibility supported all the way back to the beginning (with very niche exceptions probably not encountered in the wild).
Thanks, this is helpful. I don't see a need to identify these as separate dialects. draft-04 through 2020-12 are just different releases of the meta-schema: They're a machine-readable version of the same media type specification at different points in time. Open API 2 JSON Schema etc. don't need their own dialects either, unless they're defining custom keywords. You can implement a subset of JSON Schema functionality as long as you bail out if a schema tries to use the unsupported functionality. |
|
I really like this vision. It's very similar to where I want JSON Schema to end up. Where we disagree is that your interpretation can be applied to the current and historical state of JSON Schema. Every release (with the possible exception of draft-07) has had backwards incompatible changes. This makes it necessary to know which dialect defines the intended semantics for a given schema. In the relatively near future, I'd like to declare something like
You're thinking about this differently than what is intended. The data in an |
|
Not to get this PR too off-topic, I filed json-schema-org/community#189. Sorry I didn't realize I had a somewhat different interpretation than everyone.
I don't think this has ever been necessary. With respect to changing spec'd behavior, my thought at the time was we would not introduce behavior that would be incompatible with older drafts; implementations could continue to implement removed behavior, from the standpoint of not breaking reverse-compatibility in their own libraries. Additionally, declaring (and being valid against) the newer meta-schema is a way of verifying that you don't use any deprecated functionality. And the reason that we have different drafts described in the test suite is not to suggest the behavior can change outright, but so that implementations can test backwards compatibility. My own implementation passes every test in every draft (except a few of the newer features in 2019-09), yet it doesn't read "$schema" at all. Do you have any counter-examples? Are there any tests that flatly contradict themselves between drafts? It occurs to me now, the best way to handle backwards compatibility is not to remove the functionality from the spec, but to move it into a separate section describing deprecated functionality. The downside is this is slightly more complicated to author. Infrequently, there were also a few features that we retconned (for example:
Normally I would agree In contrast, with JSON Schema, we do define semantics. And I wouldn't ever expect that "type" can possibly mean different things in different contexts; it's a standardized name for a standard rule. |
Yes. Take While in 2019-09 and above, they must NOT be ignored. Unless a dialect is provided, how do you know the intent of the schema author? Take
Unless the behaviour of JSON Schema validation is predictable and reliable, it's useless. I'm all for discussing future drafts being backwards compatible without breaking changes (when we've reached acceptable stability), but that's future, and this is now.
That IS the exact reason they DO need their own dialect; OpenAPI defines their own custom keywords. It's totally possible to create a dialect using the core and applicator vocabularies, but not the validation vocabulary. @jdesrosiers @awwright I'm very strongly of the opinion that a "living standard" would be a huge mistake for JSON Schema.
I think "living standard" sort of works for WHATWG because the major vendors are at the table and commit to implement or reject and riase concerns over changes. The objective they have is to align the spec with the implementaitons, of which there are few, and over which they have control. Neither of those things are true for JSON Schema. WHATWG reference: https://whatwg.org/faq#living-standard I'm open to being convinced otherwise. Any implementation which attempts to mish-mash across dialects of JSON Schema is in for a hard time, and can't provide reliable and predictable validaiton. IMHO, it's a travesty. I can't imagine a situation where unreliable and unpredictable validation is even close to acceptable. This is why we have the test suite, and why JSON Schema isn't dead. |
Ok, good point. This is another example that I don't think is a problem in practice. In my implementation, I forgot, I do mask a few tests. It's one of those situations I described where it didn't really break anyone in the wild, and if it does, it can be implemented after a major version increment, and there's an easy fix. (In this case, if you had a draft-07 document with ignored keywords adjacent to a $ref keyword, you can rename/move them into a "$comment" keyword. This preserves the old behavior and makes it forward-compatible.) Specifications sometimes do things like this. HTML has done this a lot. That doesn't mean suddenly there's different "dialects" of HTML.
I had the impression we made the change understanding that case doesn't (significantly) occur in the wild. And if anything, the new behavior was found in the wild more often (because of misunderstanding).
Sure, this is what I imagine as the use case for $schema. I think there's a strong case for "$schema". In fact, I'll walk back what I said a little bit: If they want to define a "dialect" that is a strict subset of JSON Schema, that would be another good case for
Fully agreed! I'm not citing HTML for this reason. It's just that, as the biggest hypermedia format, it's easy to cite for examples. (I don't even like that name... as opposed to what, a "dead standard?" It sounds like a snub). |
|
A dead standard is how WHATWG refer to the IETF RFC approach. Sure sounds like a snub. |
|
Also, if you want to sniff |
Unequivocally no. You've assumed only annotation keywords adjacent to $ref. Part of the work we are looking to do allows us to evaluate implementations against the test suite by ourselves, and report that to users. Any implementation which doesn't pass all the required tests will be "not recommended". We actually recently refused to update the support listing of an implementation for deliberately going against the specified required behaviour. Having proper interoperability is just so critical to JSON Schema. I can't see anyone being able to convince me otherwise. |
I consider that line of justification having no weight here. We have a technical specification. It clearly defines requirements. We even have a test suite. People should be able to rely on conformance to the spec. If they cannot, it's useless. |
I'm not sure what you mean, I think my example was pretty solid. If you had a draft-04 schema such as I never understood "$schema" to be how we differentiated between these behaviors. For example, what if you write your own meta-schema? Which $ref behavior gets used? If $schema decides this question, the answer is unclear. If you understand that newer drafts replace older drafts, the answer is clear.
Ok, but the spec right now seems to say that the behavior of a schema without a |
|
I rewrote the OP for json-schema-org/community#189, hopefully that is much clearer and lays out some of the problems that have to be solved. |
In draft2020-12: "The value of "items" MUST be a valid JSON Schema. " But not so in earlier drafts -- an array was also valid (and indeed there are tests of this variant). So if your implementation accepts an array at the Another behaviour that is more limited than it used to be is the syntax of draft2019-09 also removed the |
The URI at This is how the So, if you want to support your own metaschema that's not based on any existing spec version, you need to build that logic into your evaluator, that is triggered when that schema URI is seen. |
But supporting the schema form is not mutually exclusive with supporting the array form. Is there an example where supporting a current feature precludes supporting an older one? The example of properties adjacent to $ref was given (previously they were ignored, now they're evaluated)— but my issues with this example are (1) "$ref" is a core keyword, its behavior doesn't change with respect to "$schema", and (2) nobody was using keywords next to $ref and expecting them to not do anything. We changed it because people were using them and they were expecting it to do something.
No, it is perfectly legal to support features that have since been removed from the spec. This is how we support reverse compatibility with older drafts. We don't redefine behavior that's been specified in older drafts, it only becomes undefined. For example, we would not be able to redefine the array form of "items" with a different meaning. In any event, "definitions" was always technically ignored (it was merely reserved).
This isn't specified; I thought this was what $vocabulary was for. A meta-schema can provide its own URI as its schema, or omit it entirely. For example, hyper-schema references itself. |
I gave you several -- are you not counting where earlier versions of the spec supported something and now it doesn't, even if it now says "MUST NOT"? That's a bit disingenuous don't you think? Are you suggesting that it is never possible to remove a keyword entirely, because there isn't a conflict if an implementation chooses to keep implementing it anyway? And some implementations continuing to support old keywords while others don't will lead to unpredictable behaviour in the wild, which is exactly what we're trying to prevent by having a specification document.
That's nothing to do with it being a core keyword. We've changed core keyword behaviours several times, and may do so again in the future. All we really need to lock down between versions is
Maybe, if the schema doesn't explicitly contain a
It wasn't, because schemas underneath
If it does, it better be one of the official draft specification metaschemas, or some other metaschema that is "known" to the implementation as the root source of keyword behaviours. Hyperschema is one - it has its own specification document, and cannot be implemented solely in terms of JSON Schema draft on its own. I suppose a metaschema could omit the |
The use of "MUST NOT" in the "$id" definition is a limitation on what authors & implementations may produce; this doesn't imply that validators must produce an error if they encounter the prohibited form. They can still support the older behavior, because it's not ambiguous. As far as I know, none of your other examples mention MUST NOT.
Implementations can continue to support functionality removed from the spec, as a matter of not breaking reverse compatibility in their implementation. This is not to imply that old behavior is cross-platform, of course. If we want to support deprecated functionality in a cross-platform way, we would have to write it into the spec.
Right, but there's nothing in the specification that says the behavior changes based on the $schema keyword.
This is the basis for my issue json-schema-org/community#189. Your suggestion seems reasonable, but the spec currently suggests that validators encountering schemas without a "$schema" keyword may define any arbitrary behavior, including breaking behavior; and this sounds unreasonable to me.
Agreed, this is an important clarification.
This makes sense, but this behavior isn't specified in the spec. |
Also, if by "entirely" you mean we have to pay attention to behavior long since removed—yes. This is generally the case. For example, HTTP is probably never going to redefine behavior for 402, 419, or 420. |
|
The
|
|
@ioggstream could you please re-submit your changes as a new branch from main if you are still interested in having them reviewed and adopted? Between the removal of draft-next and the very long tangential digression of the comments discussion (which is better covered in #1242 and should not be continued here), the PR in this form is too confusing to finalize. I would recommend splitting the added xref (which seems straightforward as a PR) from the debate over what is or is not a data format, which could probably use clarification through discussion in an issue since there did not seem to be a consensus here. |
|
Hi, I think I could do it for September;) adding a note for that. |
|
@handrews since you only wanted the editorial parts, it was easy (see #1273 #1274 ). CBOR defines itself as a data format, so I think we can trust it :) https://cbor.io/ |
This PR
so there's no need to reference the application/json media type
now JSON Document references the Data Model section below, so we don't need to mess with "sequence of bytes"