Determine if default non-compliant behaviour of implementations should be noted

[Relequestual]

A PR to update the draft compliance of opis/json-schema was created: #386

The implementation has a number of non-compliant behaviours by default.
After discussion, feature flags were added to allow users to, when correctly configured, have spec compliant behaviour.

For exmple, the default keyword, by default, would modify the instance BEFORE validation, should a value not be included in the instance at a location. A feature flag was added, but I still feel this is undesierable, significantly enough to warrant recomended we did not update the librareis compliance listing.

So, I now raise the question, should we note implementations that claim compliance, but which are not compliant as far as we are aware?

I'm not suggesting we audit every implementation, but I usually check if a library is using the test suite, and ask questions if they do not.

I'm not suggesting we re-visit every implementation we currently list, but that we do checks when updating compliance.

I'm not suggesting we specifically list the ways in which a library is non-compliant, just that it is non-compliant in some way.

[handrews]

Integrating the list with test suite results would also be nice, and a more objective way of reporting compliance. Do we have a test that verifies that the instance is unchanged? (EDIT: Is that even possible in our testing framework, and if not should we try to add it somehow?)

[Relequestual]

I don't believe we do, but we COULD by proxy of making the default not valid in lieu of modifying the test suite structures.

[Relequestual]

We do: https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/master/tests/draft2020-12/default.json

[Relequestual]

We would have to devise a way to make sure the tests being used are correct, and determine an output of test compliance that we'd look for (if we can even reasonably do that... I don't know what sort of test reporting is possible across multiple languages)

[handrews]

@Relequestual yeah it's probably a non-trivial undertaking.

[msarca]

My 2 cents on this issue: the fact that someone knows how to write a validation schema doesn't mean they know how to use opis/json-schema. If you want to use opis/json-schem you must read the documentation first and learn how to setup the library. You can setup the library in such a way that is fully compliant with the specification. The default setup is not fully compliant, but this was our choice and nobody has the right to dictate to us which features to be enabled by default and which not.

So, to wrap up: opis/json-schema can be setup to be fully compliant, therefore is fully compliant.

[handrews]

@msarca

nobody has the right to dictate to us which features to be enabled by default and which not.

🤨

This is about "rights" now? We're talking about a spec, not a moral obligation. The spec defines what is and isn't compliant. It is in fact the definition of a specification to dictate what is and is not compliant. That's what it means to have a spec.

If this is ambiguous, it can and should be clarified. If the purpose of a page listing implementations is ambiguous, that can and should be clarified.

At that point, you can choose to meet the criteria or not. Your "rights" are not being impeded regardless of what you choose to do or not do.

[jdesrosiers]

Generally I'm not too concerned about default configuration. As long as it can be easily configured to be compatible. Although, lately ajv's strict mode is making me think twice about that. I've seen a few times now where people get confused thinking a schema they are trying to use is not valid because ajv's strict mode is throwing errors.

For now, specially for opis/json-schema, I'm not worried about default as long as it can be turned of (I see that it can). The part that was more concerning to me was the use of Relative JSON Pointer. Last I checked, there was no way to turn that off. If that's changed, then I wouldn't be opposed to putting it on the website. However, what I really think opis/json-schema should do is declare their own dialect and do whatever they like. I'd be happy to put it on the website with a custom dialect declared and a note that it's mostly compatible with 2020-12.

[gregsdennis]

This is closely related to json-schema-org/JSON-Schema-Test-Suite#314

In this issue I mention a site that someone put together for JSON Path comparisons. I think something like this that covers our test suite would go a long way.

[sorinsarca]

The part that was more concerning to me was the use of Relative JSON Pointer. Last I checked, there was no way to turn that off.

@jdesrosiers that can be turned on/off using the allowRelativeJsonPointerInRef option https://opis.io/json-schema/2.x/php-loader.html#parser-options

[msarca]

This is about "rights" now?
At that point, you can choose to meet the criteria or not.

No doubt this is about rights! Your personal opinion is not a valid criterion because is subjective. You believe our default configuration is wrong, but we believe otherwise. In order for you to be able to judge if something is compliant or not you must have a set of objective criteria that can be applied to all. Do you have one?

[handrews]

@jdesrosiers

Generally I'm not too concerned about default configuration.

I tend to be concerned about this, and in fact the spec has had notes about default configuration for a long time (regarding format, so... meh... but default configuration has long been in the scope of the spec).

The problem is when you have a very large-scale system with different people/companies/products using different implementations. You expect that if everyone installs a nominally-conforming implementation then things will be interoperable to the extent governed by the specification.

Many (most?) people do not read the docs. I don't even mean that they decide not to. I mean that they install X which installs Y which installs Z which installs JSON Schema implementation A, and Y also installs W which installs implementation B, and going in and reconfiguring implementations that you may or may not realize are even present is pretty unlikely.

For the JSON Schema project, we do not want to get a reputation for being unreliably interoperable, or for having an inconsistently implemented ecosystem (that's part of the problem with format- as much as the keyword vexes me, if it was reliably and consistently implemented it would be OK. But it never has been).

However, what I really think opis/json-schema should do is declare their own dialect and do whatever they like. I'd be happy to put it on the website with a custom dialect declared and a note that it's mostly compatible with 2020-12.

Not everything is within the scope of a dialect. Mutating the instance is just not within the scope of regular JSON Schema behavior. It's not a part of any of the keyword classifications. So adding that is going outside the spec more than just adding a vocabulary of keywords that are within the regular scope of JSON Schema functionality would be.

---

Regardless, we do need to clarify what is required to be on the page, and what has to get called out on the page if we want some leeway in that but also want to give clarity to users.

[handrews]

@msarca this issue is about resolving that. Please read what I stated about clarifying ambiguity.

I am puzzled by you treating this as some sort of philosophical debate about subjectivity or some sort of attack on your implementation. JSON Schema is a volunteer project. There is far more to do than people to do it. When someone runs up against something that needs to be done, then we look into it. That process is going on right here, with your participation. That's all that this is.

We don't have enough time or people to have everything perfectly sorted, but we are doing our best to work it out visibly, where people can participate.

[handrews]

Having thought on this more, here are some options that I see, which are not necessarily mutually exclusive:

• Nail down more implementation behavior and expectations as requirements in the spec. This has the advantage of being clear to everyone (if you aren't looking in the spec, what are you even doing?) but the more we get into implementation behavior the more we have to anticipate the conditions of a range of implementation contexts. We might want to clarify a few things, but this is a big hammer that is often not needed.
• Require test results and display them on the page. I would be happy with just statically putting the results up- of course someone might then diverge from it, but this is all somewhat best-effort. Requiring live-updated test results is a lot and I don't see much point in requiring it.
• Allow specifying configuration information on the page, along with test results. This could allow for showing both the default results and the most-compatible-configuration results.

Some of this would require significant redesigning of the page, but TBH we should do that anyway. None of this applies to the non-validation functionality, the vast majority of which is not standardized anyway. We should clarify that distinction.

I personally like the idea of displaying test results alongside configuration information. That offers both clarity and flexibility.

I'm not bothered by putting partially conformant implementations on the page as long as it's very clear what you get if you just go grab the thing and use it as-is. I'm fairly sure we have some partially compliant implementations there now, some with comments, some probably without. I don't remember when I last audited the whole mess, but I tended to err on the side of keeping listings.

Both short-term and long-term, whatever we do should be consistent, so if we have other implementations with similar situations as the implementation in question the we should either take them all off or put this one on. If we have any, they should all be noted clearly while we figure out something better.

I will note that auditing the implementations is a giant pain- I used to do it every draft, mostly because I went around and contacted each project to let them know a new draft was out. We should not really be in the business of doing that as a general rule. Requiring test results in a PR seems reasonable. Of course people could falsify results, but then they're undercutting their own reputation and a.) that's not our problem, and b.) I'm not that worried about people doing it.

[msarca]

You are opening issues regarding our library and then you declare I am puzzled by you treating this as some sort of philosophical debate about subjectivity or some sort of attack on your implementation.

Look, guys, if you don't believe that Opis can bring some value to the JSON Schema project, feel free to remove it from your official page.