Add valid test instance for `{"contains": false}` schema

[Zac-HD]

I make rather unconventional use of this test corpus for hypothesis-jsonschema: instead of checking that the examples validate against each schema, I check that I can generate an instance if and only if there is a valid example for that schema (and then that whatever I generated validates).

JSON-Schema-Test-Suite/tests/draft7/contains.json

Lines 81 to 92 in c17161a

	"schema": {"contains": false},
	"tests": [
	{
	"description": "any non-empty array is invalid",
	"data": ["foo"],
	"valid": false
	},
	{
	"description": "empty array is invalid",
	"data": [],
	"valid": false
	}

The clear intention is that nothing should validation against this, but the effect is that only non-array instances validate. So it would be convenient for my downstream use and clear for other users to add an explicit "type": "array" to this schema.

I intend to do this myself or as a mentored good first issue at the PyCon Australia sprints next week 😄

[Julian]

Hey!

Which ones are you referring to here, is it just the one you linked to or all the ones in the file?

This makes me... possibly a bit uncomfortable to be honest. The specific test you linked to is really a "copy pasted" test across the suite that basically checks the behavior of setting keywords to booleans after boolean schemas were added in draft 6 or whatever.

I think you're right that the way it's written suggests that it cares only about the behavior on arrays, but that almost suggests we should add an additional test to all those that make sure that that schema passes on other instances.

Because for the "standard" use of the suite, what we're trying to do is a "manual cover" across all likely or known-previous bugs in any implementation and any language, so we don't want to create blind spots really in places where it's at least imaginable someone would make a mistake (e.g. by implementing contains to fail on non-arrays).

But yeah maybe help me understand exactly what you're trying to suggest change, and how wide, and maybe also if it'd help your use case to instead add a test of the form you need instead of changing an existing one (since yeah another "expected" thing is that things using the suite should support skipping tests that don't make sense for them or that don't pass, etc.).

(Sorry for the brain dump.)

[Zac-HD]

That specific test schema is the only one in the whole suite where every test has "valid": false but there are possible instances that would validate. (at least as far as hypothesis-jsonschema can currently generate)

I'm using "has no tests with "valid": true to mean "schema is equivalent to false" - I realise that this isn't an intended interpretation but it holds for every schema in the suite but one and is pretty useful!

Happy to go with your preference of:

• Working around this downstream, by adding lists of known schemas to the "is anything valid for this" logic.
• Adding "type": "array" for this specific test schema
• Adding an additional test for this schema that is valid (e.g. `"contains should only affect arrays")

[Julian]

A ha! That's a lot more reasonable of an invariant to expect, thanks for clarifying.

I'm totally open to not asking you to handle that downstream yeah -- I think #3 sounds perfectly fine to me!

[Julian]

And I'm also somewhat considering that in the future perhaps we should use hypothesis-jsonschema to enforce that constraint on the test suite in a CI check, but maybe let's discuss that... after this ticket :D