Not automatically resolving remote references (in 4.18) breaks backwards compatibility

[joooeey]

Even the most basic use case of fetching a HTTPS ref fails. I.e. with the simple schema:

{"$ref": "https://geojson.org/schema/Point.json"}

I've tried a few others.

Python 3.11.2 (main, Mar 27 2023, 23:42:44) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jsonschema
>>> jsonschema.validate(
...     instance={"type": "Point", "coordinates": [0, 1]},
...     schema={"$ref": "https://geojson.org/schema/Point.json"},
... )
Traceback (most recent call last):
  File "/home/lukas/anaconda3/envs/json/lib/python3.11/site-packages/jsonschema/validators.py", line 406, in _validate_reference
    resolved = self._resolver.lookup(ref)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lukas/anaconda3/envs/json/lib/python3.11/site-packages/referencing/_core.py", line 582, in lookup
    raise exceptions.Unresolvable(ref=ref) from None
referencing.exceptions.Unresolvable: https://geojson.org/schema/Point.json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/lukas/anaconda3/envs/json/lib/python3.11/site-packages/jsonschema/validators.py", line 1263, in validate
    error = exceptions.best_match(validator.iter_errors(instance))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lukas/anaconda3/envs/json/lib/python3.11/site-packages/jsonschema/exceptions.py", line 424, in best_match
    best = next(errors, None)
           ^^^^^^^^^^^^^^^^^^
  File "/home/lukas/anaconda3/envs/json/lib/python3.11/site-packages/jsonschema/validators.py", line 331, in iter_errors
    for error in errors:
  File "/home/lukas/anaconda3/envs/json/lib/python3.11/site-packages/jsonschema/_validators.py", line 284, in ref
    yield from validator._validate_reference(ref=ref, instance=instance)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lukas/anaconda3/envs/json/lib/python3.11/site-packages/jsonschema/validators.py", line 408, in _validate_reference
    raise exceptions._WrappedReferencingError(err)
jsonschema.exceptions._WrappedReferencingError: <exception str() failed>
>>> 

In 4.17.3, the jsonschema.validate(...) call prints nothing (returns None) as expected.

[Julian]

Oh, right, this is definitely semi-intentional (see the docs) -- as doing so is a security risk in general which has only stuck around this long for backwards compat -- I did certainly forget that if you provide neither a ref resolver nor a registry (for the new API) that this does immediately break that compatibility.

Will have to think about what to do / whether to re-introduce a deprecation period for this case.

Though you should certainly stop doing it!

[joooeey]

I see. I might have a look at this once 4.18.0 is final. For my use case, security risks appear minimal. I would hope that this library itself is protected against JSON-injection attacks?

I am however very concerned about readability and quick editing. Therefore, I might implement a registry that ignores tag: URIs, fetches https: URIs, and caches the files somewhere in the Python package, so that users do not need an internet connection.

[Julian]

For the latter you can have a look at https://referencing.readthedocs.io/en/stable/schema-packages/ or the jsonschema-specifications package that it links, which essentially do what you're describing for the "official" metaschemas.

[joooeey]

Not quite. I don't see where that solution fetches files from the web. Or am I missing something?

[Julian]

You download the files from the web however you'd like, the point was (I thought) to have it work without an internet connection.

[joooeey]

My wish is to remove that downloading workload from the developer anytime the schema is modified and re-versioned. It's not just about workload but also about traceability and consistency. It's meta-automation.

[Julian]

OK, this has now been re-enabled (with specific deprecation warnings for the behavior even if you weren't configuring RefResolver as in your example).

It will stick around until RefResolver is removed (at which point we'll fall back to referencing.Registry's behavior which is to not do this retrieval).