0.12.0 regression regarding usability of validation messages

[zupo]

Hey!

I'm the author of https://github.com/Pylons/pyramid_openapi3, and today I realized a new release of openapi-core is out, yay!

I immediately prepared a PR to bump pyramid_openapi3 to openapi-core 0.12.0, but tests on CI failed:

======================================================================
FAIL: test_name_too_short (app.FunctionalTests)
A name that is too short is picked up by openapi-core validation.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/circleci/repo/examples/singlefile/app.py", line 142, in test_name_too_short
    res.text,
AssertionError: 'Invalid parameter value for `name`: Value is shorter (2) than the minimum length of 3' not found in '400 Bad Request\n\nRequest validation failed.\n\n\nInvalid parameter value for `name`: Value not valid for schema\n\n'

----------------------------------------------------------------------

The test above fails in a single-file example that I ship with pyramid_openapi3. It fails because in 0.11.0, the validation error is more descriptive than in 0.12.0:

• 0.11.0: "Invalid parameter value for `name`: Value is shorter (2) than the minimum length of 3"
• 0.12.0: "Invalid parameter value for `name`: Value not valid for schema"

Is this an expected regression due to the move to a different validation engine that happened in 0.12.0 (or at least that is how I understood the commit messages)? Is it possible to configure the new validation to have more descriptive errors? Or am I completely missing the point here?

[p1c2u]

Hi @zupo . That's correct. 0.12 comes with new OAS Validator which will allow to return more detailed errors and even json schema validation.
More descriptive errors will come with new releases of 0.12.x branch.

[zupo]

Fantastic! If you want/need help testing it before release, I'm happy to run pyramid_openapi3 tests against a branch of openapi-core!

[zupo]

Hey! Massive thanks for the fix!

I created a new PR that uses openapi-core master, here are my findings:

1. Validation error messages are more verbose than before. I wouldn't say they are nicer, but they are certainly informative:

-            "Invalid parameter value for `name`: Value is shorter (2) "
-            "than the minimum length of 3",
+            "Invalid parameter value for `name`: "
+            "Value yo not valid for schema of type SchemaType.STRING: "
+            "[<ValidationError: \"'yo' is too short\">]\n\n",

https://github.com/Pylons/pyramid_openapi3/pull/43/files#r337661719

2. When I try to catch the RequestValidationError to return a custom JSON instead of "validation error as plain string" I can't seem to get the information of what the actual error was. self.schema_errors is [] when I catch the error.

-   'message': "Value {'title': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'} not "
-              'valid for schema of type SchemaType.OBJECT: []'}]
+   'field': 'title',
+   'message': 'Invalid schema property title: Value is longer (41) than the '
+              'maximum length of 40'}]

https://app.circleci.com/jobs/github/Pylons/pyramid_openapi3/178

Any idea why self.schema_errors would be empty?

[p1c2u]

Hi @zupo .

1. Most exceptions come from jsonschema validator. There is possibility to override some of them to be more descriptive. I will take a look at minItems and maxItems.
2. Good spot. schema_errors stores generator which becomes empty after evaluation in __str__(). I need to use lazy object instead.

Thanks for help testing it.

[zupo]

Yup, fixed indeed, works great with the latest master! Any ETA on a new release yet? Can I help in any way?

[zupo]

Hey @p1c2u, can I in any way help cutting a 0.12.1 release that incorporates your fixes in #170? My PyPI handle is zupo and I've been releasing packages to PyPI since 2010. I'll do my best to avoid a brown-bag release.

[zupo]

Hey again @p1c2u! I love the recent changes in master and I can't wait to be able to update pyramid_openapi3 to the latest openapi-core. But I can't do it until there is the next release and so pyramid_openapi3 users are stuck with 0.11.1 which is now almost half a year old :(. I can't upgrade to 0.12.0 because of the regression reported in this ticket.

Can I in any way help with getting changes in master released? I can offer time and/or money.

[p1c2u]

@zupo hey, I had to refactor couple of parts. I plan to release it shortly.