JSON decimal values should be parsed as Python decimal.Decimal in targets when reading Singer messages #344
Description
Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/346
Originally created by @edgarrmondragon on 2022-03-09 16:17:31
Summary
JSON decimal values should be parsed as Python decimal.Decimal in targets when reading Singer messages.
Steps to reproduce
Try piping a tap that outputs similar to {"type": "number", "multipleOf": "0.001"} to a target that performs JSON schema validation. Values that should pass validation will fail.
What is the current bug behavior?
Decimals are loaded as float values which may cause problems when their precision is important, e.g. when validating against JSON schema `multipleOf' .
What is the expected correct behavior?
Decimals should be loaded as decimal.Decimal to preserve precision.
Relevant logs and/or screenshots
>>> import jsonschema
>>> jsonschema.validate(48.867082, {"type": "number", "multipleOf": 0.000001})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/singer-sdk-S0flctL8-py3.9/lib/python3.9/site-packages/jsonschema/validators.py", line 934, in validate
raise error
jsonschema.exceptions.ValidationError: 48.867082 is not a multiple of 1e-06
Failed validating 'multipleOf' in schema:
{'multipleOf': 1e-06, 'type': 'number'}
On instance:
48.867082
>>> import decimal
>>> jsonschema.validate(decimal.Decimal("48.867082"), {"type": "number", "multipleOf": decimal.Decimal("0.000001")})Possible fixes
The jsonschema library dev recommends loading JSON decimals as Python decimal.Decimal: python-jsonschema/jsonschema#810 (comment).
Using json.loads(..., parse_float=decimal.Decimal) in https://gitlab.com/meltano/sdk/-/blob/b394e1a9c0d68a554640974c05be1c9d548270ee/singer_sdk/io_base.py#L77 should fix the issue.