Skip to content

Commit 80c725e

Browse files
feat: json_object_hook and serializer example (#294)
This adds a new feature, the ability to customize the json deserialization into objects using a json.loads object hook. it also updates documentation and includes an example of customizing serialization and deserialization of python objects #289
1 parent 5a94170 commit 80c725e

File tree

5 files changed

+203
-6
lines changed

5 files changed

+203
-6
lines changed

docs/serialization.rst

+3-2
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Complex data types
1616
------------------
1717
Complex data types are dumped to json with json.dumps().
1818

19-
Custom serialization is possible by overriding the serialize_partially and deserialize_partially methods in `AbstractModel <https://github.com/andrewthetechie/pydantic-aioredis/blob/main/pydantic_aioredis/abstract.py#L32>`_.
19+
Custom serialization is possible using `json_default <https://docs.python.org/3/library/json.html#:~:text=not%20None.-,If%20specified%2C%20default%20should%20be%20a%20function%20that%20gets%20called%20for%20objects%20that%20can%E2%80%99t%20otherwise%20be%20serialized.%20It%20should%20return%20a%20JSON%20encodable%20version%20of%20the%20object%20or%20raise%20a%20TypeError.%20If%20not%20specified%2C%20TypeError%20is%20raised.,-If%20sort_keys%20is>`_ and `json_object_hook <https://docs.python.org/3/library/json.html#:~:text=object_hook%20is%20an%20optional%20function%20that%20will%20be%20called%20with%20the%20result%20of%20any%20object%20literal%20decoded%20(a%20dict).%20The%20return%20value%20of%20object_hook%20will%20be%20used%20instead%20of%20the%20dict.%20This%20feature%20can%20be%20used%20to%20implement%20custom%20decoders%20(e.g.%20JSON%2DRPC%20class%20hinting).>`_.
2020

21-
It is also possilbe to override json_default in AbstractModel. json_default is a callable used to convert any objects of a type json.dump cannot natively dump to string.
21+
These methods are part of the `abstract model <https://github.com/andrewthetechie/pydantic-aioredis/blob/main/pydantic_aioredis/abstract.py#L77>`_ and can be overridden in your
22+
model to dump custom objects to json and then back to objects. An example is available in `examples <https://github.com/andrewthetechie/pydantic-aioredis/tree/main/examples/serializer>`_

examples/serializer/Makefile

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
start-redis: ## Runs a copy of redis in docker
2+
docker run -it -d --rm --name pydantic-aioredis-example -p 6379:6379 -e REDIS_PASSWORD=password bitnami/redis || echo "$(REDIS_CONTAINER_NAME) is either running or failed"
3+
4+
stop-redis: ## Stops the redis in docker
5+
docker stop pydantic-aioredis-example

examples/serializer/README.md

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# asyncio_example
2+
3+
This is a working example using python-aioredis with asyncio and a custom serializer for a python object BookCover.
4+
5+
Book.json_default is used to serialize the BookCover object to a dictionary that json.dumps can dump to a string and store in redis.
6+
Book.json_object_hook can convert a dict from redis back to a BookCover object.
7+
8+
# Requirements
9+
10+
This example requires a running redis server. You can change the RedisConfig on line 28 in the example to match connecting to your running redis.
11+
12+
For your ease of use, we've provided a Makefile in this directory that can start and stop a redis using docker.
13+
14+
`make start-redis`
15+
16+
`make stop-redis`
17+
18+
The example is configured to connect to this dockerized redis automatically
19+
20+
# Expected Output
21+
22+
This is a working example. If you try to run it and find it broken, first check your local env. If you are unable to get the
23+
example running, please raise an Issue
24+
25+
```bash
26+
python custom_serializer.py
27+
[Book(title='Great Expectations', author='Charles Dickens', published_on=datetime.date(1220, 4, 4), cover=<__main__.BookCover object at 0x10410c4c0>), Book(title='Jane Eyre', author='Charlotte Bronte', published_on=datetime.date(1225, 6, 4), cover=<__main__.BookCover object at 0x10410d4e0>), Book(title='Moby Dick', author='Herman Melville', published_on=datetime.date(1851, 10, 18), cover=<__main__.BookCover object at 0x10410d060>), Book(title='Oliver Twist', author='Charles Dickens', published_on=datetime.date(1215, 4, 4), cover=<__main__.BookCover object at 0x10410c760>), Book(title='Wuthering Heights', author='Emily Bronte', published_on=datetime.date(1600, 4, 4), cover=<__main__.BookCover object at 0x10410d690>)]
28+
[Book(title='Jane Eyre', author='Charlotte Bronte', published_on=datetime.date(1225, 6, 4), cover=<__main__.BookCover object at 0x10410cdc0>), Book(title='Oliver Twist', author='Charles Dickens', published_on=datetime.date(1215, 4, 4), cover=<__main__.BookCover object at 0x10410d7e0>)]
29+
[{'author': 'Charles Dickens', 'cover': <__main__.BookCover object at 0x10410d7b0>}, {'author': 'Charlotte Bronte', 'cover': <__main__.BookCover object at 0x10410d8d0>}, {'author': 'Herman Melville', 'cover': <__main__.BookCover object at 0x10410d840>}, {'author': 'Charles Dickens', 'cover': <__main__.BookCover object at 0x10410d960>}, {'author': 'Emily Bronte', 'cover': <__main__.BookCover object at 0x10410d900>}]
30+
```
+156
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
import asyncio
2+
import json
3+
from datetime import date
4+
from datetime import datetime
5+
from typing import Any
6+
from typing import Dict
7+
from typing import List
8+
from typing import Optional
9+
10+
from pydantic_aioredis import Model
11+
from pydantic_aioredis import RedisConfig
12+
from pydantic_aioredis import Store
13+
from pydantic_aioredis.abstract import STR_DUMP_SHAPES
14+
15+
16+
class BookCover:
17+
def __init__(self, cover_url: int, cover_size_x: int, cover_size_y: int):
18+
self.cover_url = cover_url
19+
self.cover_size_x = cover_size_x
20+
self.cover_size_y = cover_size_y
21+
22+
@property
23+
def area(self):
24+
return self.cover_size_x * self.cover_size_y
25+
26+
27+
# Create models as you would create pydantic models i.e. using typings
28+
class Book(Model):
29+
_primary_key_field: str = "title"
30+
title: str
31+
author: str
32+
published_on: date
33+
cover: BookCover
34+
35+
@classmethod
36+
def json_default(cls, obj: Any) -> str:
37+
"""Since BookCover can't be directly json serialized, we have to write our own json_default to serialize it methods to handle it."""
38+
if isinstance(obj, BookCover):
39+
return {
40+
"__BookCover__": True,
41+
"cover_url": obj.cover_url,
42+
"cover_size_x": obj.cover_size_x,
43+
"cover_size_y": obj.cover_size_y,
44+
}
45+
46+
return super().json_default(obj)
47+
48+
@classmethod
49+
def json_object_hook(cls, obj: dict):
50+
"""Since we're serializing BookCovers above, we need to write an object hook to turn them back into an Object"""
51+
if obj.get("__BookCover__", False):
52+
return BookCover(
53+
cover_url=obj["cover_url"],
54+
cover_size_x=obj["cover_size_x"],
55+
cover_size_y=obj["cover_size_y"],
56+
)
57+
super().json_object_hook(obj)
58+
59+
60+
# Redisconfig. Change this configuration to match your redis server
61+
redis_config = RedisConfig(
62+
db=5, host="localhost", password="password", ssl=False, port=6379
63+
)
64+
65+
66+
# Create the store and register your models
67+
store = Store(name="some_name", redis_config=redis_config, life_span_in_seconds=3600)
68+
store.register_model(Book)
69+
70+
71+
# Sample books. You can create as many as you wish anywhere in the code
72+
books = [
73+
Book(
74+
title="Oliver Twist",
75+
author="Charles Dickens",
76+
published_on=date(year=1215, month=4, day=4),
77+
cover=BookCover(
78+
"https://images-na.ssl-images-amazon.com/images/I/51SmEM7LUGL._SX342_SY445_QL70_FMwebp_.jpg",
79+
333,
80+
499,
81+
),
82+
),
83+
Book(
84+
title="Great Expectations",
85+
author="Charles Dickens",
86+
published_on=date(year=1220, month=4, day=4),
87+
cover=BookCover(
88+
"https://images-na.ssl-images-amazon.com/images/I/51i715XqsYL._SX311_BO1,204,203,200_.jpg",
89+
333,
90+
499,
91+
),
92+
),
93+
Book(
94+
title="Jane Eyre",
95+
author="Charlotte Bronte",
96+
published_on=date(year=1225, month=6, day=4),
97+
cover=BookCover(
98+
"https://images-na.ssl-images-amazon.com/images/I/41saarVx+GL._SX324_BO1,204,203,200_.jpg",
99+
333,
100+
499,
101+
),
102+
),
103+
Book(
104+
title="Wuthering Heights",
105+
author="Emily Bronte",
106+
published_on=date(year=1600, month=4, day=4),
107+
cover=BookCover(
108+
"https://images-na.ssl-images-amazon.com/images/I/51ZKox7zBKL._SX338_BO1,204,203,200_.jpg",
109+
333,
110+
499,
111+
),
112+
),
113+
]
114+
115+
116+
async def work_with_orm():
117+
# Insert them into redis
118+
await Book.insert(books)
119+
120+
# Select all books to view them. A list of Model instances will be returned
121+
all_books = await Book.select()
122+
print(all_books) # Will print [Book(title="Oliver Twist", author="Charles Dickens",
123+
# published_on=date(year=1215, month=4, day=4), in_stock=False), Book(...]
124+
125+
# Or select some of the books
126+
some_books = await Book.select(ids=["Oliver Twist", "Jane Eyre"])
127+
print(some_books) # Will return only those two books
128+
129+
# Or select some of the columns. THIS RETURNS DICTIONARIES not MODEL Instances
130+
# The Dictionaries have values in string form so you might need to do some extra work
131+
books_with_few_fields = await Book.select(columns=["author", "cover"])
132+
print(
133+
books_with_few_fields
134+
) # Will print [{"author": "'Charles Dickens", "covker": Cover},...]
135+
136+
# When _auto_sync = True (default), updating any attribute will update that field in Redis too
137+
this_book = Book(
138+
title="Moby Dick",
139+
author="Herman Melvill",
140+
published_on=date(year=1851, month=10, day=18),
141+
cover=BookCover(
142+
"https://m.media-amazon.com/images/I/411a8Moy1mL._SY346_.jpg", 333, 499
143+
),
144+
)
145+
await Book.insert(this_book)
146+
# oops, there was a typo. Fix it
147+
this_book.author = "Herman Melville"
148+
this_book_from_redis = await Book.select(ids=["Moby Dick"])
149+
assert this_book_from_redis[0].author == "Herman Melville"
150+
151+
# If you have _auto_save set to false on a model, you have to await .save() to update a model in tedis
152+
await this_book.save()
153+
154+
155+
if __name__ == "__main__":
156+
asyncio.run(work_with_orm())

pydantic_aioredis/abstract.py

+9-4
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,13 @@ class _AbstractModel(BaseModel):
7373
_table_name: Optional[str] = None
7474
_auto_sync: bool = True
7575

76-
@staticmethod
77-
def json_default(obj: Any) -> str:
76+
@classmethod
77+
def json_object_hook(cls, obj: dict):
78+
"""Can be overridden to handle custom json -> object"""
79+
return obj
80+
81+
@classmethod
82+
def json_default(cls, obj: Any) -> str:
7883
"""
7984
JSON serializer for objects not serializable by default json library
8085
Currently handles: datetimes -> obj.isoformat, ipaddress and ipnetwork -> str
@@ -127,9 +132,9 @@ def deserialize_partially(cls, data: Dict[bytes, Any]):
127132
if field not in columns:
128133
continue
129134
if cls.__fields__[field].type_ not in [str, float, int]:
130-
data[field] = json.loads(data[field])
135+
data[field] = json.loads(data[field], object_hook=cls.json_object_hook)
131136
if getattr(cls.__fields__[field], "shape", None) in JSON_DUMP_SHAPES:
132-
data[field] = json.loads(data[field])
137+
data[field] = json.loads(data[field], object_hook=cls.json_object_hook)
133138
if getattr(cls.__fields__[field], "allow_none", False):
134139
if data[field] == "None":
135140
data[field] = None

0 commit comments

Comments
 (0)