Skip to content

INTPYTHON-676 Add optional signing of cache data #336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

linted
Copy link

@linted linted commented Jul 10, 2025

Add HMAC signing of pickled cache data. This implementation uses Django's built in HMAC signer to avoid introducing new libraries. HMAC will introduce some overhead to performance, but for small and medium sized cache entries the impact should be minimal. The feature is easily disabled by setting "ENABLE_SIGNING" = False within the CACHE configuration.

Introduced two new cache config options:

  • ENABLE_SIGNING - boolean value to turn HMAC signing on or off. Defaults to True (on)
  • SALT - optional string to salt HMAC signatures with

@aclark4life aclark4life requested review from timgraham and WaVEV July 10, 2025 18:07
@aclark4life
Copy link
Collaborator

Thank you @linted ! Looks like obj is unexpectedly an int here …

======================================================================
ERROR: test_cache_versioning_add (cache_.tests.CacheTests.test_cache_versioning_add)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/alexclark/Developer/django-mongodb-cli/src/django-mongodb-backend/tests/cache_/tests.py", line 613, in test_cache_versioning_add
    self.assertIs(cache.add("answer1", 42, version=2), True)
                  ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/django-mongodb-backend/django_mongodb_backend/cache.py", line 131, in add
    "value": self.serializer.dumps(value),
             ~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/django-mongodb-backend/django_mongodb_backend/cache.py", line 24, in dumps
    return obj if self.signer is None else self.signer.sign(b64encode(obj))
                                                            ~~~~~~~~~^^^^^
  File "/Users/alexclark/.pyenv/versions/3.13.3/lib/python3.13/base64.py", line 58, in b64encode
    encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'int'

@Jibola Jibola requested a review from aclark4life July 11, 2025 16:01

def dumps(self, obj):
# For better incr() and decr() atomicity, don't pickle integers.
# Using type() rather than isinstance() matches only integers and not
# subclasses like bool.
if type(obj) is int: # noqa: E721
return obj
return pickle.dumps(obj, self.protocol)
return obj if self.signer is None else self.signer.sign(b64encode(obj))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't need to be b64encoded. On line 29/30, you can first unsign, and then check if the type after unsigning comes back as an int. If it does, then you can just return the int.

Comment on lines 29 to 30
if self.signer is not None:
data = b64decode(self.signer.unsign(data))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if self.signer is not None:
data = b64decode(self.signer.unsign(data))
if self.signer is not None:
if type(unsigned_data := self.signer.unsign(data)) is int:
data = unsigned_data
else:
data = b64decode(unsigned_data)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered this when I was originally writing the function, but 1234 is valid base64 and int. So, it would be ambiguous as to which was actually being stored. I fixed the issue by simply not signing int values. I do not believe this to introduce any real problems and solves the $inc test cases which were failing.

Comment on lines +99 to +100
except (BadSignature, TypeError):
self.delete(row["key"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm having trouble understanding why we're catching BadSignature and TypeError issues. What would trigger this and why would it be okay to delete a row in these instances?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the data in the cache collection is tampered with or somehow the HMAC secret key or salt is changed, a BadSignature error is thrown. I added error handling here to delete the row otherwise the exception will DOS the page until the cache entry gets culled. TypeError can be thrown if you switch from ENABLE_SIGNING=True to ENABLE_SIGNING=False without clearing all cache entries. It's highly unlikely that will ever happen in prod, but I was able to get that error while in changing settings in debug mode.

try:
results[keys_map[row["key"]]] = self.serializer.loads(row["value"])
except (BadSignature, TypeError):
self.delete(row["key"])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A better behavior than silently deleting bad data (which could happen in what circumstances besides an attacker putting malicious data in the cache?) could be to raise (or at least log) SuspiciousOperation. There is some precedent in Django for this.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained in a above comment the cases I was able to create where these two exceptions were thrown. If we do not delete the entry, won't we just create a DOS of the affected page until the cache entry is culled?

I do agree that this probably shouldn't be silent, but throwing an error here will stop the request from being handled, generate a 500, and require the request to be resent. I think that is probably ok if we delete the offending cache entry so only one request would be affected by the issue. What do you think?

linted added 2 commits July 11, 2025 12:29
…ce signed values contain characters which won't allow for conversion to int and ints wouldn't pass signature validation before being unpickled
@timgraham timgraham changed the title HMAC cache entry signing INTPYTHON-676 Add optional signing of cache data Jul 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants