INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

(see previous attempts in #318, #319 and #323 for additional context)

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

aclark4life · 2025-06-27T18:08:35Z

Wrong commit message for 65bd15a and I don't want to force push yet. It should have said:

"Only create an encrypted connection once then reuse it."

I'm aware that _nodb_cursor is slated for removal but in the meantime I can keep going with other fixes with this approach, and it does satisfy the design we all agree on (I think) of maintaining two simultaneous connections:

Unencrypted connection unless we need it
Encrypted connection when we need that can be used.

timgraham · 2025-06-27T20:05:16Z

I'm aware that _nodb_cursor is slated for removal but in the meantime I can keep going with other fixes with this approach, and it does satisfy the design we all agree on (I think) of maintaining two simultaneous connections:

It's not working as you think it is. As I said elsewhere, _nodb_cursor is not used by this backend.

Does this fix the "command not supported for auto encryption: buildinfo" error? If so, it's perhaps because self.settings_dict["OPTIONS"].pop("auto_encryption_opts") is having the side effect of altering settings_dict before DatabaseWrapper.connection is initialized.

I'd suggest to use my patch is as a starting point for maintaining two connections. self.connection should be the encrypted version (secure by default) with a fallback to a non-encrypted connection only as needed (e.g. for commands like buildInfo). At least it will help us understand whether that's a viable approach. As I mentioned in the design doc, I'm not sure if using an encrypted connection for non-encrypted collections is problematic. If so, we'll have to go back to the drawing board on the design.

aclark4life · 2025-06-27T21:51:58Z

It's not working as you think it is. As I said elsewhere, _nodb_cursor is not used by this backend.

I don't disagree, but it feels a lot like _start_transaction_under_autocommit which gets called by start_transaction_under_autocommit because autocommit is False. Django appears to stumble into _nodb_cursor when the encrypted connection fails to get the database version and while we don't use a cursor in this backend, we do have a "nosql" cursor that has __enter__ and __exit__ (I assume) to meet Django's expectations and we get an opportunity to modify the connection. @Jibola mentioned this design is suspect yesterday and I agree with both of you, particularly with regard to the desire to start with and maintain an encrypted connection first.

Does this fix the "command not supported for auto encryption: buildinfo" error? If so, it's perhaps because self.settings_dict["OPTIONS"].pop("auto_encryption_opts") is having the side effect of altering settings_dict before DatabaseWrapper.connection is initialized.

Yes it works by design, not a side effect. I'm deep.copying settings_dict when DatabaseWrapper is initialized and so when DatabaseWrapper.connection is initialized it's unencrypted. When the schema needs encryption later, it's retrieved from _settings_dict.

I'd suggest to use my patch is as a starting point for maintaining two connections. self.connection should be the encrypted version (secure by default) with a fallback to a non-encrypted connection only as needed (e.g. for commands like buildInfo). At least it will help us understand whether that's a viable approach. As I mentioned in the design doc, I'm not sure if using an encrypted connection for non-encrypted collections is problematic. If so, we'll have to go back to the drawing board on the design.

I'd make a few passes at it but did not get anywhere, I'll try again though.

timgraham · 2025-06-27T22:12:08Z

Your "stumble" theory of how it's working isn't correct. _nodb_cursor is only used on one place: to create the test database. As I said, I could imagine that perhaps this method causes connection to later be initialized without auto_encryption_opts because of self.settings_dict["OPTIONS"].pop("auto_encryption_opts"). The connection that's created in your _nodb_cursor is never used.

aclark4life · 2025-06-28T00:10:12Z

Your "stumble" theory of how it's working isn't correct. _nodb_cursor is only used on one place: to create the test database. As I said, I could imagine that perhaps this method causes connection to later be initialized without auto_encryption_opts because of self.settings_dict["OPTIONS"].pop("auto_encryption_opts"). The connection that's created in your _nodb_cursor is never used.

Copy that, thanks!

I've removed _nodb_cursor in 8e83ada and discovered the version check is the only time that error occurs. I now get errors like:

Traceback (most recent call last):
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 124, in _wrap_encryption_errors
    yield
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 466, in encrypt
    encrypted_cmd = self._auto_encrypter.encrypt(database, encoded_cmd)
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/auto_encrypter.py", line 44, in encrypt
    return run_state_machine(ctx, self.callback)
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/state_machine.py", line 136, in run_state_machine
    result = callback.mark_command(ctx.database, mongocryptd_cmd)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 286, in mark_command
    res = self.mongocryptd_client[database].command(
        inflated_cmd, codec_options=DEFAULT_RAW_BSON_OPTIONS
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 930, in command
    return self._command(
           ~~~~~~~~~~~~~^
        connection,
        ^^^^^^^^^^^
    ...<7 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 770, in _command
    return conn.command(
           ~~~~~~~~~~~~^
        self._name,
        ^^^^^^^^^^^
    ...<8 lines>...
        client=self._client,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/helpers.py", line 47, in inner
    return func(*args, **kwargs)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/pool.py", line 414, in command
    return command(
        self,
    ...<20 lines>...
        write_concern=write_concern,
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/network.py", line 212, in command
    helpers_shared._check_command_response(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        response_doc,
        ^^^^^^^^^^^^^
    ...<2 lines>...
        parse_write_concern_error=parse_write_concern_error,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/helpers_shared.py", line 250, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection., full error: RawBSONDocument(b"\xa7\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00d\x00\x00\x00Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection.\x00\x10code\x00\x08\xc8\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location51208\x00\x00", codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME))

Still working on an unencrypted connection, but perhaps the only time we need it is for the version check.

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

- Refactor tests

django_mongodb_backend/schema.py

aclark4life · 2025-07-02T02:49:19Z

@ShaneHarvey @Jibola @timgraham FYI here is the pipeline that causes the let error:

(Pdb) pprint.pprint(pipeline)
[{'$lookup': {'as': 'django_content_type',
              'from': 'django_content_type',
              'let': {'parent__field__0': '$content_type_id'},
              'pipeline': [{'$match': {'$expr': {'$and': [{'$eq': ['$$parent__field__0',
                                                                   '$_id']}]}}}]}},
 {'$unwind': '$django_content_type'},
 {'$match': {'$expr': {'$in': ['$content_type_id',
                               (ObjectId('6864933ec7cf8179e3ef1f8d'),)]}}},
 {'$project': {'codename': 1,
               'content_type_id': 1,
               'django_content_type': {'app_label': 1, 'model': 1}}},
 {'$sort': SON([('django_content_type.app_label', 1), ('django_content_type.model', 1), ('codename', 1)])}]

And here is the error again with some additional debug:

(Pdb) errmsg
"Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection."
(Pdb) code
51208
(Pdb) response
RawBSONDocument(b"\xa7\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00d\x00\x00\x00Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection.\x00\x10code\x00\x08\xc8\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location51208\x00\x00", codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME))
(Pdb) max_wire_version
26

And the full traceback:


Running post-migrate handlers for application contenttypes
Traceback (most recent call last):
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 124, in _wrap_encryption_errors
    yield
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 466, in encrypt
    encrypted_cmd = self._auto_encrypter.encrypt(database, encoded_cmd)
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/auto_encrypter.py", line 44, in encrypt
    return run_state_machine(ctx, self.callback)
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/state_machine.py", line 136, in run_state_machine
    result = callback.mark_command(ctx.database, mongocryptd_cmd)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 286, in mark_command
    res = self.mongocryptd_client[database].command(
        inflated_cmd, codec_options=DEFAULT_RAW_BSON_OPTIONS
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 930, in command
    return self._command(
           ~~~~~~~~~~~~~^
        connection,
        ^^^^^^^^^^^
    ...<7 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 770, in _command
    return conn.command(
           ~~~~~~~~~~~~^
        self._name,
        ^^^^^^^^^^^
    ...<8 lines>...
        client=self._client,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/helpers.py", line 47, in inner
    return func(*args, **kwargs)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/pool.py", line 414, in command
    return command(
        self,
    ...<20 lines>...
        write_concern=write_concern,
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/network.py", line 212, in command
    helpers_shared._check_command_response(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        response_doc,
        ^^^^^^^^^^^^^
    ...<2 lines>...
        parse_write_concern_error=parse_write_concern_error,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/helpers_shared.py", line 250, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection., full error: RawBSONDocument(b"\xa7\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00d\x00\x00\x00Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection.\x00\x10code\x00\x08\xc8\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location51208\x00\x00", codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME))

Test settings:

import os

from django_mongodb_backend import encryption, parse_uri

kms_providers = encryption.get_kms_providers()

auto_encryption_opts = encryption.get_auto_encryption_opts(
    kms_providers=kms_providers,
)

DATABASE_URL = os.environ.get("MONGODB_URI", "mongodb://localhost:27017")
DATABASES = {
    "default": parse_uri(
        DATABASE_URL, db_name="djangotests",
    ),
    "encrypted": parse_uri(
        DATABASE_URL, options={"auto_encryption_opts": auto_encryption_opts},
            db_name="encrypted_djangotests",
    ),
}

DEFAULT_AUTO_FIELD = "django_mongodb_backend.fields.ObjectIdAutoField"
PASSWORD_HASHERS = ("django.contrib.auth.hashers.MD5PasswordHasher",)
SECRET_KEY = "django_tests_secret_key"
USE_TZ = False

This is happening in the encryption_ tests with a database router configured to use the encrypted database, but it happens before any tests are run or any routing occurs. I've confirmed that the encrypted database is created, so it appears that something needs to be done to address this issue in either our backend or PyMongo with the ideal candidate, perhaps, being a change to the MQL in the pipeline if possible.

Co-authored-by: Tim Graham <[email protected]>

aclark4life · 2025-07-14T13:48:10Z

I would love to see this happen but how do we support it with content types requiring an unencrypted connection?

You can use Django without the contrib apps.

😮

Factor out field init into mixin and add int field

The main title in the left hand navigation should go to index.

timgraham · 2025-07-14T22:15:38Z

tests/encryption_/tests.py

+            )
+
+    def test_auto_encryption_opts(self):
+        management.call_command("get_encrypted_fields_map", "--database", "encrypted", verbosity=0)


I know Django tests use this format, but I think it's better to simply import call_command rather than to use management.call_command.

And you can add testing the output:

from io import StringIO out = StringIO() call_command("dance", stdout=out) self.assertIn("I don't feel like dancing Rock'n'Roll.\n", out.getvalue())

timgraham · 2025-07-14T22:27:02Z

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

+    def get_encrypted_fields_map(self, connection):
+        return {
+            "fields": [
+                field
+                for app_config in apps.get_app_configs()
+                for model in router.get_migratable_models(
+                    app_config, connection.alias, include_auto_created=False
+                )
+                if getattr(model, "encrypted", False)
+                for field in connection.schema_editor()._get_encrypted_fields_map(model)
+            ]
+        }


This format doesn't look correct. Doesn't it have to include the database and collection information? Look at the pymongo example.

Maybe. If I pass the results as schema_map to AutoEncryptionOpts I don't get an error, but I'll try with namespace.

timgraham · 2025-07-14T22:28:34Z

django_mongodb_backend/encryption.py

+    return AutoEncryptionOpts(
+        key_vault_namespace=key_vault_namespace,
+        kms_providers=kms_providers,
+        crypt_shared_lib_path=crypt_shared_lib_path,
+        schema_map=schema_map,
+    )


Perhaps this will evolve, but did you notice that get_auto_encryption_opts is currently nothing more than an alias of AutoEncryptionOpts (passing all kwargs straight through). :-D

😂 e562718

timgraham · 2025-07-14T22:49:10Z

django_mongodb_backend/encryption.py

+    def kms_credentials(self, model):
+        # return KMS_CREDENTIALS.get(provider, None)
+        return {}


My impression is that kms_credentials is more of a global setting.. it doesn't change per model, thus doesn't need a router method. Possibly it could be different for each database, so it might be appropriate to use a new setting, e.g. DATABASES["alias"]["KMS_CREDENTIALS"].

timgraham · 2025-07-14T22:50:04Z

django_mongodb_backend/encryption.py

+    def kms_provider(self, model):
+        return getattr(settings, "KMS_PROVIDER", None)


The user will provide a value rather than it be retrieved from a setting.

timgraham · 2025-07-14T22:51:01Z

docs/source/howto/encryption.rst

+``encryption.EncryptedRouter``
+------------------------------


I imagined this wouldn't be a public API. This is the global router for the Django test suite that lives in our test settings. It makes certain assumptions about database aliases that may not be true of user projects.

timgraham · 2025-07-14T22:52:42Z

django_mongodb_backend/encryption.py

+    Return a `ClientEncryption` instance for use with Queryable Encryption.
+    """
+
+    codec_options = CodecOptions(uuid_representation=STANDARD)


Our backend doesn't use uuid_representation. UUIDs are stored as strings.

timgraham · 2025-07-14T22:55:16Z

tests/backend_/test_features.py

+        connection.features.__dict__.pop("supports_queryable_encryption", None)
+
+    def tearDown(self):
+        connection.features.__dict__.pop("supports_queryable_encryption", None)


This can use the del ... version since it will exist by now.

This can use the del ... version since it will exist by now.

By now? I still get an attribute error …

WaVEV · 2025-07-15T03:55:01Z

django_mongodb_backend/encryption.py

+
+class EncryptedRouter:
+    def _get_db_for_model(self, model):
+        if getattr(model, "encrypted", False):


Sorry If I ask something that was previously discussed, but how does it know if a model was an encrypted model or a model that has a field encrypted = models.BooleanField(). Maybe with getattr(model, "encrypted", False) is True could save for some false positives.

Good catch, thanks. I don't think we've finalized using that conditional yet and more importantly there is one in schema._create_collection.

WaVEV · 2025-07-15T04:08:39Z

django_mongodb_backend/fields/encryption.py

+        return name, path, args, kwargs
+
+
+class EncryptedCharField(EncryptedFieldMixin, models.CharField):


I didn't find in the docs if an encrypted collection could have an aggregate query. So my question is:
does it support all the lookups from CharField ?

No aggregation stages are supported and two tests from Django's test_charfield are failing, though only one is an aggregation stage failure:

====================================================================== ERROR: test_assignment_from_choice_enum (encryption_.test_charfield.TestEncryptedCharField.test_assignment_from_choice_enum) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 124, in _wrap_encryption_errors yield File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 466, in encrypt encrypted_cmd = self._auto_encrypter.encrypt(database, encoded_cmd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/.venv/lib/python3.12/site-packages/pymongocrypt/synchronous/auto_encrypter.py", line 44, in encrypt return run_state_machine(ctx, self.callback) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/.venv/lib/python3.12/site-packages/pymongocrypt/synchronous/state_machine.py", line 136, in run_state_machine result = callback.mark_command(ctx.database, mongocryptd_cmd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 286, in mark_command res = self.mongocryptd_client[database].command( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/_csot.py", line 125, in csot_wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 930, in command return self._command( ^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 770, in _command return conn.command( ^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/helpers.py", line 47, in inner return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/pool.py", line 414, in command return command( ^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/network.py", line 212, in command helpers_shared._check_command_response( File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/helpers_shared.py", line 250, in _check_command_response raise OperationFailure(errmsg, code, response, max_wire_version) pymongo.errors.OperationFailure: Comparison disallowed between fields where one is randomly encrypted; field 'title' is randomly encrypted., full error: RawBSONDocument(b"\xae\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00k\x00\x00\x00Comparison disallowed between fields where one is randomly encrypted; field 'title' is randomly encrypted.\x00\x10code\x00\xb6y\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location31158\x00\x00", codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME)) ERROR: test_lookup_integer_in_charfield (encryption_.test_charfield.TestEncryptedCharField.test_lookup_integer_in_charfield) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 124, in _wrap_encryption_errors yield File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 466, in encrypt encrypted_cmd = self._auto_encrypter.encrypt(database, encoded_cmd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/.venv/lib/python3.12/site-packages/pymongocrypt/synchronous/auto_encrypter.py", line 44, in encrypt return run_state_machine(ctx, self.callback) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/.venv/lib/python3.12/site-packages/pymongocrypt/synchronous/state_machine.py", line 136, in run_state_machine result = callback.mark_command(ctx.database, mongocryptd_cmd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 286, in mark_command res = self.mongocryptd_client[database].command( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/_csot.py", line 125, in csot_wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 930, in command return self._command( ^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 770, in _command return conn.command( ^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/helpers.py", line 47, in inner return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/pool.py", line 414, in command return command( ^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/network.py", line 212, in command helpers_shared._check_command_response( File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/helpers_shared.py", line 250, in _check_command_response raise OperationFailure(errmsg, code, response, max_wire_version) pymongo.errors.OperationFailure: Aggregation stage $internalFacetTeeConsumer is not allowed or supported with automatic encryption., full error: RawBSONDocument(b'\xa6\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00c\x00\x00\x00Aggregation stage $internalFacetTeeConsumer is not allowed or supported with automatic encryption.\x00\x10code\x00#y\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location31011\x00\x00', codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME))

Only testing EncryptedIntegerField

Both checks cannot exist in the same class else one may be interrupted by the other and fail as a result. Instead, check the version once and cache the results so subsequent checks can check the cache instead of the connection.

Regression in daa9a8e

INTPYTHON-527 Add Queryable Encryption support

bc52c8e

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

Fix test for unencrypted field not in field map

38fb110

timgraham reviewed Jun 27, 2025

View reviewed changes

Fix test for unencrypted field not in field map

65bd15a

Add comment about suppressing EncryptedCollectionError

e08945b

aclark4life added 2 commits June 27, 2025 18:12

Don't rely on features to fall back to unencrypted

7b34b44

Remove _nodb_cursor and disable version check

8e83ada

aclark4life added 2 commits June 27, 2025 20:15

Don't surpress encrypted error

4da895c

Rename get_encrypted_client -> get_client_encryption

ed54a9b

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

aclark4life added 2 commits June 30, 2025 18:05

Add encryption router

8a7766c

Add "encryption" database to encryption tests

eab2f2e

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

aclark4life added 2 commits July 1, 2025 15:02

Move encrypted_fields_map to schema (1/2)

10a361e

Move encrypted_fields_map to schema (2/x)

01d5485

- Refactor tests

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

aclark4life added 4 commits July 1, 2025 20:14

Refactor helpers

db32487

Restore get_database_version functionality

b2be223

Move encrypted router to tests

27d4b8e

Fix router tests

c4d1c66

Test feature supports_queryable_encryption

2772aff

Update django_mongodb_backend/features.py

acb0554

Co-authored-by: Tim Graham <[email protected]>

aclark4life added 3 commits July 14, 2025 12:06

Update test models to match design doc

67a640d

Factor out field init into mixin and add int field

Refactor management command and fix test

9e76295

Update Sphinx root_doc to index (the default value)

97196ed

The main title in the left hand navigation should go to index.

aclark4life mentioned this pull request Jul 14, 2025

PYTHON-5429 Server command is case sensitive mongodb/mongo-python-driver#2421

Open

aclark4life added 3 commits July 14, 2025 16:23

Move kms_provider to user router

1614919

Move kms_credentials to user router

a1bc5f3

Update docs

75c3cd1

timgraham reviewed Jul 14, 2025

View reviewed changes

aclark4life added 9 commits July 14, 2025 20:28

Move kms_credentials to settings

a81d2ae

Remove get_auto_encryption_opts

e562718

Remove get_client_encryption

3dca177

Define public helpers API

3432818

Refactor test_auto_encryption_opts

cb7f153

Assert the entire expected output via json.dumps

e0ef5b3

Update docs

ba4a6c8

Rename test methods

1b9a714

Fix doc string

3340ae7

WaVEV reviewed Jul 15, 2025

View reviewed changes

aclark4life and others added 9 commits July 15, 2025 07:56

Rename tests -> test_schema & add charfield tests

c90406b

Add test_integerfield from django

8a1f381

Only testing EncryptedIntegerField

Avoid reentrancy issue checking mongodb version

edb2fa6

Both checks cannot exist in the same class else one may be interrupted by the other and fail as a result. Instead, check the version once and cache the results so subsequent checks can check the cache instead of the connection.

Add encrypted Post model schema to expected schema

0701160

Re-add namespace to schema_map

08f7934

Add a note about copying field tests from Django

2c4d53b

Add query type tests, remove django field tests

9919ce9

Restored documentation navigation

45ea5b5

Regression in daa9a8e

Test negative query type cases

3e468e7

		def kms_provider(self, model):
		return getattr(settings, "KMS_PROVIDER", None)

		``encryption.EncryptedRouter``
		------------------------------

		return name, path, args, kwargs


		class EncryptedCharField(EncryptedFieldMixin, models.CharField):

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Conversation

aclark4life commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aclark4life commented Jun 27, 2025

Uh oh!

timgraham commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aclark4life commented Jun 27, 2025

Uh oh!

timgraham commented Jun 27, 2025

Uh oh!

aclark4life commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aclark4life commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aclark4life commented Jul 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aclark4life Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aclark4life Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aclark4life commented Jun 27, 2025 •

edited

Loading

timgraham commented Jun 27, 2025 •

edited

Loading

aclark4life commented Jun 28, 2025 •

edited

Loading

aclark4life commented Jul 2, 2025 •

edited

Loading

aclark4life Jul 15, 2025 •

edited

Loading

aclark4life Jul 15, 2025 •

edited

Loading