Context expressions - Add support for Scan and Query using LINQ #3872

irina-herciu · 2025-06-11T16:14:30Z

This pull request introduces native support for LINQ expression trees in the IDynamoDBContext API for ScanAsync() and QueryAsync().

Description

Implement support to automatically translate LINQ expression trees to DynamoDB Expression

Example translation:
u => u.Age > 21 && u.Status == "Active"
becomes a ExpressionStatement like: #C0 > :C0 AND #C1 = :C1
with appropriate ExpressionAttributeNames and ExpressionAttributeValues.

Introduce ContextExpression that will be used to define filter LINQ expressions as follows:

var contextExpression = new ContextExpression();
contextExpression.SetFilter<Product>(p => p.CompanyInfo.Name == "Acme");

Add in LinqDdbExtensions methods to be used inside expression trees to generate DynamoDB Expression functions as follows:

Between
AttributeExists
AttributeNotExists
AttributeType

Extend IDynamoDBContext with a new overloads for Scan API:

IEnumerable<T> Scan<T>(ContextExpression filterExpression);
IEnumerable<T> Scan<T>(ContextExpression filterExpression, ScanConfig scanConfig);

Extend QueryConfig to allow setting ExpressionFilter as ContextExpression.

var resultExpressionFilter = Context.Query<Employee>("Diane", new QueryConfig
            {
                ExpressionFilter = contextExpression 
            }).ToList();

Motivation and Context

Testing

unit and integration tests added
benchmarking between filters methods and LINQ expression tree ones

Screenshots (if appropriate)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

My code follows the code style of this project
My change requires a change to the documentation
I have updated the documentation accordingly
I have read the README document
I have added tests to cover my changes
All new and existing tests passed

License

I confirm that this pull request can be released under the Apache 2 license

normj · 2025-06-18T23:56:45Z

sdk/src/Services/DynamoDBv2/Custom/DataModel/Configs.cs

+        /// <remarks>
+        /// Note: Conditions must be against non-key properties.
+        /// </remarks>
+        public ContextExpression Expression { get; set; }


We should be consistent also make the Expression available on ScanConfig. I see currently it is on DynamoDBOperationConfig and QueryConfig.

Also since you added the Scan overloads that take in the expression seems we should also add the overloads for Query.

for Query there is no overload that allows passing the QueryFilter as 'List' as arguments, as it is for Scan.

Also looks like QueryFilter from ScanConfig is ignored at this moment. Should Expression follow the same pattern and be added to config just for consistency?

Are you sure QueryFilter is being ignored for Scan? I see it being copied over in the ToDynamoDBOperationConfig operation. If it is not being used that seems like a bug.

I saw we didn't have a list of arguments overload and I'm not sure why. It seems like from a discoverability point of view it makes sense to get the feature more upfront to users as an overload. Besides having yet another overload is there a downside?

I've added a integration test for this scenario TestContext_ScanConfigFilter in 'DataModelTests', and at this point it fails. To be sure my changes did not affect the existing behavior the same test I executed on main and also fails.

I will push a fix to this issue in this PR.

As for having both an overload and the config, apart form what should happen when both 'Expression's are provided I am not a big fan of having too many arguments in public interfaces and in Query method case, for 'IEnumerable Query(object hashKeyValue, QueryOperator op, IEnumerable values, QueryConfig queryConfig);' will look like 'IEnumerable Query(object hashKeyValue, QueryOperator op, IEnumerable values, ContextExpression filterExpression, QueryConfig queryConfig);' and this just just for visibility.

sdk/src/Services/DynamoDBv2/Custom/DataModel/ContextExpression.cs

normj

We need to get the Native AOT fixed and tested.

normj · 2025-06-20T01:33:42Z

sdk/src/Services/DynamoDBv2/Custom/DataModel/ContextExpression.cs

+            if (value != null)
+            {
+                // Use reflection to get the value of the member
+                var memberInfo = value.GetType().GetMember(memberName, BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic).FirstOrDefault();


Suspecting this would cause problems for Native AOT I made GetConstant public and then create a console application with PublishAot set to true.

using Amazon.DynamoDBv2.DataModel; using System.Linq.Expressions; var constExpr = Expression.Constant(42); var result = ContextExpressionsUtils.GetConstant(constExpr);

When I ran dotnet publish I got Native AOT trim warnings on this.

> dotnet publish Restore complete (1.1s) You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy AWSSDK.Core.NetStandard net8.0 succeeded (3.3s) → C:\codebase\v4\aws-sdk-net-v4\sdk\src\Core\bin\Release\net8.0\AWSSDK.Core.dll AWSSDK.DynamoDBv2.NetStandard net8.0 succeeded (2.8s) → C:\codebase\v4\aws-sdk-net-v4\sdk\src\Services\DynamoDBv2\bin\Release\net8.0\AWSSDK.DynamoDBv2.dll DDBNativeAOTTest succeeded with 1 warning(s) (6.3s) → bin\Release\net8.0\win-x64\publish\ C:\codebase\v4\aws-sdk-net-v4\sdk\src\Services\DynamoDBv2\Custom\DataModel\ContextExpression.cs(167): Trim analysis warning IL2075: Amazon.DynamoDBv2.DataModel.ContextExpressionsUtils.GetConstantFromMember(MemberExpression): 'this' argument does not satisfy 'DynamicallyAccessedMemberTypes.PublicConstructors', 'DynamicallyAccessedMemberTypes.NonPublicConstructors', 'DynamicallyAccessedMemberTypes.PublicMethods', 'DynamicallyAccessedMemberTypes.NonPublicMethods', 'DynamicallyAccessedMemberTypes.PublicFields', 'DynamicallyAccessedMemberTypes.NonPublicFields', 'DynamicallyAccessedMemberTypes.PublicNestedTypes', 'DynamicallyAccessedMemberTypes.NonPublicNestedTypes', 'DynamicallyAccessedMemberTypes.PublicProperties', 'DynamicallyAccessedMemberTypes.NonPublicProperties', 'DynamicallyAccessedMemberTypes.PublicEvents', 'DynamicallyAccessedMemberTypes.NonPublicEvents' in call to 'System.Type.GetMember(String,BindingFlags)'. The return value of method 'System.Object.GetType()' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to.

This is the tricky thing of Native AOT is the checks done in Roslyn are not as extensive compared to actually doing a dotnet publish. Given this PR is doing quite a bit of .NET typesystem/reflection work we need to make sure we do some Native AOT compilation exercising the work.

In your next revision of the PR can you include in the PR description your sample console application that you compiled for Native AOT so we can tell what code paths have been exercised.

I've reimplemented the GetConstant to no longer use reflection and method 'System.Object.GetType()'
console app used to test it is here: TestNativeConsole

normj · 2025-06-20T01:38:53Z

sdk/src/Services/DynamoDBv2/Custom/DataModel/Configs.cs

+        /// <remarks>
+        /// Note: Conditions must be against non-key properties.
+        /// </remarks>
+        public ContextExpression Expression { get; set; }


Are you sure QueryFilter is being ignored for Scan? I see it being copied over in the ToDynamoDBOperationConfig operation. If it is not being used that seems like a bug.

I saw we didn't have a list of arguments overload and I'm not sure why. It seems like from a discoverability point of view it makes sense to get the feature more upfront to users as an overload. Besides having yet another overload is there a downside?

irina-herciu added 9 commits June 5, 2025 16:42

implement query with expressions - wip

030aba4

wip

f2418a0

refactoring

26753ae

unit tests

c59a0a9

merge from development

d416dd8

unit tests on context internal

66bd10a

add changeLog Messages

9e07b49

small refactoring

3865c1a

clenup

4f93f28

boblodgett requested a review from normj June 18, 2025 17:35

normj requested changes Jun 19, 2025

View reviewed changes

increase test coverage and address PR feedback

d7293c9

normj requested changes Jun 20, 2025

View reviewed changes

irina-herciu added 4 commits June 20, 2025 10:24

add unit test for QueryFilter usage on Scan methods

11c48e8

fix QueryFilter usage from scanConfig

0955526

fix native aot for dictionary path

94a65b5

fix native AOT worning on GetConstant

bfe8642

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Context expressions - Add support for Scan and Query using LINQ #3872

Context expressions - Add support for Scan and Query using LINQ #3872

Uh oh!

irina-herciu commented Jun 11, 2025 •

edited

Loading

Uh oh!

normj Jun 18, 2025

Uh oh!

irina-herciu Jun 19, 2025

Uh oh!

normj Jun 20, 2025

Uh oh!

irina-herciu Jun 20, 2025

Uh oh!

Uh oh!

Uh oh!

normj left a comment

Uh oh!

normj Jun 20, 2025

Uh oh!

irina-herciu Jun 23, 2025

Uh oh!

normj Jun 20, 2025

Uh oh!

Uh oh!

Context expressions - Add support for Scan and Query using LINQ #3872

Are you sure you want to change the base?

Context expressions - Add support for Scan and Query using LINQ #3872

Uh oh!

Conversation

irina-herciu commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Testing

Screenshots (if appropriate)

Types of changes

Checklist

License

Uh oh!

normj Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

irina-herciu Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

normj Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

irina-herciu Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

normj left a comment

Choose a reason for hiding this comment

Uh oh!

normj Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

irina-herciu Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

normj Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

irina-herciu commented Jun 11, 2025 •

edited

Loading