Skip to content

Feature request: Enable TumblingWindow support for KinesisDataStreamRecord and DynamoDB #3915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 tasks done
leandrodamascena opened this issue May 12, 2025 · 9 comments · Fixed by #3931
Closed
2 tasks done
Assignees
Labels
completed This item is complete and has been merged/shipped feature-request This item refers to a feature request for an existing or new utility good-first-issue Something that is suitable for those who want to start contributing parser This item relates to the Parser Utility

Comments

@leandrodamascena
Copy link
Contributor

Use case

Original issue: aws-powertools/powertools-lambda-python#6627

To implement stateful Kinesis Data Streams, customers can enable tumbling window in their ESM configuration for Kinesis and DynamoDB stream. When they do this, the payload has some extra keys such as window, state, shardId, isFinalInvokeForWindow and isWindowTerminatedEarly.

{
    "Records": [
        {
            "kinesis": {
                "kinesisSchemaVersion": "1.0",
                "partitionKey": "1",
                "sequenceNumber": "49590338271490256608559692538361571095921575989136588898",
                "data": "SGVsbG8sIHRoaXMgaXMgYSB0ZXN0Lg==",
                "approximateArrivalTimestamp": 1607497475.000
            },
            "eventSource": "aws:kinesis",
            "eventVersion": "1.0",
            "eventID": "shardId-000000000006:49590338271490256608559692538361571095921575989136588898",
            "eventName": "aws:kinesis:record",
            "invokeIdentityArn": "arn:aws:iam::123456789012:role/lambda-kinesis-role",
            "awsRegion": "us-east-1",
            "eventSourceARN": "arn:aws:kinesis:us-east-1:123456789012:stream/lambda-stream"
        }
    ],
    "window": {
        "start": "2020-12-09T07:04:00Z",
        "end": "2020-12-09T07:06:00Z"
    },
    "state": {
        "1": 282,
        "2": 715
    },
    "shardId": "shardId-000000000006",
    "eventSourceARN": "arn:aws:kinesis:us-east-1:123456789012:stream/lambda-stream",
    "isFinalInvokeForWindow": false,
    "isWindowTerminatedEarly": false
}

While adding these fields to our zod schemas is enough to parse events, we still need to discuss - in a separate issue maybe - how to modify our BatchProcessing utility to support these scenarios, as we need to return an additional state field when using PartialFailure.

Kinesis documentation: https://docs.aws.amazon.com/lambda/latest/dg/services-kinesis-windows.html
DynamoDB documentation: https://docs.aws.amazon.com/lambda/latest/dg/services-ddb-windows.html

Solution/User Experience

Add new fields.

Alternative solutions

Acknowledgment

Future readers

Please react with 👍 and your use case to help us understand customer demand.

@leandrodamascena leandrodamascena added triage This item has not been triaged by a maintainer, please wait feature-request This item refers to a feature request for an existing or new utility labels May 12, 2025
@dreamorosi
Copy link
Contributor

I'm not following, is this an issue to add these fields to the schemas in Parser or to support this behavior with Batch Processing?

@dreamorosi dreamorosi moved this from Triage to Backlog in Powertools for AWS Lambda (TypeScript) May 12, 2025
@dreamorosi dreamorosi added discussing The issue needs to be discussed, elaborated, or refined and removed triage This item has not been triaged by a maintainer, please wait labels May 12, 2025
@leandrodamascena
Copy link
Contributor Author

Add these fields to the schemas in Parser

@dreamorosi dreamorosi added good-first-issue Something that is suitable for those who want to start contributing help-wanted We would really appreciate some support from community for this one confirmed The scope is clear, ready for implementation parser This item relates to the Parser Utility researching and removed discussing The issue needs to be discussed, elaborated, or refined good-first-issue Something that is suitable for those who want to start contributing help-wanted We would really appreciate some support from community for this one confirmed The scope is clear, ready for implementation labels May 12, 2025
@dreamorosi
Copy link
Contributor

Cool, thanks.

For Kinesis, we should add the fields mentioned above to this schema.

For DynamoDB I am not sure how the new payload looks like, so we'll have to test before we know for sure.

@leandrodamascena
Copy link
Contributor Author

The new fields for DDB Stream are almost the same - or exactly the same - as the ones you need to add to Kinesis:

{
   "Records":[
      {
         "eventID":"1",
         "eventName":"INSERT",
         "eventVersion":"1.0",
         "eventSource":"aws:dynamodb",
         "awsRegion":"us-east-1",
         "dynamodb":{
            "Keys":{
               "Id":{
                  "N":"101"
               }
            },
            "NewImage":{
               "Message":{
                  "S":"New item!"
               },
               "Id":{
                  "N":"101"
               }
            },
            "SequenceNumber":"111",
            "SizeBytes":26,
            "StreamViewType":"NEW_AND_OLD_IMAGES"
         },
         "eventSourceARN":"stream-ARN"
      },
      {
         "eventID":"2",
         "eventName":"MODIFY",
         "eventVersion":"1.0",
         "eventSource":"aws:dynamodb",
         "awsRegion":"us-east-1",
         "dynamodb":{
            "Keys":{
               "Id":{
                  "N":"101"
               }
            },
            "NewImage":{
               "Message":{
                  "S":"This item has changed"
               },
               "Id":{
                  "N":"101"
               }
            },
            "OldImage":{
               "Message":{
                  "S":"New item!"
               },
               "Id":{
                  "N":"101"
               }
            },
            "SequenceNumber":"222",
            "SizeBytes":59,
            "StreamViewType":"NEW_AND_OLD_IMAGES"
         },
         "eventSourceARN":"stream-ARN"
      },
      {
         "eventID":"3",
         "eventName":"REMOVE",
         "eventVersion":"1.0",
         "eventSource":"aws:dynamodb",
         "awsRegion":"us-east-1",
         "dynamodb":{
            "Keys":{
               "Id":{
                  "N":"101"
               }
            },
            "OldImage":{
               "Message":{
                  "S":"This item has changed"
               },
               "Id":{
                  "N":"101"
               }
            },
            "SequenceNumber":"333",
            "SizeBytes":38,
            "StreamViewType":"NEW_AND_OLD_IMAGES"
         },
         "eventSourceARN":"stream-ARN"
      }
   ],
    "window": {
        "start": "2020-07-30T17:00:00Z",
        "end": "2020-07-30T17:05:00Z"
    },
    "state": {
        "1": "state1"
    },
    "shardId": "shard123456789",
    "eventSourceARN": "stream-ARN",
    "isFinalInvokeForWindow": false,
    "isWindowTerminatedEarly": false
}

@dreamorosi dreamorosi added good-first-issue Something that is suitable for those who want to start contributing help-wanted We would really appreciate some support from community for this one confirmed The scope is clear, ready for implementation and removed researching labels May 12, 2025
@dreamorosi
Copy link
Contributor

Thank you, adding the help-wanted label. If anyone is interested in picking this up, please leave a comment so I can assign this to you before you start working on a PR.

@kiitosu
Copy link
Contributor

kiitosu commented May 12, 2025

@dreamorosi
I am interested!

@dreamorosi
Copy link
Contributor

Thank you @kiitosu, I have assigned the issue to you - let me know if you have any questions, otherwise happy to iterate on the pull request once it's ready!

Copy link
Contributor

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

@github-actions github-actions bot added pending-release This item has been merged and will be released soon and removed help-wanted We would really appreciate some support from community for this one confirmed The scope is clear, ready for implementation labels May 19, 2025
Copy link
Contributor

This is now released under v2.20.0 version!

@github-actions github-actions bot added completed This item is complete and has been merged/shipped and removed pending-release This item has been merged and will be released soon labels May 20, 2025
@dreamorosi dreamorosi moved this from Coming soon to Shipped in Powertools for AWS Lambda (TypeScript) May 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
completed This item is complete and has been merged/shipped feature-request This item refers to a feature request for an existing or new utility good-first-issue Something that is suitable for those who want to start contributing parser This item relates to the Parser Utility
Projects
3 participants