Skip to content

Feature request: Improve support for SQS-wrapped S3 event notifications #2078

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 tasks done
theipster opened this issue Apr 2, 2023 · 6 comments · Fixed by #2108
Closed
2 tasks done

Feature request: Improve support for SQS-wrapped S3 event notifications #2078

theipster opened this issue Apr 2, 2023 · 6 comments · Fixed by #2108
Assignees
Labels
feature-request feature request parser Parser (Pydantic) utility

Comments

@theipster
Copy link
Contributor

theipster commented Apr 2, 2023

Use case

Originally discussed in #1656 (comment).

S3 event notifications can be sometimes be ingested into Lambda via an intermediary such as an SQS queue (i.e. Lambda event source), for various architectural reasons - batching, retries, etc.. However, from the Lambda function's perspective, the intermediary might not be too important; what's important is the S3 event notification itself.

With the current Powertools built-ins, it is possible to parse the S3 event notification data structure out of the SQS message data structure, but this requires some awkward boilerplate code to chain the two data structures together (mostly due to the JSON-formatted wrapper in the middle):

from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor
from aws_lambda_powertools.utilities.parser.models import SqsRecordModel, S3Model
from pydantic import Json

class SqsS3EventNotificationModel(SqsRecordModel):
  body: Json[S3Model]

processor = BatchProcessor(event_type=EventType.SQS, model=SqsS3EventNotificationModel)

def record_handler(record: SqsS3EventNotificationModel):
  for s3_record in record.body.Records:
    do_something_useful(s3_record)

@batch_processor(record_handler=record_handler, processor=processor)
def handler(event, context):
  return processor.response()

Solution/User Experience

A simple quick-win to the user-experience would be to make a new data model available as a built-in:

from aws_lambda_powertools.utilities.parser.models import SqsRecordModel, S3Model
from pydantic import Json

class SqsS3EventNotificationModel(SqsRecordModel):
  body: Json[S3Model]

This would be compatible with the existing framework and could be used as just another model, e.g.:

from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor

processor = BatchProcessor(event_type=EventType.SQS, model=SqsS3EventNotificationModel)

def record_handler(record: SqsS3EventNotificationModel):
  for s3_record in record.body.Records:
    do_something_useful(s3_record)

@batch_processor(record_handler=record_handler, processor=processor)
def handler(event, context):
  return processor.response()

Note: see this comment about Mypy compatibility, although I believe it may no longer be a concern due to a recent typing improvement.

Alternative solutions

If there's a way to skip the intermediate handler function, that would be even nicer (but see trade-off below).

In other words, rather than needing to define a def record_handler(record: SqsS3EventNotificationModel) that iterates through the record.body.Records, it would be nice to just directly define a def record_handler(record: S3Model) and let the processor unwrap everything instead.

This would keep the business logic clean, and not having to care whether it came via SQS or EventBridge, etc..

However, the trade-off to this even simpler interface would be that any event metadata (from both the SQS event and the S3 event notification) would be unavailable.

Acknowledgment

@theipster theipster added feature-request feature request triage Pending triage from maintainers labels Apr 2, 2023
@boring-cyborg
Copy link

boring-cyborg bot commented Apr 2, 2023

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link

@heitorlessa
Copy link
Contributor

heitorlessa commented Apr 3, 2023 via email

@leandrodamascena leandrodamascena added parser Parser (Pydantic) utility and removed triage Pending triage from maintainers labels Apr 4, 2023
@leandrodamascena
Copy link
Contributor

Hi @theipster! We are planning the next release and would like to know if you have the bandwidth to submit this PR! It would be amazing to have your contribution here. But don't worry if you can't (we know everyone is busy), we can do it and give you credit.

@theipster
Copy link
Contributor Author

Hi @leandrodamascena - no guarantees, but I can try to squeeze something in! 🙂

The class itself shouldn't be a problem, but I'm not too familiar with the existing test suite, so any high level guidance on what categories of tests is appropriate for this change would be greatly appreciated - thanks!

@leandrodamascena
Copy link
Contributor

Hi @leandrodamascena - no guarantees, but I can try to squeeze something in! slightly_smiling_face

The class itself shouldn't be a problem, but I'm not too familiar with the existing test suite, so any high level guidance on what categories of tests is appropriate for this change would be greatly appreciated - thanks!

I saw you submitted the PR and that's awesome! What do you think converting the PR from draft to work in progress? So we can work together to check the best user experience, to write the tests and documentation.

We use pytest to write our tests; you can see examples of how we do this.

Example: S3 Object Event

Test: https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/tests/functional/parser/test_s3%20object_event.py

Event: https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/tests/events/s3ObjectEventIAMUser.json

You can copy these files and change the event and fields to test that the parser is parsing the fields as expected.

Please reach out if you need additional guidance and we can work together to get this done.

@github-actions
Copy link
Contributor

github-actions bot commented May 2, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request feature request parser Parser (Pydantic) utility
Projects
None yet
3 participants