Feature request: Ensure parser schema types are compatible with @types/aws-lambda

[bilalq]

Use case

Hello,

For a long time now, we, like many others, have been using @types/aws-lambda as our reference types for Lambda events. For safety, we used our own typeguard functions. Now that Powertools vends a parser, we'd like to start using it in our middleware. However, when working with the SQS event, I saw a type incompatibility around the md5OfMessageAttributes field.

The @types/aws-lambda package defines it as such:

// SQS
// https://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html#supported-event-source-sqs
export interface SQSRecord {
    messageId: string;
    receiptHandle: string;
    body: string;
    attributes: SQSRecordAttributes;
    messageAttributes: SQSMessageAttributes;
    md5OfBody: string;
    md5OfMessageAttributes?: string;
    eventSource: string;
    eventSourceARN: string;
    awsRegion: string;
}

The Powertools parser schema has it modeled like this:

/**
 * Zod schema for a SQS record inside an SQS event.
 */
const SqsRecordSchema = z.object({
  messageId: z.string(),
  receiptHandle: z.string(),
  body: z.string(),
  attributes: SqsAttributesSchema,
  messageAttributes: z.record(z.string(), SqsMsgAttributeSchema),
  md5OfBody: z.string(),
  md5OfMessageAttributes: z.string().optional().nullable(),
  eventSource: z.literal('aws:sqs'),
  eventSourceARN: z.string(),
  awsRegion: z.string(),
});

Note the difference of md5OfMessageAttributes?: string vs md5OfMessageAttributes: z.string().optional().nullable().

Solution/User Experience

The SQS or Lambda teams can probably best speak to whether null is possible or it really should be string | undefined. Depending on that answer, either the zod schema here should update or an update should be made to @types/aws-lambda.

Going forward though, it would be nice if there were unit tests that compared the Zod schema parsed shapes against the types from @types/aws-lambda. There are many libraries and utilities that have been written around the types from that package, so i t would be great if the parsed events from Powertools were compatible with those.

Alternative solutions

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[dreamorosi]

Hi @bilalq, thanks for the issue.

I'll need some time to verify whether the field you shared can be null or not. If the error is on our side we'll fix the schemas; if instead the issue is on the DefinitelyTyped repo I'll submit a PR to fix it there.

Given that we both don't control the types in the DefinitelyTyped repo and we also don't have an automated way of detecting changes in the AWS payloads, adding tests would guarantee compatibility only at a specific point in time.

We offer our own types, that are derived from the Zod schema and can be imported via @aws-lambda-powertools/parser/types. These are guaranteed to always match the Zod schemas, compatibility with the @types/aws-lambda is offered as best-effort at this stage.

[bilalq]

Yeah, that's all reasonable. Maybe this isn't compelling enough on it's own, but I think even point-in-time checks are still useful. With them setup, a simple version bump would catch outdated schemas on the Powertools side and help with maintainability.

[bilalq]

Oh, and on the subject of forward compatibility, would it make sense to add a .passthrough() on all object schemas? So if AWS ever extends shapes with new fields, it won't result in parsing errors?

I believe Zod 4 has a different API for this where you write .looseObject({ ... }) instead of .object({ ... }).passthrough(), but it looks like Powertools is on Zod 3 for now.

https://v3.zod.dev/?id=passthrough

[dreamorosi]

Oh, and on the subject of forward compatibility, would it make sense to add a .passthrough() on all object schemas? So if AWS ever extends shapes with new fields, it won't result in parsing errors?

When there are extraneous keys in an object Zod just drops them instead of throwing an error.

I'm not inclined to add a modifier to the schemas as I believe this should be a decision for customers. While in your case .passthrough() is acceptable, other customers might instead want to opt for .strict(), while others might be fine with just dropping keys (current behavior).

Ultimately our built-in schemas are meant to be extended, as long as they are accurate (which is the subject of this issue) customers can decide whether they want to make them more strict or loose.

We'll pick up this issue this week, apologies for the delay.

[sdangol]

Hi @bilalq @dreamorosi, I did some research on it, but wasn't able to find the exact schema type in the documentation as a source of truth. In the Powertools Python version as well we're using md5OfMessageAttributes: Optional[str] = None. Both @types/aws-lambda and Powertools Python have more usage than us, if that type definition works for them it should work for us. So, we've decided to move forward with z.string().optional(). I'll submit a PR to fix this.