Idempotency in progress timeout

[scottgerring]

Two expirations are stored on the DDB table:

• expiration - how long to cache the results of the operation. User-tunable ( seconds since epoch )
• in_progress_expiration - expire records that timed out during lambda execution. Set by powertools to the remainingExecutionTime. ( milliseconds since epoch)

When a request comes in, we try and conditionally upsert the DDB record - if the in_progress_expiration time is before now(), we can overwrite it, else we can't. The bug is here:

powertools-lambda-java/powertools-idempotency/src/main/java/software/amazon/lambda/powertools/idempotency/persistence/DynamoDBPersistenceStore.java

Line 154 in a19def3

new AbstractMap.SimpleEntry<>(":now", AttributeValue.builder().n(String.valueOf(now.getEpochSecond())).build()),

powertools-lambda-java/powertools-idempotency/src/main/java/software/amazon/lambda/powertools/idempotency/persistence/DynamoDBPersistenceStore.java

Line 163 in a19def3

.conditionExpression("attribute_not_exists(#id) OR #expiry < :now OR (attribute_exists(#in_progress_expiry) AND #in_progress_expiry < :now AND #status = :inprogress)")

We set now once, but we use it in a comparison where the units should be seconds, and another where the units should be milliseconds.

The upshot of this is that if a handler times out within an idempotent operation, we'll start seeing exceptions thrown for subsequent invocations.

What were you trying to accomplish?

Expected Behavior

Subsequent executions after a function timeout should update the DDB table and complete.

Current Behavior

Possible Solution

1. Provide nowMs and now in the DDB conditional update
2. Increase test coverage

[github-actions]

This is now released under 1.16.1 version!