Do not try to delete lambda@edge functions with replicas

[rarguelloF]

Terraform fails to delete lambda@edge functions that were already replicated, as AWS just doesn't let it.

From http://docs.aws.amazon.com/lambda/latest/dg/lambda-edge.html:

When you create a trigger, Lambda replicates the function to AWS Regions and CloudFront edge locations around the globe. Note that replicas can't be edited or deleted.

Terraform Version

Terraform v0.10.6

Affected Resource(s)

• aws_lambda_function

Expected Behavior

Terraform should not fail and should give you some kind of warning that it couldn't delete the resource because AWS just doesn't let you.

Actual Behavior

Terraform fails as it's trying to delete lambda function and AWS doesn't allow it.

Steps to Reproduce

1. Create some configuration file with a lambda@edge function.
2. Associate it to a cloudfront distribution and let it get replicated.
3. Remove the lambda from your configuration.
4. terraform apply

[pinfoldd]

I'm having this problem also.

[ctd]

I can't reproduce a crash in Terraform v0.10.7, but I do still hit a dead-end where Terraform is unable to delete the resource:

Error applying plan:

1 error(s) occurred:

* module.xxx.aws_lambda_function.yyy (destroy): 1 error(s) occurred:

* aws_lambda_function.yyy: Error deleting Lambda Function: InvalidParameterValueException: Lambda was unable to delete arn:aws:lambda:us-east-1:123456789123:function:zzz:2 because it is a replicated function.
	status code: 400, request id: ........-....-....-....-............

It would be useful for the aws_lambda_function resource to support a retain_on_delete parameter, similar to cloudfront_distribution so that the resource can be "forgotten" from the terraform state despite the limitation that AWS APIs won't actually let you remove it.

[yvele]

You can delete a CF trigger, but apparently there is actually no way to delete replicated functions

Consequence: You can no longer delete the original function that was used as a CF trigger 😞

https://forums.aws.amazon.com/thread.jspa?threadID=260242&tstart=0

I reached out to my AWS contact during the Lambda@Edge preview, and he said they have no plans to support deleting replicated functions manually. But they're working on a system to automatically delete the replicated functions when you remove the trigger.

On another note, we can no longer delete the original function that was used as a CF trigger. Says "There was an error deleting your function: Lambda was unable to delete arn:aws:lambda:us-east-1:...:1 because it is a replicated function." I'd assume this is a bug in the system, since we need to be able to delete functions we created ourselves (aside from the replicated functions created automatically by CF.)

[dcalhoun]

@ctd I believe the "crash" @rarguelloF describes is likely the error that is thrown. The error effectively prevents you from altering the Lambda function (e.g. renaming, deleting), as Terraform aborts any changes when failing to delete the original, replicated function.

For my project, I had to destroy and recreate the terraform.tfstate file in order to rename a function that was replicated by a CloudFront distribution. I agree with you that it would be nice for Terraform to at least support "forgetting" Lambda functions that are incapable of being deleted by the AWS API.

[rarguelloF]

@ctd @dcalhoun replaced the word "crash" for "error" 👍

[willfarrell]

Support reply regarding this on Nov 27, 2017

Hello,

Thank you for contacting Amazon Web Services.

Unfortunately this is an open issue which our team are looking to support. 

Today, Lambda Master Functions do not get deleted once the association and versions 
are deleted. This is something which the team is expecting to resolve very soon, at this 
moment unfortunately there is no option for customer or AWS team to delete these functions.

If you are running into any limit errors due to these stale functions do let me know, I 
will be happy to work with our teams to help increase the limits.

We appreciate your patience while this feature is made available.

Let me know if I could be of any further help on this case.

Best regards,

Dilip S | Sydney Support Center
Amazon Web Services

Any one have an idea what very soon means to AWS? 6-8m?

[leecookson]

If a Lambda@Edge can be used for other behaviors and/or other CloudFront distributions, it makes sense to me to put in it's own terrform project, and just refer to it, so that destroying a distribution or removing a Lambda trigger from a behavior shouldn't attempt to destroy the (reusable) Lambda function. Other than using arn, I don't see a way to associate a Lambda function with a CF behavior. A data object to refer to a lambda would be very useful here, and we could pull the latest version using variables on the data object. Also, Lambda@Edge triggers have to use specific versions, and can't use the $LATEST alias. Is there another resource we can use to allow a CloudFront distribution to be "stable" and not need a variable updated each time a new Lambda function is released.

[lfreneda]

Same problem here..

[oheck]

-_- ... sometimes aws is killing me ;)

[scott-r-lindsey]

January 12, 2018. I just did my level best to find a way, any way, to delete a function that has been replicated. I'm guessing that someone inside of AWS closed the bug report not understanding how serious this issue is.

[ziggythehamster]

The doc moved here btw: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-edge-delete-replicas.html

[jbellmore]

I am also blocked by this issue unfortunately.

We are trying to implement automated tests using the kitchen terraform plugin and aws-spec and this issue completely blocks this due to the terraform destroy erroring if you have a lambda@edge function in your stack.

My first thought for getting around this issue was to remove the lambda@edge functionality from the module only when tests are run but I have been unable to figure out how to accomplish this. It seems impossible to do this since the lambda_function_association property can only be set using the interpolation syntax (lambda_function_association = [ "${conditional logic here}" ] ) and that doesn't appear to support passing in a list of map objects.

Is there some other way that I can conditionally set that property based on a variable value to selectively enable or disable those associations?

Since the lambda replicas will be deleted by cloudfront with eventual consistency once the associations to the cloudfront distro has been deleted is it enough to delete the associations on destroy and trust cloudfront will eventually get them cleaned up or is the aws api erroring deleting the main lambda function due to the fact that replicas exist?

Either way, until AWS fixes their issue, which seems like they aren't going to be doing anytime soon, is it possible to have the deletion be treated as a warning so it doesn't just completely block anyone trying to use it now? Even if we get a warning and have to clean up the function manually at a later date that seems the best we can hope for until AWS addresses this properly.

Thanks!