Inconsistent behavior in ServerlessApp cloudformation deployment

[vasylk]

Hello,

we have been experiencing unexpected? behavior affecting API Gateway when deploying the serverless app.

Along with SAM resources the cloudformation template contains API Gateway proxy integration with S3 (see simplified Cloudformation sippet for reference).
The stack is deployed via CodePipeline + integration with CloudFormation

AWS::Region = eu-central-1

The unexpected behavior is:

1. After some random deployment execution (happened already a few times) the API Gateway resources related to ApiGateway/S3 Integration (not SAM) get removed from API Gateway (S3FileImportRootResource, S3FileImportSubResource, S3FileImport) even though they do exist in CloudFormation stack and not affected by stack update.

The same happens with BinaryMediaTypes property of AWS::Serverless::Api

It gets restored when we manually comment out these resources in CloudFormation template and run two deployments in a row - without these resources in the template and with them. I assume it is not expected behavior.

2. The changes in the resource S3FileImport (Integration section) do not get automatically deployed in API Gateway Stage specified in WebApi.StageName
   We have to manually run Deploy API action in API Gateway to get these changes available.

Regards
Vasyl

Template Snippet:

 AWSTemplateFormatVersion: "2010-09-09"
 Transform: 'AWS::Serverless-2016-10-31'
 .....
  ApiGwS3ProxyRole:
    Type: "AWS::IAM::Role"
    ....
  S3FileImportRootResource:
    Type: AWS::ApiGateway::Resource
    ....
  S3FileImportSubResource:
    Type: AWS::ApiGateway::Resource
    ....
  S3ReportFileImport:
    Type: AWS::ApiGateway::Method
    Properties:
      RestApiId: !Ref WebApi
      ResourceId: !Ref S3FileImportSubResource
      HttpMethod: POST
      Integration:
        Type: AWS
        IntegrationResponses:
  WebApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: api
      BinaryMediaTypes:
          - application~1octet-stream

[vasylk]

Just updating that this issue happened again. The mentioned resources exist in the cloudformation stack but disappeared from the API Gateway resources

[jlhood]

@vasylk Sorry you're experiencing this issue. Could you please provide a full example template so we can try to help troubleshoot?

[vasylk]

@jlhood
Here is a "full example". I needed to clean it up a little bit for simplicity, but affected resources are exactly the same.

Just to repeat the behaviour:

• API gateway resources with logical name starting with S3ReportFileImport* along with WebApi.BinaryMediaTypes definition get removed from API Gateway and the rest resource is no longer available
• After manually restoring (deploy the stack without the resources and afterwards with the resources) we have to manually execute "Deploy API" action to make those changes available again.

The app has continuously been deployed via CodePipeline (Integration with CloudFormation)

AWS Region: eu-central-1

Codepipeline template example

AWSTemplateFormatVersion: "2010-09-09"

  Pipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:

    Stages:
       .................
          cloudformation package is executed before and the output template.yaml is passed to 
Deployment stage as "PackageBackend" InputArtefact
       
        - Name: "Deployment-Test"
          Actions:
            -
              Name: "Backend-AppStack-CreateChangeSet"
              RunOrder: 1
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Provider: CloudFormation
                Version: 1
              Configuration:
                ActionMode: CHANGE_SET_REPLACE
                Capabilities: CAPABILITY_NAMED_IAM
                ChangeSetName: !Sub "${AWS::StackName}-DeployAppBackend-ChangeSet"
                RoleArn:
                  Fn::ImportValue: !Sub "${CodePipelineBaseStack}-CodePipelineRoleArn"
                StackName: !Sub "${StackNamespace}-${Branch}-backend"
                TemplatePath: !Sub "PackageBackend::backend.yaml"
              InputArtifacts:
                - Name: "PackageBackend"
            -
              Name: "Backend-AppStack-ExecuteChangeSet"
              RunOrder: 3
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Provider: CloudFormation
                Version: 1
              Configuration:
                ActionMode: CHANGE_SET_EXECUTE
                Capabilities: CAPABILITY_NAMED_IAM
                ChangeSetName: !Sub "${AWS::StackName}-DeployAppBackend-ChangeSet"
                RoleArn:
                  Fn::ImportValue: !Sub "${CodePipelineBaseStack}-CodePipelineRoleArn"
                StackName: !Sub "${StackNamespace}-${Branch}-backend"
              InputArtifacts:
                - Name: "BackendBuild"
       ............

Serverless app template

AWSTemplateFormatVersion: "2010-09-09"
Transform: 'AWS::Serverless-2016-10-31'

Parameters:
  DatabaseNamespace:
    Type: String
  CognitoUserPoolArn:
    Type: String

Globals:
  Function:
    Runtime: nodejs8.10
    MemorySize: 3008
    Timeout: 300

Resources:
  S3DataBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub "${AWS::StackName}-${AWS::AccountId}-${AWS::Region}-data"

  AppFunctionRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
        - "arn:aws:iam::aws:policy/service-role/AWSLambdaDynamoDBExecutionRole"

  ## This is just an example of one of the functions. Original file contains many of them but no overlapping "UrlPath"
  WebApiOverviewFunctions:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../backend/dist/
      Handler: webapi-overview.handler
      Role: !GetAtt AppFunctionRole.Arn
      Events:
        Proxy:
          Type: Api
          Properties:
            Path: /reports/overview/{proxy+}
            Method: ANY
            RestApiId: !Ref WebApi
            Auth:
              Authorizer: LoginAuthorizer

# BEGIN: API Gateway/S3 Integration
  ApiGwS3ProxyRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: apigateway.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: !Sub "${AWS::StackName}-ApiGwProxyAccessPolicy"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "s3:Put*"
                Resource: !Sub "arn:aws:s3:::${S3DataBucket}/uploads/*"

  S3ReportFileImportAuthorizer:
    Type: AWS::ApiGateway::Authorizer
    Properties:
      Name: !Sub "${AWS::StackName}-S3ReportFileImportAuthorizer"
      RestApiId: !Ref WebApi
      Type: COGNITO_USER_POOLS
      ProviderARNs:
        - !Ref CognitoUserPoolArn
      IdentitySource: "method.request.header.Authorization"

  S3ReportFileImportRootResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !GetAtt WebApi.RootResourceId
      PathPart: reports-fileimport
      RestApiId: !Ref WebApi

  S3ReportFileImportOrgResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !Ref S3ReportFileImportRootResource
      PathPart: "{organization}"
      RestApiId: !Ref WebApi

  S3ReportFileImport:
    Type: AWS::ApiGateway::Method
    Properties:
      RestApiId: !Ref WebApi
      ResourceId: !Ref S3ReportFileImportOrgResource
      AuthorizationType: COGNITO_USER_POOLS
      AuthorizerId: !Ref S3ReportFileImportAuthorizer
      HttpMethod: POST
      RequestParameters:
        method.request.path.organization: true
        method.request.path.fileName: true
        method.request.header.Content-Type: true
        method.request.header.x-abcdef-file-content-type: true
        method.request.header.x-abcdef-file-name: true
      MethodResponses:
        - StatusCode: 200
          ResponseParameters:
            method.response.header.Date: true
            method.response.header.Content-Length: true
        - StatusCode: 400
          ResponseParameters:
            method.response.header.Date: true
            method.response.header.Content-Length: true
            method.response.header.Content-Type: true
      Integration:
        Type: AWS
        Uri: !Sub "arn:aws:apigateway:${AWS::Region}:s3:path/${S3DataBucket}/uploads/{organization}/{item}"
        IntegrationHttpMethod: PUT
        Credentials: !GetAtt ApiGwS3ProxyRole.Arn
        RequestParameters:
          integration.request.path.item: context.requestId
          integration.request.path.organization: method.request.path.organization
          integration.request.header.Content-Type: method.request.header.Content-Type
          integration.request.header.x-amz-meta-abcdef-organization-id: method.request.path.organization
          integration.request.header.x-amz-meta-abcdef-file-content-type: method.request.header.x-abcdef-file-content-type
          integration.request.header.x-amz-meta-abcdef-file-name: method.request.header.x-abcdef-file-name
          integration.request.header.x-amz-meta-abcdef-username: context.authorizer.claims.cognito:username
          integration.request.header.x-amz-meta-abcdef-userfullname: context.authorizer.claims.name
          integration.request.header.x-amz-meta-abcdef-principalId: context.authorizer.claims.principalId
        IntegrationResponses:
          - StatusCode: 200
            ResponseParameters:
              method.response.header.Date: integration.response.header.Date
              method.response.header.Content-Length: integration.response.header.Content-Length
          - StatusCode: 400
            SelectionPattern: "4\\d{2}"
            ResponseParameters:
              method.response.header.Date: integration.response.header.Date
              method.response.header.Content-Length: integration.response.header.Content-Length
              method.response.header.Content-Type: integration.response.header.Content-Type
  # END: API Gateway/S3 Integration

  WebApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: api
      Auth:
        Authorizers:
          LoginAuthorizer:
            UserPoolArn: !Ref CognitoUserPoolArn
      BinaryMediaTypes:
        - application~1octet-stream

[brooksidesw]

+1. I just had this happen with BinaryMediaTypes. My guess is that this is something in CloudFormation. If I edit the CloudFormation template in Designer, adding a new BinaryMediaType, the resulting API has not two but one binary media types.

For example, I start with BinaryMediaTypes include "application~1octet-stream". The resulting API has no binary media types.

Then, in Designer, I add "image~1gif". Two types in the template. The resulting API has only "image/gif". If I add a third entry, before "image~1gif", and it reads "application~1octet-stream" (that is, my BinaryMediaTypes entry now lists "application~1octet-stream", "image~1gif", "application~1octet-stream"), I get no errors and the resulting API has no binary media types.

[praneetap]

@brooksidesw Your issue seems to be related to this pr #954 which is staged for release.

@vasylk There are two ways to define Api gateway resources in cloudformation: 1. defining an explicit swagger doc in AWS::ApiGateway::RestApi resource, and 2. Using explicit Cloudformation resources like AWS::ApiGateway::Resource and AWS::ApiGateway::Method. However you can't use both. Since SAM generates Swagger for you, you can't use the explicit resources.