aws · mgrandis · Mar 2, 2021 · Jan 18, 2021 · Jan 25, 2021 · Feb 1, 2021
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -0,0 +1,39 @@
+---
+name: Bug report
+about: Create an issue to report a bug for the SAM Translator
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+<!-- Make sure we don't have an existing Issue that reports the bug you are seeing (both open and closed). 
+If you do find an existing Issue, re-open or add a comment to that Issue instead of creating a new one. -->
+
+### Description:
+<!-- Briefly describe the bug you are facing.-->
+
+
+
+### Steps to reproduce:
+<!-- Provide detailed steps to replicate the bug, including steps from third party tools (CDK, etc.) -->
+
+
+
+### Observed result:
+<!-- Please provide command output with `--debug` flag set.-->
+
+
+
+### Expected result:
+<!-- Describe what you expected.-->
+
+
+
+### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
+
+1. OS:
+2. If using SAM CLI, `sam --version`:
+3. AWS region:
+
+`Add --debug flag to any SAM CLI commands you are running`
diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md
@@ -0,0 +1,25 @@
+---
+name: Feature request
+about: Suggest an idea/feature/enhancement for the SAM Translator
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+<!-- Make sure we don't have an existing Issue for that feature request (open or closed). -->
+
+### Describe your idea/feature/enhancement
+
+Provide a clear description.
+
+Ex: I wish the SAM Translator would [...]
+
+### Proposal
+
+Add details on how to add this to the product.
+
+Things to consider:
+[ ] The [SAM documentation](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-specification.html) will need to be updated
+
+### Additional Details
diff --git a/DEVELOPMENT_GUIDE.md b/DEVELOPMENT_GUIDE.md
@@ -128,6 +128,10 @@ will not work in Python3.6). If you want to test in many versions, you can creat
 each version and flip between them (sourcing the activate script). Typically, we run all tests in
 one python version locally and then have our ci (appveyor) run all supported versions.
 
+### Integration tests
+
+Integration tests are covered in detail in the [INTEGRATION_TESTS.md file](INTEGRATION_TESTS.md) of this repository.
+
 Code Conventions
 ----------------
 

diff --git a/INTEGRATION_TESTS.md b/INTEGRATION_TESTS.md
@@ -0,0 +1,131 @@
+# AWS SAM integration tests
+
+These tests run SAM against AWS services by translating SAM templates, deploying them to Cloud Formation and verifying the resulting objects.
+
+They must run successfully under Python 2 and 3.
+
+## Run the tests
+
+### Prerequisites
+
+#### User and rights
+
+An Internet connection and an active AWS account are required to run the tests as they will interact with AWS services (most notably Cloud Formation) to create and update objects (Stacks, APIs, ...).
+
+AWS credentials must be configured either through a [credentials file](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) or [environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html)
+
+The user running the tests must have the following roles:
+
+```
+AmazonSQSFullAccess
+AmazonSNSFullAccess
+AmazonAPIGatewayAdministrator
+AWSKeyManagementServiceFullAccess
+AWSStepFunctionsFullAccess
+```
+
+If you plan on running the full tests suite, ensure that the user credentials you are running the tests with have a timeout of at least 30 minutes as the full suite can take more than 20 minutes to execute.
+
+#### Initialize the development environment
+
+If you haven't done so already, run the following command in a terminal at the root of the repository to initialize the development environment:
+
+```
+make init
+```
+
+### Running all the tests
+
+From the root of the repository, run:
+
+```
+make integ-test
+```
+
+### Running a specific test file
+
+From the command line, run:
+
+```
+pytest --no-cov path/to/the/test_file.py
+```
+
+For example, from the root of the project:
+
+```sh
+pytest --no-cov integration/single/test_basic_api.py
+```
+
+### Running a specific test
+
+From the command line, run:
+
+```
+pytest --no-cov path/to/the/test_file.py::test_class::test_method
+```
+
+For example, from the root of the project:
+
+```sh
+pytest --no-cov integration/single/test_basic_api.py::TestBasicApi::test_basic_api
+```
+
+*We don't measure coverage for integration tests.*
+
+## Write a test
+
+1. Add your test templates to the `integration/resources/templates` single or combination folder.
+2. Write an expected json file for all the expected resources and add it to the `integration/resources/expected`.
+3. (Optional) Add the resource files (zip, json, etc.) to `integration/resources/code` and update the dictionaries in `integration/helpers/file_resources.py`.
+4. Write and add your python test code to the `integration` single or combination folder.
+5. Run it!
+
+## Skip tests for a specific service in a region
+
+1. Add the service you want to skip to the `integration/config/region_service_exclusion.yaml` under the region
+2. Add the @skipIf decorator to the test with the service name, take 'XRay' for example:
+```@skipIf(current_region_does_not_support('XRay'), 'XRay is not supported in this testing region')```
+
+## Directory structure
+
+### Helpers
+
+Common classes and tools used by tests.
+
+```
++-- helpers/
+|   +-- deployer           Tools to deploy to Cloud Formation
+|   +-- base_test.py       Common class from which all test classes inherit
+|   +-- file_resources.py  Files to upload to S3
+|   +-- resource.py        Helper functions to manipulate resources
+|   +-- template.py        Helper functions to translate the template
+```
+
+`base_test.py` contains `setUpClass` and `tearDownClass` methods to respectively upload and clean the `file_resources.py` resources (upload the files to a new S3 bucket, empty and delete this bucket).
+
+### Resources
+
+File resources used by tests.
+
+```
++-- resources
+|   +-- code               Files to upload to S3
+|   +-- expected           Files describing the expected created resources
+|   +-- templates          Source SAM templates to translate and deploy
+```
+
+The matching *expected* and *template* files should have the same name.
+
+For example, the `test_basic_api` test in the class `tests_integ/single/test_basic_api.py` takes `templates/single/basic_api.yaml` SAM template as input and verifies its result against `expected/single/basic_api.json`.
+
+### Single
+
+Basic tests which interact with only one service should be put here.
+
+### Combination
+
+Tests which interact with multiple services should be put there.
+
+### Tmp
+
+This directory is created on the first run and contains temporary and intermediary files used by the tests: sam templates with substituted variable values, translated temporary cloud formation templates, ...
diff --git a/Makefile b/Makefile
@@ -6,13 +6,16 @@ init:
 	pip install -e '.[dev]'
 
 test:
-	pytest --cov samtranslator --cov-report term-missing --cov-fail-under 95 tests
+	pytest --cov samtranslator --cov-report term-missing --cov-fail-under 95 tests/*
+
+integ-test:
+	pytest --no-cov integration/*
 
 black:
-	black setup.py samtranslator/* tests/* bin/*.py
+	black setup.py samtranslator/* tests/* integration/* bin/*.py
 
 black-check:
-	black --check setup.py samtranslator/* tests/* bin/*.py
+	black --check setup.py samtranslator/* tests/* integration/* bin/*.py
 
 # Command to run everytime you make changes to verify everything works
 dev: test
@@ -30,6 +33,7 @@ Usage: $ make [TARGETS]
 TARGETS
 	init        Initialize and install the requirements and dev-requirements for this project.
 	test        Run the Unit tests.
+	integ-test  Run the Integration tests.
 	dev         Run all development tests after a change.
 	pr          Perform all checks before submitting a Pull Request.
 

diff --git a/README.md b/README.md
@@ -3,18 +3,19 @@
 
 # AWS Serverless Application Model (AWS SAM)
 
-![Apache-2.0](https://img.shields.io/github/license/awslabs/serverless-application-model.svg)
-![SAM_CLI release](https://img.shields.io/github/release/awslabs/aws-sam-cli.svg?label=CLI%20Version)
+![Apache-2.0](https://img.shields.io/github/license/aws/serverless-application-model.svg)
+![SAM_CLI release](https://img.shields.io/github/release/aws/aws-sam-cli.svg?label=CLI%20Version)
+[![codecov](https://codecov.io/gh/aws/serverless-application-model/branch/master/graphs/badge.svg?style=flat)](https://codecov.io/gh/aws/serverless-application-model)
 
-The AWS Serverless Application Model (SAM) is an open-source framework for building serverless applications. 
-It provides shorthand syntax to express functions, APIs, databases, and event source mappings. 
+The AWS Serverless Application Model (SAM) is an open-source framework for building serverless applications.
+It provides shorthand syntax to express functions, APIs, databases, and event source mappings.
 With just a few lines of configuration, you can define the application you want and model it.
 
 [![Getting Started with AWS SAM](./docs/get-started-youtube.png)](https://www.youtube.com/watch?v=1dzihtC5LJ0)
 
 ## Get Started
 
-To get started with building SAM-based applications, use the SAM CLI. SAM CLI provides a Lambda-like execution 
+To get started with building SAM-based applications, use the SAM CLI. SAM CLI provides a Lambda-like execution
 environment that lets you locally build, test, debug, and deploy applications defined by SAM templates.
 
 * [Install SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html)
@@ -35,14 +36,14 @@ environment that lets you locally build, test, debug, and deploy applications de
 ## Why SAM
 
 + **Single\-deployment configuration**\. SAM makes it easy to organize related components and resources, and operate on a single stack\. You can use SAM to share configuration \(such as memory and timeouts\) between resources, and deploy all related resources together as a single, versioned entity\.
-   
+
 + **Local debugging and testing**\. Use SAM CLI to locally build, test, and debug SAM applications on a Lambda-like execution environment. It tightens the development loop by helping you find & troubleshoot issues locally that you might otherwise identify only after deploying to the cloud.
 
 + **Deep integration with development tools**. You can use SAM with a suite of tools you love and use.
     + IDEs: [PyCharm](https://aws.amazon.com/pycharm/), [IntelliJ](https://aws.amazon.com/intellij/), [Visual Studio Code](https://aws.amazon.com/visualstudiocode/), [Visual Studio](https://aws.amazon.com/visualstudio/), [AWS Cloud9](https://aws.amazon.com/cloud9/)
     + Build: [CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/)
     + Deploy: [CodeDeploy](https://docs.aws.amazon.com/codedeploy/latest/userguide/), [Jenkins](https://wiki.jenkins.io/display/JENKINS/AWS+SAM+Plugin)
-    + Continuous Delivery Pipelines: [CodePipeline](https://docs.aws.amazon.com/codepipeline/latest/userguide/) 
+    + Continuous Delivery Pipelines: [CodePipeline](https://docs.aws.amazon.com/codepipeline/latest/userguide/)
     + Discover Serverless Apps & Patterns: [AWS Serverless Application Repository](https://docs.aws.amazon.com/serverlessrepo/latest/devguide/)
 
 + **Built\-in best practices**\. You can use SAM to define and deploy your infrastructure as configuration. This makes it possible for you to use and enforce best practices through code reviews. Also, with a few lines of configuration, you can enable safe deployments through CodeDeploy, and can enable tracing using AWS X\-Ray\.
@@ -51,40 +52,40 @@ environment that lets you locally build, test, debug, and deploy applications de
 
 ## What is this Github repository? 💻
 
-This GitHub repository contains the SAM Specification, the Python code that translates SAM templates into AWS CloudFormation stacks and lots of examples applications. 
+This GitHub repository contains the SAM Specification, the Python code that translates SAM templates into AWS CloudFormation stacks and lots of examples applications.
 In the words of SAM developers:
 
-> SAM Translator is the Python code that deploys SAM templates via AWS CloudFormation. Source code is high quality (95% unit test coverage), 
-with tons of tests to ensure your changes don't break compatibility. Change the code, run the tests, and if they pass, you should be good to go! 
+> SAM Translator is the Python code that deploys SAM templates via AWS CloudFormation. Source code is high quality (95% unit test coverage),
+with tons of tests to ensure your changes don't break compatibility. Change the code, run the tests, and if they pass, you should be good to go!
 Clone it and run `make pr`!
 
 ## Contribute to SAM
 
-We love our contributors ❤️ We have over 100 contributors who have built various parts of the product. 
+We love our contributors ❤️ We have over 100 contributors who have built various parts of the product.
 Read this [testimonial from @ndobryanskyy](https://www.lohika.com/aws-sam-my-exciting-first-open-source-experience/) to learn
-more about what it was like contributing to SAM. 
+more about what it was like contributing to SAM.
 
-Depending on your interest and skill, you can help build the different parts of the SAM project; 
+Depending on your interest and skill, you can help build the different parts of the SAM project;
 
 **Enhance the SAM Specification**
 
 Make pull requests, report bugs, and share ideas to improve the full SAM template specification.
-Source code is located on Github at [awslabs/serverless-application-model](https://github.com/awslabs/serverless-application-model). 
+Source code is located on Github at [awslabs/serverless-application-model](https://github.com/awslabs/serverless-application-model).
 Read the [SAM Specification Contributing Guide](https://github.com/awslabs/serverless-application-model/blob/master/CONTRIBUTING.md)
 to get started.
-    
+
 **Strengthen SAM CLI**
 
 Add new commands or enhance existing ones, report bugs, or request new features for the SAM CLI.
-Source code is located on Github at [awslabs/aws-sam-cli](https://github.com/awslabs/aws-sam-cli). Read the [SAM CLI Contributing Guide](https://github.com/awslabs/aws-sam-cli/blob/develop/CONTRIBUTING.md) to 
-get started. 
+Source code is located on Github at [awslabs/aws-sam-cli](https://github.com/awslabs/aws-sam-cli). Read the [SAM CLI Contributing Guide](https://github.com/awslabs/aws-sam-cli/blob/develop/CONTRIBUTING.md) to
+get started.
 
 **Update SAM Developer Guide**
 
 [SAM Developer Guide](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/index.html) provides comprehensive getting started guide and reference documentation.
 Source code is located on Github at [awsdocs/aws-sam-developer-guide](https://github.com/awsdocs/aws-sam-developer-guide).
 Read the [SAM Documentation Contribution Guide](https://github.com/awsdocs/aws-sam-developer-guide/blob/master/CONTRIBUTING.md) to get
-started. 
+started.
 
 ### Join the SAM Community on Slack
 [Join the SAM developers channel (#samdev)](https://join.slack.com/t/awsdevelopers/shared_invite/zt-idww18e8-Z1kXhI7GNuDewkweCF3YjA) on Slack to collaborate with fellow community members and the AWS SAM team.

diff --git a/appveyor-integration-test.yml b/appveyor-integration-test.yml
@@ -0,0 +1,24 @@
+version: 1.0.{build}
+image: Ubuntu
+
+environment:
+  matrix:
+  - TOXENV: py27
+    PYTHON_VERSION: '2.7'
+  - TOXENV: py36
+    PYTHON_VERSION: '3.6'
+  - TOXENV: py37
+    PYTHON_VERSION: '3.7'
+  - TOXENV: py38
+    PYTHON_VERSION: '3.8'
+
+build: off
+
+install:
+- sh: "source ${HOME}/venv${PYTHON_VERSION}/bin/activate"
+- sh: "python --version"
+- make init
+
+test_script:
+- make integ-test
+
diff --git a/integration/__init__.py b/integration/__init__.py
diff --git a/integration/config/__init__.py b/integration/config/__init__.py