test: Add simple integration tests

DEVELOPMENT_GUIDE.md

@@ -128,6 +128,10 @@ will not work in Python3.6). If you want to test in many versions, you can creat
 each version and flip between them (sourcing the activate script). Typically, we run all tests in
 one python version locally and then have our ci (appveyor) run all supported versions.
 
+### Integration tests
+
+Integration tests are covered in detail in the [INTEGRATION_TESTS.md file](INTEGRATION_TESTS.md) of this repository.
+
 Code Conventions
 ----------------
 

INTEGRATION_TESTS.md

@@ -0,0 +1,125 @@
+# AWS SAM integration tests
+
+These tests run SAM against AWS services by translating SAM templates, deploying them to Cloud Formation and verifying the resulting objects.
+
+They must run successfully under Python 2 and 3.
+
+## Run the tests
+
+### Prerequisites
+
+#### User and rights
+
+An Internet connection and an active AWS account are required to run the tests as they will interact with AWS services (most notably Cloud Formation) to create and update objects (Stacks, APIs, ...).
+
+AWS credentials must be configured either through a [credentials file](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) or [environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html)
+
+The user running the tests must have the following roles:
+
+```
+AmazonSQSFullAccess
+AmazonSNSFullAccess
+AmazonAPIGatewayAdministrator
+AWSKeyManagementServiceFullAccess
+AWSStepFunctionsFullAccess
+```
+
+If you plan on running the full tests suite, ensure that the user credentials you are running the tests with have a timeout of at least 30 minutes as the full suite can take more than 20 minutes to execute.
+
+#### Initialize the development environment
+
+If you haven't done so already, run the following command in a terminal at the root of the repository to initialize the development environment:
+
+```
+make init
+```
+
+### Running all the tests
+
+From the root of the repository, run:
+
+```
+make integ-test
+```
+
+### Running a specific test file
+
+From the command line, run:
+
+```
+pytest --no-cov path/to/the/test_file.py
+```
+
+For example, from the root of the project:
+
+```sh
+pytest --no-cov integration/single/test_basic_api.py
+```
+
+### Running a specific test
+
+From the command line, run:
+
+```
+pytest --no-cov path/to/the/test_file.py::test_class::test_method
+```
+
+For example, from the root of the project:
+
+```sh
+pytest --no-cov integration/single/test_basic_api.py::TestBasicApi::test_basic_api
+```
+
+*We don't measure coverage for integration tests.*
+
+## Write a test
+
+1. Add your test templates to the `integration/resources/templates` single or combination folder.
+2. Write an expected json file for all the expected resources and add it to the `integration/resources/expected`.
+3. (Optional) Add the resource files (zip, json, etc.) to `integration/resources/code` and update the dictionaries in `integration/helpers/file_resources.py`.
+4. Write and add your python test code to the `integration` single or combination folder.
+5. Run it!
+
+## Directory structure
+
+### Helpers
+
+Common classes and tools used by tests.
+
+```
++-- helpers/
+|   +-- deployer           Tools to deploy to Cloud Formation
+|   +-- base_test.py       Common class from which all test classes inherit
+|   +-- file_resources.py  Files to upload to S3
+|   +-- resource.py        Helper functions to manipulate resources
+|   +-- template.py        Helper functions to translate the template
+```
+
+`base_test.py` contains `setUpClass` and `tearDownClass` methods to respectively upload and clean the `file_resources.py` resources (upload the files to a new S3 bucket, empty and delete this bucket).
+
+### Resources
+
+File resources used by tests.
+
+```
++-- resources
+|   +-- code               Files to upload to S3
+|   +-- expected           Files describing the expected created resources
+|   +-- templates          Source SAM templates to translate and deploy
+```
+
+The matching *expected* and *template* files should have the same name.
+
+For example, the `test_basic_api` test in the class `tests_integ/single/test_basic_api.py` takes `templates/single/basic_api.yaml` SAM template as input and verifies its result against `expected/single/basic_api.json`.
+
+### Single
+
+Basic tests which interact with only one service should be put here.
+
+### Combination
+
+Tests which interact with multiple services should be put there.
+
+### Tmp
+
+This directory is created on the first run and contains temporary and intermediary files used by the tests: sam templates with substituted variable values, translated temporary cloud formation templates, ...

Makefile

@@ -6,13 +6,16 @@ init:
 	pip install -e '.[dev]'
 
 test:
-	pytest --cov samtranslator --cov-report term-missing --cov-fail-under 95 tests
+	pytest --cov samtranslator --cov-report term-missing --cov-fail-under 95 tests/*
+
+integ-test:
+	pytest --no-cov integration/*
 
 black:
-	black setup.py samtranslator/* tests/* bin/*.py
+	black setup.py samtranslator/* tests/* integration/* bin/*.py
 
 black-check:
-	black --check setup.py samtranslator/* tests/* bin/*.py
+	black --check setup.py samtranslator/* tests/* integration/* bin/*.py
 
 # Command to run everytime you make changes to verify everything works
 dev: test
@@ -30,6 +33,7 @@ Usage: $ make [TARGETS]
 TARGETS
 	init        Initialize and install the requirements and dev-requirements for this project.
 	test        Run the Unit tests.
+	integ-test  Run the Integration tests.
 	dev         Run all development tests after a change.
 	pr          Perform all checks before submitting a Pull Request.
 

appveyor-integration-test.yml

@@ -0,0 +1,24 @@
+version: 1.0.{build}
+image: Ubuntu
+
+environment:
+  matrix:
+  - TOXENV: py27
+    PYTHON_VERSION: '2.7'
+  - TOXENV: py36
+    PYTHON_VERSION: '3.6'
+  - TOXENV: py37
+    PYTHON_VERSION: '3.7'
+  - TOXENV: py38
+    PYTHON_VERSION: '3.8'
+
+build: off
+
+install:
+- sh: "source ${HOME}/venv${PYTHON_VERSION}/bin/activate"
+- sh: "python --version"
+- make init
+
+test_script:
+- make integ-test
+