fix: Py27hash fix (#2182)

* Add third party py27hash code

* Add Py27UniStr and unit tests

* Add py27hash_fix utils and tests

* Add to_py27_compatible_template and tests

* Apply py27hash fix to wherever it is needed

* Apply py27hash fix, all tests pass except api_with_any_method_in_swagger

* apply py27hash fix in openapi + run black

* remove py27 testing

* remove other py27 references

* black fixes

* fixes/typos

* remove py27 from tox.ini

* refactoring

* third party notice

* black

* Fix py27hash fix to deal with null events

* Fix Py27UniStr repr for unicode literals

* black reformat

* Update _template_has_api_resource to check data type more defensively

* Apply py27Dict in _get_authorizers

* Apply Py27Dict to authorizers and gateway responses which will go into swagger

* Update to_py27_compatible_template to handle parameter_values; Add Py27LongInt class

* Rename _convert_to_py27_dict to _convert_to_py27_type

* Apply Py27UniStr to path param name

* Handle HttpApi resource under to_py27_compatible_template

* Fix InvalidDocumentException to not sort different exceptions

* black reformat

* Remove unnecessary test files

Co-authored-by: Wing Fung Lau <[email protected]>

Author: torresxb1

DEVELOPMENT_GUIDE.md

@@ -26,9 +26,8 @@ Environment Setup
 -----------------
 ### 1. Install Python Versions
 
-Our officially supported Python versions are 2.7, 3.6, 3.7 and 3.8. Follow the idioms from this [excellent cheatsheet](http://python-future.org/compatible_idioms.html) to
-make sure your code is compatible with both Python 2.7 and 3 (>=3.6) versions.
-Our CI/CD pipeline is setup to run unit tests against both Python 2.7 and 3 versions. So make sure you test it with both versions before sending a Pull Request.
+Our officially supported Python versions are 3.6, 3.7 and 3.8. 
+Our CI/CD pipeline is setup to run unit tests against Python 3 versions. Make sure you test it before sending a Pull Request.
 See [Unit testing with multiple Python versions](#unit-testing-with-multiple-python-versions).
 
 [pyenv](https://github.com/pyenv/pyenv) is a great tool to
@@ -41,12 +40,11 @@ easily setup multiple Python versions. For
 1.  Install PyEnv -
     `curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash`
 1. Restart shell so the path changes take effect - `exec $SHELL`
-1.  `pyenv install 2.7.17`
 1.  `pyenv install 3.6.12`
 1.  `pyenv install 3.7.9`
 1.  `pyenv install 3.8.6`
 1.  Make Python versions available in the project:
-    `pyenv local 2.7.17 3.6.12 3.7.9 3.8.6`
+    `pyenv local 3.6.12 3.7.9 3.8.6`
 
 Note: also make sure the following lines were written into your `.bashrc` (or `.zshrc`, depending on which shell you are using):
 ```
@@ -117,11 +115,10 @@ Running Tests
 ### Unit testing with one Python version
 
 If you're trying to do a quick run, it's ok to use the current python version.  Run `make pr`.
-If you're using Python2.7, you can run `make pr2.7` instead.
 
 ### Unit testing with multiple Python versions
 
-Currently, our officially supported Python versions are 2.7, 3.6, 3.7 and 3.8. For the most
+Currently, our officially supported Python versions are 3.6, 3.7 and 3.8. For the most
 part, code that works in Python3.6 will work in Python3.7 and Python3.8. You only run into problems if you are
 trying to use features released in a higher version (for example features introduced into Python3.7
 will not work in Python3.6). If you want to test in many versions, you can create a virtualenv for

MANIFEST.in

@@ -5,5 +5,6 @@ recursive-include samtranslator/validator/sam_schema *.json
 include samtranslator/policy_templates_data/policy_templates.json
 include samtranslator/policy_templates_data/schema.json
 include README.md
+include THIRD_PARTY_LICENSES
 
 prune tests

Makefile

@@ -26,9 +26,6 @@ dev: test
 # Verifications to run before sending a pull request
 pr: black-check init dev
 
-# Verifications to run before sending a pull request, skipping black check because black requires Python 3.6+
-pr2.7: init dev
-
 define HELP_MESSAGE
 
 Usage: $ make [TARGETS]

THIRD_PARTY_LICENSES

@@ -0,0 +1,23 @@
+The AWS Serverless Application Model includes the following third-party software/licensing:
+
+** py27hash; version 1.0.2 -- https://pypi.org/project/py27hash/
+Copyright (c) 2020 NeuML LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+----------------

appveyor-integration-test.yml

@@ -3,8 +3,6 @@ image: Ubuntu
 
 environment:
   matrix:
-  - TOXENV: py27
-    PYTHON_VERSION: '2.7'
   - TOXENV: py36
     PYTHON_VERSION: '3.6'
   - TOXENV: py37

appveyor.yml

@@ -3,8 +3,6 @@ image: Ubuntu
 
 environment:
   matrix:
-  - TOXENV: py27
-    PYTHON_VERSION: '2.7'
   - TOXENV: py36
     PYTHON_VERSION: '3.6'
   - TOXENV: py37

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.black]
 line-length = 120
-target_version = ['py27', 'py37', 'py36', 'py38']
+target_version = ['py37', 'py36', 'py38']
 exclude = '''
 
 (

samtranslator/intrinsics/actions.py

@@ -1,6 +1,7 @@
 import re
 
 from six import string_types
+from samtranslator.utils.py27hash_fix import Py27UniStr
 from samtranslator.model.exceptions import InvalidTemplateException, InvalidDocumentException
 
 
@@ -375,13 +376,13 @@ def handler_method(full_ref, ref_value):
 
         # Find all the pattern, and call the handler to decide how to substitute them.
         # Do the substitution and return the final text
-        return re.sub(
-            ref_pattern,
-            # Pass the handler entire string ${logicalId.property} as first parameter and "logicalId.property"
-            # as second parameter. Return value will be substituted
-            lambda match: handler_method(match.group(0), match.group(1)),
-            text,
-        )
+        # NOTE: in order to make sure Py27UniStr strings won't be converted to plain string,
+        # we need to iterate through each match and do the replacement
+        substituted = text
+        for match in re.finditer(ref_pattern, text):
+            sub_value = handler_method(match.group(0), match.group(1))
+            substituted = substituted.replace(match.group(0), sub_value, 1)
+        return substituted
 
 
 class GetAttAction(Action):

samtranslator/model/api/api_generator.py

@@ -27,6 +27,7 @@
 from samtranslator.translator import logical_id_generator
 from samtranslator.translator.arn_generator import ArnGenerator
 from samtranslator.model.tags.resource_tagging import get_tag_list
+from samtranslator.utils.py27hash_fix import Py27Dict, Py27UniStr
 
 LOG = logging.getLogger(__name__)
 
@@ -338,7 +339,18 @@ def _construct_body_s3_dict(self):
                     "'s3://bucket/key' with optional versionId query parameter.",
                 )
 
-        body_s3 = {"Bucket": s3_pointer["Bucket"], "Key": s3_pointer["Key"]}
+            if isinstance(self.definition_uri, Py27UniStr):
+                # self.defintion_uri is a Py27UniStr instance if it is defined in the template
+                # we need to preserve the Py27UniStr type
+                s3_pointer["Bucket"] = Py27UniStr(s3_pointer["Bucket"])
+                s3_pointer["Key"] = Py27UniStr(s3_pointer["Key"])
+                if "Version" in s3_pointer:
+                    s3_pointer["Version"] = Py27UniStr(s3_pointer["Version"])
+
+        # Construct body_s3 as py27 dict
+        body_s3 = Py27Dict()
+        body_s3["Bucket"] = s3_pointer["Bucket"]
+        body_s3["Key"] = s3_pointer["Key"]
         if "Version" in s3_pointer:
             body_s3["Version"] = s3_pointer["Version"]
         return body_s3
@@ -922,12 +934,13 @@ def _add_gateway_responses(self):
 
         swagger_editor = SwaggerEditor(self.definition_body)
 
-        gateway_responses = {}
+        # The dicts below will eventually become part of swagger/openapi definition, thus requires using Py27Dict()
+        gateway_responses = Py27Dict()
         for response_type, response in self.gateway_responses.items():
             gateway_responses[response_type] = ApiGatewayResponse(
                 api_logical_id=self.logical_id,
-                response_parameters=response.get("ResponseParameters", {}),
-                response_templates=response.get("ResponseTemplates", {}),
+                response_parameters=response.get("ResponseParameters", Py27Dict()),
+                response_templates=response.get("ResponseTemplates", Py27Dict()),
                 status_code=response.get("StatusCode", None),
             )
 
@@ -985,12 +998,12 @@ def _openapi_postprocess(self, definition_body):
             SwaggerEditor.get_openapi_version_3_regex(), self.open_api_version
         ):
             if definition_body.get("securityDefinitions"):
-                components = definition_body.get("components", {})
+                components = definition_body.get("components", Py27Dict())
                 components["securitySchemes"] = definition_body["securityDefinitions"]
                 definition_body["components"] = components
                 del definition_body["securityDefinitions"]
             if definition_body.get("definitions"):
-                components = definition_body.get("components", {})
+                components = definition_body.get("components", Py27Dict())
                 components["schemas"] = definition_body["definitions"]
                 definition_body["components"] = components
                 del definition_body["definitions"]
@@ -1016,15 +1029,17 @@ def _openapi_postprocess(self, definition_body):
                                 if field_val.get("200") and field_val.get("200").get("headers"):
                                     headers = field_val["200"]["headers"]
                                     for header, header_val in headers.items():
-                                        new_header_val_with_schema = {"schema": header_val}
+                                        new_header_val_with_schema = Py27Dict()
+                                        new_header_val_with_schema["schema"] = header_val
                                         definition_body["paths"][path]["options"][field]["200"]["headers"][
                                             header
                                         ] = new_header_val_with_schema
 
         return definition_body
 
     def _get_authorizers(self, authorizers_config, default_authorizer=None):
-        authorizers = {}
+        # The dict below will eventually become part of swagger/openapi definition, thus requires using Py27Dict()
+        authorizers = Py27Dict()
         if default_authorizer == "AWS_IAM":
             authorizers[default_authorizer] = ApiGatewayAuthorizer(
                 api_logical_id=self.logical_id, name=default_authorizer, is_aws_iam_authorizer=True

samtranslator/model/apigateway.py

@@ -1,11 +1,13 @@
 import json
 from re import match
+from functools import reduce
 from samtranslator.model import PropertyType, Resource
 from samtranslator.model.exceptions import InvalidResourceException
 from samtranslator.model.types import is_type, one_of, is_str, list_of
 from samtranslator.model.intrinsics import ref, fnSub
 from samtranslator.translator import logical_id_generator
 from samtranslator.translator.arn_generator import ArnGenerator
+from samtranslator.utils.py27hash_fix import Py27Dict, Py27UniStr
 
 
 class ApiGatewayRestApi(Resource):
@@ -128,15 +130,16 @@ def __init__(self, api_logical_id=None, response_parameters=None, response_templ
             raise InvalidResourceException(api_logical_id, "Property 'StatusCode' must be numeric")
 
         self.api_logical_id = api_logical_id
-        self.response_parameters = response_parameters or {}
-        self.response_templates = response_templates or {}
+        # Defaults to Py27Dict() as these will go into swagger
+        self.response_parameters = response_parameters or Py27Dict()
+        self.response_templates = response_templates or Py27Dict()
         self.status_code = status_code_str
 
     def generate_swagger(self):
-        swagger = {
-            "responseParameters": self._add_prefixes(self.response_parameters),
-            "responseTemplates": self.response_templates,
-        }
+        # Applying Py27Dict here as this goes into swagger
+        swagger = Py27Dict()
+        swagger["responseParameters"] = self._add_prefixes(self.response_parameters)
+        swagger["responseTemplates"] = self.response_templates
 
         # Prevent "null" being written.
         if self.status_code:
@@ -146,13 +149,17 @@ def generate_swagger(self):
 
     def _add_prefixes(self, response_parameters):
         GATEWAY_RESPONSE_PREFIX = "gatewayresponse."
-        prefixed_parameters = {}
-        for key, value in response_parameters.get("Headers", {}).items():
-            prefixed_parameters[GATEWAY_RESPONSE_PREFIX + "header." + key] = value
-        for key, value in response_parameters.get("Paths", {}).items():
-            prefixed_parameters[GATEWAY_RESPONSE_PREFIX + "path." + key] = value
-        for key, value in response_parameters.get("QueryStrings", {}).items():
-            prefixed_parameters[GATEWAY_RESPONSE_PREFIX + "querystring." + key] = value
+        # applying Py27Dict as this is part of swagger
+        prefixed_parameters = Py27Dict()
+
+        parameter_prefix_pairs = [("Headers", "header."), ("Paths", "path."), ("QueryStrings", "querystring.")]
+        for parameter, prefix in parameter_prefix_pairs:
+            for key, value in response_parameters.get(parameter, {}).items():
+                param_key = GATEWAY_RESPONSE_PREFIX + prefix + key
+                if isinstance(key, Py27UniStr):
+                    # if key is from template, we need to convert param_key to Py27UniStr
+                    param_key = Py27UniStr(param_key)
+                prefixed_parameters[param_key] = value
 
         return prefixed_parameters
 
@@ -288,21 +295,20 @@ def _is_missing_identity_source(self, identity):
     def generate_swagger(self):
         authorizer_type = self._get_type()
         APIGATEWAY_AUTHORIZER_KEY = "x-amazon-apigateway-authorizer"
-        swagger = {
-            "type": "apiKey",
-            "name": self._get_swagger_header_name(),
-            "in": "header",
-            "x-amazon-apigateway-authtype": self._get_swagger_authtype(),
-        }
+        swagger = Py27Dict()
+        swagger["type"] = "apiKey"
+        swagger["name"] = self._get_swagger_header_name()
+        swagger["in"] = "header"
+        swagger["x-amazon-apigateway-authtype"] = self._get_swagger_authtype()
 
         if authorizer_type == "COGNITO_USER_POOLS":
-            swagger[APIGATEWAY_AUTHORIZER_KEY] = {
-                "type": self._get_swagger_authorizer_type(),
-                "providerARNs": self._get_user_pool_arn_array(),
-            }
+            authorizer_dict = Py27Dict()
+            authorizer_dict["type"] = self._get_swagger_authorizer_type()
+            authorizer_dict["providerARNs"] = self._get_user_pool_arn_array()
+            swagger[APIGATEWAY_AUTHORIZER_KEY] = authorizer_dict
 
         elif authorizer_type == "LAMBDA":
-            swagger[APIGATEWAY_AUTHORIZER_KEY] = {"type": self._get_swagger_authorizer_type()}
+            swagger[APIGATEWAY_AUTHORIZER_KEY] = Py27Dict({"type": self._get_swagger_authorizer_type()})
             partition = ArnGenerator.get_partition_name()
             resource = "lambda:path/2015-03-31/functions/${__FunctionArn__}/invocations"
             authorizer_uri = fnSub(
@@ -341,35 +347,39 @@ def generate_swagger(self):
     def _get_identity_validation_expression(self):
         return self.identity and self.identity.get("ValidationExpression")
 
-    def _get_identity_source(self):
-        identity_source_headers = []
-        identity_source_query_strings = []
-        identity_source_stage_variables = []
-        identity_source_context = []
-
-        if self.identity.get("Headers"):
-            identity_source_headers = list(map(lambda h: "method.request.header." + h, self.identity.get("Headers")))
-
-        if self.identity.get("QueryStrings"):
-            identity_source_query_strings = list(
-                map(lambda qs: "method.request.querystring." + qs, self.identity.get("QueryStrings"))
-            )
+    def _build_identity_source_item(self, item_prefix, prop_value):
+        item = item_prefix + prop_value
+        if isinstance(prop_value, Py27UniStr):
+            item = Py27UniStr(item)
+        return item
 
-        if self.identity.get("StageVariables"):
-            identity_source_stage_variables = list(
-                map(lambda sv: "stageVariables." + sv, self.identity.get("StageVariables"))
-            )
-
-        if self.identity.get("Context"):
-            identity_source_context = list(map(lambda c: "context." + c, self.identity.get("Context")))
+    def _build_identity_source_item_array(self, prop_key, item_prefix):
+        arr = []
+        if self.identity.get(prop_key):
+            arr = [
+                self._build_identity_source_item(item_prefix, prop_value) for prop_value in self.identity.get(prop_key)
+            ]
+        return arr
 
-        identity_source_array = (
-            identity_source_headers
-            + identity_source_query_strings
-            + identity_source_stage_variables
-            + identity_source_context
+    def _get_identity_source(self):
+        key_prefix_pairs = [
+            ("Headers", "method.request.header."),
+            ("QueryStrings", "method.request.querystring."),
+            ("StageVariables", "stageVariables."),
+            ("Context", "context."),
+        ]
+
+        identity_source_array = reduce(
+            lambda accumulator, key_prefix_pair: accumulator
+            + self._build_identity_source_item_array(key_prefix_pair[0], key_prefix_pair[1]),
+            key_prefix_pairs,
+            [],
         )
+
         identity_source = ", ".join(identity_source_array)
+        if any(isinstance(i, Py27UniStr) for i in identity_source_array):
+            # Convert identity_source to Py27UniStr if any part of it is Py27UniStr
+            identity_source = Py27UniStr(identity_source)
 
         return identity_source