Test request throws exception with cloudformation-cli 0.1.6. Fails to parse new fields.

[drch-]

aws-cloudformation/cloudformation-cli@b8466be added new fields to the test event, which are causing an exception when running cfn test

START RequestId: af819c5a-99f5-1d9c-ad69-7614c2c1d235 Version: $LATEST
[ERROR]	2020-08-07T21:56:25.652Z	af819c5a-99f5-1d9c-ad69-7614c2c1d235	Invalid request
Traceback (most recent call last):
  File "/var/task/cloudformation_cli_python_lib/resource.py", line 103, in _parse_test_request
    **event.request
TypeError: __init__() got an unexpected keyword argument 'region'
[ERROR]	2020-08-07T21:56:25.653Z	af819c5a-99f5-1d9c-ad69-7614c2c1d235	Handler error
Traceback (most recent call last):
  File "/var/task/cloudformation_cli_python_lib/resource.py", line 103, in _parse_test_request
    **event.request
TypeError: __init__() got an unexpected keyword argument 'region'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/task/cloudformation_cli_python_lib/resource.py", line 119, in test_entrypoint
    session, request, action, callback_context = self._parse_test_request(event)
  File "/var/task/cloudformation_cli_python_lib/resource.py", line 110, in _parse_test_request
    raise InternalFailure(f"{e} ({type(e).__name__})") from e
cloudformation_cli_python_lib.exceptions.InternalFailure: __init__() got an unexpected keyword argument 'region' (TypeError)
END RequestId: af819c5a-99f5-1d9c-ad69-7614c2c1d235

[wnkz]

It looks like the wrong cloudformation-cli-python-lib is packaged in the build directory after running cfn submit --dry-run.

For example, after build if I check in build/cloudformation_cli_python_lib/utils.py I can see that the class UnmodelledRequest does not match current version.

@dataclass
class UnmodelledRequest:
    clientRequestToken: str
    desiredResourceState: Optional[Mapping[str, Any]] = None
    previousResourceState: Optional[Mapping[str, Any]] = None
    logicalResourceIdentifier: Optional[str] = None
    nextToken: Optional[str] = None

    def to_modelled(self, model_cls: Type[BaseModel]) -> BaseResourceHandlerRequest:
        # pylint: disable=protected-access
        return BaseResourceHandlerRequest(
            clientRequestToken=self.clientRequestToken,
            desiredResourceState=model_cls._deserialize(self.desiredResourceState),
            previousResourceState=model_cls._deserialize(self.previousResourceState),
            logicalResourceIdentifier=self.logicalResourceIdentifier,
            nextToken=self.nextToken,
        )

This might be caused by the way the libraries are used / deployed. We have cloudformation-cli, cloudformation-cli-python-plugin and cloudformation-cli-python-lib and the latter does not seem to have a new version on pypi.

Did the maintainers forgot to make a new release for this package ? Version in setup.py is still 2.0.0 by the way

cloudformation-cli-python-plugin/src/setup.py

Line 6 in d00dd26

version="2.0.0",

ping @PatMyron ? 🙏

[johnttompkins]

It looks like the wrong cloudformation-cli-python-lib is packaged in the build directory after running cfn submit --dry-run.

For example, after build if I check in build/cloudformation_cli_python_lib/utils.py I can see that the class UnmodelledRequest does not match current version.

@dataclass
class UnmodelledRequest:
    clientRequestToken: str
    desiredResourceState: Optional[Mapping[str, Any]] = None
    previousResourceState: Optional[Mapping[str, Any]] = None
    logicalResourceIdentifier: Optional[str] = None
    nextToken: Optional[str] = None

    def to_modelled(self, model_cls: Type[BaseModel]) -> BaseResourceHandlerRequest:
        # pylint: disable=protected-access
        return BaseResourceHandlerRequest(
            clientRequestToken=self.clientRequestToken,
            desiredResourceState=model_cls._deserialize(self.desiredResourceState),
            previousResourceState=model_cls._deserialize(self.previousResourceState),
            logicalResourceIdentifier=self.logicalResourceIdentifier,
            nextToken=self.nextToken,
        )

This might be caused by the way the libraries are used / deployed. We have cloudformation-cli, cloudformation-cli-python-plugin and cloudformation-cli-python-lib and the latter does not seem to have a new version on pypi.

Did the maintainers forgot to make a new release for this package ? Version in setup.py is still 2.0.0 by the way

cloudformation-cli-python-plugin/src/setup.py

Line 6 in d00dd26

version="2.0.0",

ping @PatMyron ? 🙏

It looks as if we haven't added these fields into the test request that is used when invoking the contract test specific entrypoint:

cloudformation-cli-python-plugin/src/cloudformation_cli_python_lib/resource.py

Lines 91 to 100 in 797b9b3

	def _parse_test_request(
	self, event_data: MutableMapping[str, Any]
	) -> Tuple[
	Optional[SessionProxy],
	BaseResourceHandlerRequest,
	Action,
	MutableMapping[str, Any],
	]:
	try:
	event = TestEvent(**event_data)

TestEvent here is missing the options passed in by contract tests:

cloudformation-cli-python-plugin/src/cloudformation_cli_python_lib/utils.py

Lines 31 to 37 in 64f454e

	@dataclass
	class TestEvent:
	credentials: Mapping[str, str]
	action: Action
	request: Mapping[str, Any]
	callbackContext: MutableMapping[str, Any] = field(default_factory=dict)
	region: Optional[str] = None

We need to add region, partition, and account to this request to this event so that the test requests can properly deserialize and release a new version. As a mitigation to unblock testing, you can install cloudformation-cli version 0.1.5 or below where these keys were not included in the test request as they were added in 0.1.6: https://github.com/aws-cloudformation/cloudformation-cli/releases/tag/v0.1.6. We will look into getting these added so the latest cloudformation-cli can be used.

[wnkz]

Downgrading does not work and I don't think the issue is what you described. I'll try to open another issue detailing my findings as this is a much broader problem than described here.

[wesleywh]

Has anyone come up with a solution to this yet? I'm developing my own custom resource and I have just hit this issue as well. Makes it impossible to run any tests with this or debug my code...

[jamesoff]

I was able to work around this by doing the following:

• Clone this repo
• Apply this patch

diff --git a/src/cloudformation_cli_python_lib/utils.py b/src/cloudformation_cli_python_lib/utils.py
index e4921ff..47f8aab 100644
--- a/src/cloudformation_cli_python_lib/utils.py
+++ b/src/cloudformation_cli_python_lib/utils.py
@@ -119,6 +119,7 @@ class UnmodelledRequest:
     logicalResourceIdentifier: Optional[str] = None
     nextToken: Optional[str] = None
     region: Optional[str] = None
+    awsPartition: Optional[str] = None

     def to_modelled(self, model_cls: Type[BaseModel]) -> BaseResourceHandlerRequest:
         # pylint: disable=protected-access

• cd src; python setup.py sdist
• Copy the .tar.gz file in src/dist to your custom resource project
• Change the line in requirements.txt in your custom resource project to /project/cloudformation-cli-python-lib-2.0.0.tar.gz

This was enough for me to build a resource which passes cfn test and works after being submitted in us-east-1, at least.

[miparnisari]

aws-cloudformation/cloudformation-cli@b8466be added new fields to the test event, which are causing an exception when running cfn test

Ping @anshikg ?

[ammokhov]

closing as being tracked in #112