feature: ecs,vpc,sg optionals (#57)

* chore: ecs,vpc,sg optionals
* refact: sepparate sns/sns permissions
* refact: /ecs-vpc-secgroup/ecs-vpc/
* docs: add troubleshoot sns
* docs: reorganize examples-internal
* docs: use-case for org-existing-cloudtrail-ecs-vpc-subnet
* docs: fix compliance docs link
* docs: fix pre-commit and add notes for contributing -_-
* docs: update diagrams
* docs: link readme with use-cases
* docs: cloudtrail org role default clarification

Author: iru

.github/workflows/ci-integration-tests.yaml

@@ -74,7 +74,12 @@ jobs:
           TF_VAR_cloudnative_secretAccessKey: ${{ secrets.AWS_QA_CLOUDNATIVE_SECRET_ACCESS_KEY }}
           TF_VAR_region: ${{secrets.AWS_QA_MANAGED_RESOURCES_REGION }}
           TF_VAR_cloudtrail_s3_name: ${{ secrets.AWS_QA_MANAGED_CLOUDTRAIL_NAME }}
-        run: bundle exec kitchen test "organizational-k8s-aws"
+        run: |
+          bundle exec kitchen create "organizational-k8s-aws"
+          bundle exec kitchen converge "organizational-k8s-aws"
+          bundle exec kitchen setup "organizational-k8s-aws"
+          bundle exec kitchen verify "organizational-k8s-aws"
+          bundle exec kitchen destroy "organizational-k8s-aws"
 
       - name: Inspect k8s failures
         if: ${{ failure() }}
@@ -139,7 +144,12 @@ jobs:
           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_QA_MANAGED_SECRET_ACCESS_KEY }}
           AWS_REGION: ${{ secrets.AWS_REGION }}
           TF_VAR_sysdig_secure_for_cloud_member_account_id: ${{ secrets.AWS_QA_CLOUDNATIVE_ACCOUNT_ID }}
-        run: bundle exec kitchen test "organizational-aws"
+        run: |
+          bundle exec kitchen create "organizational-aws"
+          bundle exec kitchen converge "organizational-aws"
+          bundle exec kitchen setup "organizational-aws"
+          bundle exec kitchen verify "organizational-aws"
+          bundle exec kitchen destroy "organizational-aws"
 
       - name: Destroy organizational resources
         env:

.github/workflows/ci-pull-request.yaml

@@ -9,6 +9,8 @@ on:
       - master
     paths:
       - '**.tf'
+env:
+  GO_VERSION: "^1.17"
 
 jobs:
   getModules:
@@ -100,14 +102,18 @@ jobs:
         uses: actions/checkout@v2
       - name: Install Python
         uses: actions/setup-python@v2
+      - name: Set up Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: ${{ env.GO_VERSION }}
       - name: Install Terraform v${{ matrix.version }}
         uses: hashicorp/setup-terraform@v1
         with:
           terraform_version: ${{ matrix.version }}
       - name: Install pre-commit dependencies
         run: |
           pip install pre-commit
-          curl -Lo ./terraform-docs.tar.gz https://github.com/terraform-docs/terraform-docs/releases/download/v0.13.0/terraform-docs-v0.13.0-$(uname)-amd64.tar.gz && tar -xzf terraform-docs.tar.gz && chmod +x terraform-docs && sudo mv terraform-docs /usr/bin/
+          go install github.com/terraform-docs/terraform-docs@v0.16.0
           curl -L "$(curl -s https://api.github.com/repos/terraform-linters/tflint/releases/latest | grep -o -E "https://.+?_linux_amd64.zip")" > tflint.zip && unzip tflint.zip && rm tflint.zip && sudo mv tflint /usr/bin/
       - name: Execute pre-commit
         # Run all pre-commit checks on max version supported

.gitignore

@@ -40,6 +40,7 @@ override.tf.json
 /lib/bundler/man/
 
 
-#Kitchen test
+# test
 .kitchen/*
 test/fixtures/single-account/.kitchen/
+test/snippets/*

.pre-commit-config.yaml

@@ -1,9 +1,9 @@
 #
 # requirements
 # - install pre-commit (latest)
-# - install terraform-docs (0.13)
+# - install terraform-docs (latest)
 # - see .github/workflows/ci-pull-request.yaml for more info on what's being valiated on CI
-# `$ pre-commit run --color=always --show-diff-on-failure --all-files`
+# $ pre-commit run --color=always --show-diff-on-failure --all-files
 #
 
 

CONTRIBUTE.md

@@ -59,6 +59,15 @@ We're using **pre-commit** |  https://pre-commit.com
   <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
   ```
 
+- If pre-commit fails on Github but not on your local, try cleaning-up `terraform` files with
+```bash
+-- resources/terraform-clean.sh
+$ find . -name ".terraform" -exec rm -fr {} \;
+$ find . -name "terraform.tfstate*" -exec rm -fr {} \;
+$ find . -name ".terraform.lock.hcl*" -exec rm -fr {} \;
+```
+
+
 ## 2. Check::Integration tests
 
 Final user validation. Checks that the snippets for the usage, stated in the official Sysdig Terraform Registry, are working correctly.

README.md

@@ -5,7 +5,7 @@ Terraform module that deploys the [**Sysdig Secure for Cloud** stack in **AWS**]
 
 Provides unified threat-detection, compliance, forensics and analysis through these major components:
 
-* **[CSPM/Compliance](https://docs.sysdig.com/en/docs/sysdig-secure/benchmarks/)**: It evaluates periodically your cloud configuration, using Cloud Custodian, against some benchmarks and returns the results and remediation you need to fix. Managed through `cloud-bench` module. <br/>
+* **[CSPM/Compliance](https://docs.sysdig.com/en/docs/sysdig-secure/posture/compliance-unified-/)**: It evaluates periodically your cloud configuration, using Cloud Custodian, against some benchmarks and returns the results and remediation you need to fix. Managed through `cloud-bench` module. <br/>
 
 * **[CIEM](https://docs.sysdig.com/en/docs/sysdig-secure/posture/)**: Permissions and Entitlements management. Requires BOTH modules  `cloud-connector` and `cloud-bench`. <br/>
 
@@ -41,7 +41,12 @@ For other Cloud providers check: [GCP](https://github.com/sysdiglabs/terraform-g
 
 ## Usage
 
-There are several ways to deploy this in you AWS infrastructure:
+  - There are several ways to deploy this in you AWS infrastructure, gathered under **[`/examples`](./examples)**
+  - Many module,examples and use-cases provide ways to **re-use existing resources (as optionals)** in your infrastructure (cloudtrail, ecs, vpc, k8s cluster,...)
+  - Find some real **use-case scenario explanations** under [`/examples-internal/use-cases*`](./examples-internal)
+    - [Single Account - Existing Cloudtrail](./examples-internal/use-cases-reuse-resources/single-existing-cloudtrail.md)
+    - [Organizational - Existing Cloudtrail, ECS, VPC, Subnet](./examples-internal/use-cases-reuse-resources/org-existing-cloudtrail-ecs-vpc-subnet.md)
+    - [Organizational - Existing Cloudtrail withouth SNS, but with S3 configuration, with K8s Cluster and Filtered Cloudtrail Event Account](./examples-internal/use-cases-self-baked/org-s3-k8s-filtered-account.md)
 
 ### - Single-Account
 
@@ -135,9 +140,23 @@ Upload any image to the ECR repository of AWS.
 
 ## Troubleshooting
 
+### Q: Getting error "404 Invalid parameter: TopicArn" when trying to reuse an existing cloudtrail-sns
+
+```text
+│ Error: error creating SNS Topic Subscription: InvalidParameter: Invalid parameter: TopicArn
+│ 	status code: 400, request id: 1fe94ceb-9f58-5d39-a4df-169f55d25eba
+│
+│   with module.cloudvision_aws_single_account.module.cloud_connector.module.cloud_connector_sqs.aws_sns_topic_subscription.this,
+│   on ../../../modules/infrastructure/sqs-sns-subscription/main.tf line 6, in resource "aws_sns_topic_subscription" "this":
+│    6: resource "aws_sns_topic_subscription" "this" {
 
-### Q: Getting error when creating the ECS subnet due to nats not being supported
 ```
+
+A: In order to subscribe to a SNS Topic, SQS queue must be in the same region
+S: Change `aws provider` `region` variable to match same region for all resources
+
+### Q: Getting error "400 availabilityZoneId is invalid" when creating the ECS subnet
+```text
 │ Error: error creating subnet: InvalidParameterValue: Value (apne1-az3) for parameter availabilityZoneId is invalid. Subnets can currently only be created in the following availability zones: apne1-az1, apne1-az2, apne1-az4.
 │ 	status code: 400, request id: 6e32d757-2e61-4220-8106-22ccf814e1fe
 │

examples-internal/organizational-k8s-threat-reuse_cloudtrail/README.md renamed to examples-internal/organizational-k8s-threat-reuse_cloudtrail_s3/README.md

@@ -81,15 +81,15 @@ Notice that:
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.50.0 |
-| <a name="provider_helm"></a> [helm](#provider\_helm) | >=2.3.0 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | 3.74.0 |
+| <a name="provider_helm"></a> [helm](#provider\_helm) | 2.4.1 |
 
 ## Modules
 
 | Name | Source | Version |
 |------|--------|---------|
-| <a name="module_resource_group"></a> [resource\_group](#module\_resource\_group) | ../../modules/infrastructure/resource-group |  |
-| <a name="module_ssm"></a> [ssm](#module\_ssm) | ../../modules/infrastructure/ssm |  |
+| <a name="module_resource_group"></a> [resource\_group](#module\_resource\_group) | ../../modules/infrastructure/resource-group | n/a |
+| <a name="module_ssm"></a> [ssm](#module\_ssm) | ../../modules/infrastructure/ssm | n/a |
 
 ## Resources