v3.0.0

tianchu · tianchu · commit 22d3a3fca6d4 · 2020-01-17T15:34:24.000-05:00
diff --git a/aws/logs_monitoring/.gitignore b/aws/logs_monitoring/.gitignore
@@ -0,0 +1 @@
+*.zip
diff --git a/aws/logs_monitoring/README.md b/aws/logs_monitoring/README.md
@@ -1,129 +1,72 @@
-**IMPORTANT NOTE: When upgrading, please ensure your forwarder Lambda function has [the latest Datadog Lambda Layer installed](https://github.com/DataDog/datadog-serverless-functions/tree/master/aws/logs_monitoring#3-add-the-datadog-lambda-layer).**
-
 # Datadog Forwarder
 
-AWS Lambda function to ship logs and metrics from ELB, S3, CloudTrail, VPC, CloudFront, and CloudWatch logs to Datadog
+AWS Lambda function to ship ELB, S3, CloudTrail, VPC, CloudFront logs from S3 buckets, and Lambda metrics, traces and logs from CloudWatch logs to Datadog.
 
 ## Features
 
-- Forward logs through HTTPS (defaulted to port 443)
-- Use AWS Lambda to re-route triggered S3 events to Datadog
-- Use AWS Lambda to re-route triggered Kinesis data stream events to Datadog, only the Cloudwatch logs are supported
-- Cloudwatch, ELB, S3, CloudTrail, VPC and CloudFront logs can be forwarded
-- SSL Security
-- JSON events providing details about S3 documents forwarded
-- Structured meta-information can be attached to the events
-- Scrubbing / Redaction rules
-- Filtering rules (`INCLUDE_AT_MATCH` and `EXCLUDE_AT_MATCH`)
-- Multiline Log Support (S3 Only)
-- Forward custom metrics from logs
-- Submit `aws.lambda.enhanced.*` Lambda metrics parsed from the AWS REPORT log: duration, billed_duration, max_memory_used, estimated_cost
-
-## Quick Start
-
-The provided Python script must be deployed into your AWS Lambda service to collect your logs and send them to Datadog.
-
-### 1. Create a new Lambda function
-
-1. [Navigate to the Lambda console](https://console.aws.amazon.com/lambda/home) and create a new function.
-2. Select `Author from scratch` and give the function a unique name: `datadog-log-monitoring-function`
-3. For `Role`, select `Create new role from template(s)` and give the role a unique name: `datadog-log-monitoring-function-role`
-4. Under Policy templates, select `s3 object read-only permissions`.
-
-### 2. Provide the code
-
-1. Copy paste the code of the Lambda function from the `lambda_function.py` file.
-2. Set the runtime to `Python 2.7`, `Python 3.6`, or `Python 3.7`
-3. Set the handler to `lambda_function.lambda_handler`
+- Forward CloudWatch, ELB, S3, CloudTrail, VPC and CloudFront logs to Datadog
+- Forward S3 events to Datadog
+- Forward Kinesis data stream events to Datadog, only CloudWatch logs are supported
+- Forward custom metrics from AWS Lambda functions via CloudWatch logs
+- Generate and submit enhanced Lambda metrics (`aws.lambda.enhanced.*`) parsed from the AWS REPORT log: duration, billed_duration, max_memory_used, and estimated_cost
 
-### 3. Add the Datadog Lambda Layer
-The [Datadog Lambda Layer]((https://github.com/DataDog/datadog-lambda-layer-python)) **MUST** be added to the log forwarder Lambda function. Use the Lambda layer ARN below, and replace `<AWS_REGION>` with the actual region (e.g., `us-east-1`), `<PYTHON_RUNTIME>` with the runtime of your forwarder (e.g., `Python27`), and `<VERSION>` with the latest version from the [CHANGELOG](https://github.com/DataDog/datadog-lambda-layer-python/blob/master/CHANGELOG.md).
+## Installation
 
-```
-arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-<PYTHON_RUNTIME>:<VERSION>
-```
+Use the [AWS Serverless Repository](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:464622532012:applications~Datadog-Log-Forwarder) to deploy the Lambda in your AWS account.
 
-For example:
+## Basic Settings
 
-```
-arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Python37:8
-```
+The basic settings can be configured through the SAM app installation interface. Some of them are simply mapped to environment variables that can be updated after the installation from the Lambda configuration console.
 
+### Application Name
 
-### 4. Set your Parameters
+DO **NOT** change unless for advanced use cases. If changed, you **MUST** provide the same application name when upgrade in the future.
 
-At the top of the script you'll find a section called `PARAMETERS`, that's where you want to edit your code, available paramters are:
+### Datadog API Key (REQUIRED)
 
-#### DD_API_KEY
-
-Set the Datadog API key for your Datadog platform, it can be found here:
+The Datadog API key for your Datadog platform MUST be provided, it can be found here:
 
 * Datadog US Site: https://app.datadoghq.com/account/settings#api
 * Datadog EU Site: https://app.datadoghq.eu/account/settings#api
 
-There are 3 possibilities to set your Datadog API key:
+Provide the API key using one of the following SAM application settings:
+
+* `DdApiKeySecretArn` (or environment variable `DD_API_KEY_SECRET_ARN`): Recommended. The Secret ARN to fetch the Datadog API key from Secrets Manager. `KMSKeyId` also needs to be provided if a customer managed CMK is used for encryption.
+  * Permission `secretsmanager:GetSecretValue` to access the provided Secret ARN will be added automatically.
+  * Permission `kms:Decrypt` to access the `KMSKeyId`, if provided, will be added automatically.
+* `DdKmsApiKey` (or environment variable `DD_KMS_API_KEY`): Recommended. The Datadog API Key encrypted by KMS. `KMSKeyId` also needs to be provided for decryption.
+  * Permission `kms:Decrypt` to access the `KMSKeyId`, will be added automatically.
+* `DdApiKey` (or environment variable `DD_API_KEY`): NOT recommended. The Datadog API Key in plain text.
 
-1. **KMS Encrypted key (recommended)**: Use the `DD_KMS_API_KEY` environment variable to use a KMS encrypted key. Make sure that the Lambda execution role is listed in the KMS Key user in https://console.aws.amazon.com/iam/home#encryptionKeys.
-2. **Environment Variable**: Use the `DD_API_KEY` environment variable for the Lambda function.
-3. **Manual**: Replace `<YOUR_DATADOG_API_KEY>` in the code:
+### Custom Tags
 
-  ```python
-  ## @param DD_API_KEY - String - required - default: none
-  ## The Datadog API key associated with your Datadog Account
-  ## It can be found here:
-  ##
-  ##   * Datadog US Site: https://app.datadoghq.com/account/settings#api
-  ##   * Datadog EU Site: https://app.datadoghq.eu/account/settings#api
-  #
-  DD_API_KEY = "<YOUR_DATADOG_API_KEY>"
-  ```
+Add custom tags to logs forwarded by your function using the `DdTags` setting (or environment variable `DD_TAGS`). It must be a comma-delimited string with no trailing comma, e.g., `env:prod,stack:classic`.
 
-#### Custom Tags
+### Datadog Site
 
-Add custom tags to all data forwarded by your function, either:
+Define your Datadog Site to send data to using the `DdSite` setting (or environment variable `DD_SITE`). It must be either `datadoghq.com` for the Datadog US site or `datadoghq.eu` for the Datadog EU site.
 
-* Use the `DD_TAGS` environment variable. Your tags must be a comma-separated list of strings with no trailing comma.
-* Edit the lambda code directly:
+### Function Name
 
-  ```python
-  ## @param DD_TAGS - list of comma separated strings - optional -default: none
-  ## Pass custom tags as environment variable or through this variable.
-  ## Ensure your tags are a comma separated list of strings with no trailing comma in the envvar!
-  #
-  DD_TAGS = os.environ.get("DD_TAGS", "")
-  ```
+You can optionally customize the Datadog Forwarder Lambda function name using the `FunctionName` setting. DO **NOT** change unless you need to install multiple forwarders in the same region for advanced use cases. If changed, you **MUST** provide the same function name when upgrade in the future.
 
-#### Datadog Site
+### Reserved Concurrency
 
-Define your Datadog Site to send data to, `datadoghq.com` for Datadog US site or `datadoghq.eu` for Datadog EU site, either:
+You can optionally customize the reserved concurrency for the Datadog Forwarder Lambda function using the `ReservedConcurrency` setting.
 
-* Use the `DD_SITE` environment variable.
-* Edit the lambda code directly:
+### Log Retention
 
-  ```python
-  ## @param DD_SITE - String - optional -default: datadoghq.com
-  ## Define the Datadog Site to send your logs and metrics to.
-  ## Set it to `datadoghq.eu` to send your logs and metrics to Datadog EU site.
-  #
-  DD_SITE = os.getenv("DD_SITE", default="datadoghq.com")
-  ```
+You can optionally customize the CloudWatch log retention for logs generated by the Datadog Forwarder Lambda function using the `LogRetentionInDays` setting.
 
-#### Send logs through TCP or HTTP.
+## Advanced Settings
 
-By default, the forwarder sends logs using HTTPS through the port `443`. To send logs over a SSL encrypted TCP connection either:
+The advanced settings can be configured after the installation as environment variables from the Lambda configuration console.
 
-* Set the environment variable `DD_USE_TCP` to `true`.
-* Edit the lambda code directly:
+### Send logs through TCP or HTTP
 
-  ```python
-  ## @param DD_USE_TCP - boolean - optional -default: false
-  ## Change this value to `true` to send your logs and metrics using the HTTP network client
-  ## By default, it use the TCP client.
-  #
-  DD_USE_TCP = os.getenv("DD_USE_TCP", default="false").lower() == "true"
-  ```
+By default, the forwarder sends logs using HTTPS through the port `443`. To send logs over a SSL encrypted TCP connection, set the environment variable `DD_USE_TCP` to `true`.
 
-#### Proxy
+### Proxy
 
 Ensure that you disable SSL between the lambda and your proxy by setting `DD_NO_SSL` to `true`
  
@@ -132,25 +75,11 @@ Two environment variables can be used to forward logs through a proxy:
 * `DD_URL`: Define the proxy endpoint to forward the logs to.
 * `DD_PORT`: Define the proxy port to forward the logs to.
 
-#### DD_FETCH_LAMBDA_TAGS
+### Fetch Lambda Tags (Beta)
 
 If the `DD_FETCH_LAMBDA_TAGS` env variable is set to `true` then the log forwarder will fetch Lambda tags using [GetResources](https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/API_GetResources.html) API calls and apply them to the `aws.lambda.enhanced.*` metrics parsed from the REPORT log. For this to work the log forwarder function needs to be given the `tag:GetResources` permission. The tags are cached in memory so that they'll only be fetched when the function cold starts or when the TTL (1 hour) expires. The log forwarder increments the `aws.lambda.enhanced.get_resources_api_calls` metric for each API call made.
 
-### 5. Configure your function
-
-To configure your function:
-
-1. Set the memory to 1024 MB.
-2. Also set the timeout limit. 120 seconds is recommended to deal with big files.
-3. Hit the `Save` button.
-
-### 6. Test it
-
-Hit the `Test` button, and select `CloudWatch Logs` as the sample event. If the test "succeeded", you are all set! The test log doesn't show up in the platform.
-
-**Note**: For S3 logs, there may be some latency between the time a first S3 log file is posted and the Lambda function wakes up.
-
-### 7. (optional) Scrubbing / Redaction rules
+### Scrubbing / Redaction rules
 
 Multiple scrubbing options are available.  `REDACT_IP` and `REDACT_EMAIL` match against hard-coded patterns, while `DD_SCRUBBING_RULE` allows users to supply a regular expression.
 - To use `REDACT_IP`, add it as an environment variable and set the value to `true`.
@@ -163,7 +92,7 @@ Multiple scrubbing options are available.  `REDACT_IP` and `REDACT_EMAIL` match
 - Scrubbing rules are applied to the full JSON-formatted log, including any metadata that is automatically added by the Lambda function.
 - Each instance of a pattern match is replaced until no more matches are found in each log.
 
-### 8. (optional) Filtering rules
+### Filtering rules
 
 Use the `EXCLUDE_AT_MATCH` OR `INCLUDE_AT_MATCH` environment variables to filter logs based on a regular expression match:
 
@@ -172,17 +101,52 @@ Use the `EXCLUDE_AT_MATCH` OR `INCLUDE_AT_MATCH` environment variables to filter
 - If a log matches both the inclusion and exclusion criteria, it is excluded.
 - Filtering rules are applied to the full JSON-formatted log, including any metadata that is automatically added by the function.
 
-### 9. (optional) Multiline Log support for s3
+### Multiline Log support for s3
 
 If there are multiline logs in s3, set `DD_MULTILINE_LOG_REGEX_PATTERN` environment variable to the specified regex pattern to detect for a new log line.
 
 - Example: for multiline logs beginning with pattern `11/10/2014`: `DD_MULTILINE_LOG_REGEX_PATTERN="\d{2}\/\d{2}\/\d{4}"`
 
-### 10. (optional) Disable log forwarding
+### Disable log forwarding
 
 The datadog forwarder **ALWAYS** forwards logs by default. If you do NOT use the Datadog log management product, you **MUST** set environment variable `DD_FORWARD_LOG` to `False`, to avoid sending logs to Datadog. The forwarder will then only forward other observability data, such as metrics.
 
-### 11. (optional) Disable SSL validation
+### Disable SSL validation
 
 If you need to ignore SSL certificate validation when forwarding logs using HTTPS, you can set the environment variable `DD_SKIP_SSL_VALIDATION` to `True`.
 This will still encrypt the traffic between the forwarder and the endpoint provided with `DD_URL` but will not check if the destination SSL certificate is valid. 
+
+## Test
+
+Hit the `Test` button, and select `CloudWatch Logs` as the sample event. If the test "succeeded", you are all set! The test log doesn't show up in the platform.
+
+## Triggers
+
+Follow the steps below to set up triggers on the Datadog Forwarder.
+
+1. Click "Publish new version" from the "Actions" menu. Put the Datadog Forwarder version (e.g., 3.0.1) in the description for reference.
+1. Click "Create alias" from the "Actions" menu, name it `live` and point it to the version that was just published.
+1. Now you can set up triggers against the Datadog Forwarder `live` alias (e.g., the ARN should be `arn:aws:lambda:us-east-1:xxxx:function:serverlessrepo-Datadog-Forwarder-xxxx:live`), instead of the unqualified version. This ensures a safe and gradual upgrade in the future. Learn more about [AWS Lambda alias](https://docs.aws.amazon.com/lambda/latest/dg/configuration-aliases.html).
+1. You can either manually subscribe the Datadog Forwarder Lambda function to the desired CloudWatch Log groups or S3 buckets, or let Datadog [manage the triggers automatically for you](https://docs.datadoghq.com/integrations/amazon_web_services/?tab=allpermissions#automatically-setup-triggers).
+
+## Upgrade
+
+Follow the steps below to upgrade an existing installation of the Datadog Forwarder.
+
+1. Make sure the triggers (S3 and CloudWatch Log events) are against the Datadog Forwarder lambda function alias, rather than the unqualified version. If not, follow the steps in the [Triggers section](#triggers) to create an alias and update the existing triggers to be against the alias. This ensures a [gradual upgrade](https://docs.aws.amazon.com/lambda/latest/dg/configuration-aliases.html) next.
+1. Open a new tab and navigate to the [Datadog Forwarder SAM application](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:464622532012:applications~Datadog-Log-Forwarder) and "Deploy".
+1. Configure the application settings.
+   1. The `Application name` **MUST** be the same as the existing Datadog Forwarder application, in order to *update* the CloudFormation stack behind the scene. Please ignore the SAM prefix `serverlessrepo-`, e.g., if your existing forwarder's application name is `serverlessrepo-DatadogForwarder`, enter `DatadogForwarder` for application name.
+   1. The `FunctionName` **MUST** be the same as the existing Datadog Forwarder Lambda function, which defaults to `DatadogForwarder`.
+   1. Configure the rest of the application settings based the settings (environment variables) of the existing Datadog Forwarder.
+1. Click "Deploy", and the unqualified version (i.e. $LATEST) of `DatadogForwarder` will be updated in a few minutes. This shouldn't affect anything until you point the `live` alias to the new version of the Lambda.
+1. Copy over any missing environment variables for advanced settings from the `live` forwarder to the new (unqualified) version.
+1. If you have directly customized the source code of the forwarder perviously, you need to copy over the changes manually to the unqualified version.
+1. Click "Publish new version" from the "Actions" menu. Put the Datadog Forwarder version (e.g., 3.0.1) in the description for reference.
+1. Switch to the `live` alias, and set the newly published version as the additional version.
+1. Adjust the weight gradually to shift traffic from the old version to the new.
+1. If the new version works fine, point the `live` alias to the new version completely. Otherwise, point the `live` alias back to the old version, and report the issue to Datadog support.
+
+## Notes
+
+* For S3 logs, there may be some latency between the time a first S3 log file is posted and the Lambda function wakes up.
diff --git a/aws/logs_monitoring/lambda_function.py b/aws/logs_monitoring/lambda_function.py
@@ -1,8 +1,3 @@
-# IMPORTANT NOTE: When upgrading, please ensure your forwarder Lambda function
-# has the latest Datadog Lambda Layer installed.
-# https://github.com/DataDog/datadog-serverless-functions/tree/master/aws/logs_monitoring#3-add-the-datadog-lambda-layer
-
-
 # Unless explicitly stated otherwise all files in this repository are licensed
 # under the Apache License Version 2.0.
 # This product includes software developed at Datadog (https://www.datadoghq.com/).
@@ -194,7 +189,12 @@ def compileRegex(rule, pattern):
 EXCLUDE_AT_MATCH = os.getenv("EXCLUDE_AT_MATCH", default=None)
 exclude_regex = compileRegex("EXCLUDE_AT_MATCH", EXCLUDE_AT_MATCH)
 
-if "DD_KMS_API_KEY" in os.environ:
+if "DD_API_KEY_SECRET_ARN" in os.environ:
+    SECRET_ARN = os.environ["DD_API_KEY_SECRET_ARN"]
+    DD_API_KEY = boto3.client("secretsmanager").get_secret_value(
+        SecretId=SECRET_ARN
+    )["SecretString"]
+elif "DD_KMS_API_KEY" in os.environ:
     ENCRYPTED = os.environ["DD_KMS_API_KEY"]
     DD_API_KEY = boto3.client("kms").decrypt(
         CiphertextBlob=base64.b64decode(ENCRYPTED)
@@ -255,7 +255,7 @@ def compileRegex(rule, pattern):
 DD_CUSTOM_TAGS = "ddtags"
 DD_SERVICE = "service"
 DD_HOST = "host"
-DD_FORWARDER_VERSION = "2.3.4"
+DD_FORWARDER_VERSION = os.environ.get("DD_FORWARDER_VERSION", "unknown")
 
 class RetriableException(Exception):
     pass
diff --git a/aws/logs_monitoring/log-sam-template.yaml b/aws/logs_monitoring/log-sam-template.yaml
diff --git a/aws/logs_monitoring/release.sh b/aws/logs_monitoring/release.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+
+set -e
+
+# Determine the S3 bucket to publish the template
+if [ -z "$1" ]; then
+    echo "Must specify a S3 bucket to publish the template"
+    exit 1
+else
+    BUCKET=$1
+fi
+
+VERSION=$(grep -o 'Version: \d\.\d\.\d' template.yaml | cut -d' ' -f2)
+
+# Validate the template
+echo "Validating template.yaml"
+aws cloudformation validate-template --template-body file://template.yaml
+
+# Confirm to proceed
+read -p "About to create a Github release aws-dd-forwarder-${VERSION} and upload the template.yaml to s3://${BUCKET}/templates/${VERSION}.yaml. Continue (y/n)?" CONT
+if [ "$CONT" != "y" ]; then
+  echo "Exiting"
+  exit 1
+fi
+
+# Create a github release
+echo "Release aws-dd-forwarder-${VERSION} to github"
+go get github.com/github/hub
+zip -r aws-dd-forwarder-${VERSION}.zip .
+hub release create -a aws-dd-forwarder-${VERSION}.zip -m "aws-dd-forwarder-${VERSION}" aws-dd-forwarder-${VERSION}
+
+# Upload the template to the S3 bucket
+echo "Uploading template.yaml to s3://${BUCKET}/templates/${VERSION}.yaml"
+aws s3 cp template.yaml s3://${BUCKET}/templates/${VERSION}.yaml --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers
+echo "Done uploading the template, and here is the CloudFormation quick launch URL"
+echo "https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=datadog-serverless&templateURL=https://${BUCKET}.s3.amazonaws.com/templates/${VERSION}.yaml"
+
+echo "Done!"
diff --git a/aws/logs_monitoring/template.yaml b/aws/logs_monitoring/template.yaml
