Add log stream name prefix filtering, especially for AWS Batch logs #98
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cloudwatchlogs-with-dlq/Readme.md
Outdated
| * SOURCE_NAME_OVERRIDE (OPTIONAL) - Override _sourceName metadata field within SumoLogic. | ||
| * INCLUDE_LOG_INFO (OPTIONAL) - Set it to true when loggroup/logstream values needs to be included in logs. Default is false | ||
| * LOG_FORMAT - Default is Others. One can choose VPC-JSON for VPC flow logs in json format and VPC-RAW for only RAW message line | ||
| * LOG_STREAM_PREFIX (OPTIONAL) - Comma separated list of logStream name prefixes to filter by logStream, expecially for AWS Batch logs |
| "includeSecurityGroupInfo": ("INCLUDE_SECURITY_GROUP_INFO" in env) ? env.INCLUDE_SECURITY_GROUP_INFO === "true" : false, | ||
| // Regex to filter by logStream name prefixes | ||
| "logStreamPrefixRegex": ("LOG_STREAM_PREFIX" in env) | ||
| ? new RegExp('^(' + env.LOG_STREAM_PREFIX.replace(/,/g, '|') + ')', 'i') |
There was a problem hiding this comment.
you might want to escape the values from the CSV, unsure the range of characters that you're allowed to use but any regex modifier that might be in one of the items would corrupt the final regex.
ykitamura-mdsol
changed the title
Contributor
Author
|
@himanshu219, could you please review this PR? |
himanshu219
reviewed
Sep 13, 2019
Contributor
himanshu219
left a comment
himanshu219
left a comment
There was a problem hiding this comment.
LGTM I think we can add this variable in Cloudformation template as well. what do you think?
Contributor
Author
|
Good idea 💡 Added the variable in e64128c |
Contributor
|
Thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
All of AWS Batch logs are sent to one log group,
/aws/batch/job, and the log stream name format isjobDefinitionName/default/ecs_task_id:https://docs.aws.amazon.com/batch/latest/userguide/job_states.html
Changes
In order to stream logs of specific AWS Batch jobs to SumoLogic, this PR is adding log stream name prefix filtering for CloudWatch logs