refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only log message #996
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…gainst JSON, now only on log Signed-off-by: Vincent Boutour <[email protected]>
ViBiOh
changed the title
… done against JSON, now only on log Signed-off-by: Vincent Boutour <[email protected]>
ge0Aja
reviewed
Sep 26, 2025
| logs_to_forward = filter_logs( | ||
| logs_to_forward, INCLUDE_AT_MATCH, EXCLUDE_AT_MATCH | ||
| ) | ||
| if matcher.match(evaluated_log): |
Contributor
There was a problem hiding this comment.
We should properly catch the exception raised by the matcher here and maybe log a warn?
Contributor
Author
There was a problem hiding this comment.
The previous behavior was to catch the exception and raise a new one. I think it's better to crash in this case. Otherwise Lambda will "succeed" but with bad filtering which is hidding a problem.
ge0Aja
approved these changes
Sep 26, 2025
Contributor
ge0Aja
left a comment
ge0Aja
left a comment
There was a problem hiding this comment.
lgtm, minor comment about exception handling
ViBiOh
added a commit
that referenced
this pull request
Oct 2, 2025
…gainst JSON, now only log message (#996) * refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <[email protected]> * fixup! refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <[email protected]> --------- Signed-off-by: Vincent Boutour <[email protected]>
ViBiOh
added a commit
that referenced
this pull request
Oct 6, 2025
…gainst JSON, now only log message (#996) * refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <[email protected]> * fixup! refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <[email protected]> --------- Signed-off-by: Vincent Boutour <[email protected]>
ViBiOh
added a commit
that referenced
this pull request
Oct 8, 2025
…gainst JSON, now only log message (#996) * refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <[email protected]> * fixup! refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <[email protected]> --------- Signed-off-by: Vincent Boutour <[email protected]>
ViBiOh
added a commit
that referenced
this pull request
Oct 23, 2025
* refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only log message (#996) * refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <[email protected]> * fixup! refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <[email protected]> --------- Signed-off-by: Vincent Boutour <[email protected]> * feat(aws)!: Drop deprecated TCP support (#1000) Signed-off-by: Vincent Boutour <[email protected]> * feat(aws)!: Remove deprecated PrivateLink env variable (#1002) Signed-off-by: Vincent Boutour <[email protected]> * docs(aws): Removing JSON explanation in the include at match documentation (#1007) Signed-off-by: Vincent Boutour <[email protected]> * Add environment variable to support backend storage tag enrichment (#1006) * feat(aws): AWSX-1566 Adding storage tag HTTP header Signed-off-by: Vincent Boutour <[email protected]> * feat(aws): AWSX-1566 Adding new variable in the cloudformation template Signed-off-by: Vincent Boutour <[email protected]> * feat: AWSX-1566 Restoring fetch in the integration test to see metrics Signed-off-by: Vincent Boutour <[email protected]> * fixup! feat: AWSX-1566 Restoring fetch in the integration test to see metrics Signed-off-by: Vincent Boutour <[email protected]> * docs(aws): AWSX-1566 Refine documentation around the new variable Signed-off-by: Vincent Boutour <[email protected]> --------- Signed-off-by: Vincent Boutour <[email protected]> * fix(aws): Fixing discrepancies of the fetch configuration between cloudformation and code Signed-off-by: Vincent Boutour <[email protected]> * feat: Add a changelog for the v5 (#1009) * feat: Add a changelog for the v5 Signed-off-by: Vincent Boutour <[email protected]> * fixup! feat: Add a changelog for the v5 Signed-off-by: Vincent Boutour <[email protected]> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Raphaël Allier <[email protected]> * fixup! Update aws/logs_monitoring/CHANGELOG.md Signed-off-by: Vincent Boutour <[email protected]> --------- Signed-off-by: Vincent Boutour <[email protected]> Co-authored-by: Raphaël Allier <[email protected]> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <[email protected]> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <[email protected]> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <[email protected]> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <[email protected]> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <[email protected]> * docs: Updating changelog from comments Signed-off-by: Vincent Boutour <[email protected]> * fixup! docs: Updating changelog from comments Signed-off-by: Vincent Boutour <[email protected]> * Add reference to the changelog in README (for public doc) (#1011) * feat: Add link to the changelog in the read toe reflect the documentation The readme is feeding this page https://docs.datadoghq.com/logs/guide/forwarder/?tab=cloudformation Signed-off-by: Vincent Boutour <[email protected]> * fixup! feat: Add link to the changelog in the read toe reflect the documentation Signed-off-by: Vincent Boutour <[email protected]> --------- Signed-off-by: Vincent Boutour <[email protected]> * refactor: Improve error handling for exception (#1012) * refactor: Improve error handling for exception Signed-off-by: Vincent Boutour <[email protected]> * fixup! refactor: Improve error handling for exception Signed-off-by: Vincent Boutour <[email protected]> * fixup! refactor: Improve error handling for exception Signed-off-by: Vincent Boutour <[email protected]> --------- Signed-off-by: Vincent Boutour <[email protected]> * Update aws/logs_monitoring/lambda_function.py Co-authored-by: Georgi <[email protected]> * fixup! Update aws/logs_monitoring/lambda_function.py Signed-off-by: Vincent Boutour <[email protected]> --------- Signed-off-by: Vincent Boutour <[email protected]> Co-authored-by: Raphaël Allier <[email protected]> Co-authored-by: Georgi <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fix a behavior of running the
INCLUDE_AT_MATCHandEXCLUDE_AT_MATCHagainst the JSON dump of the log being sent to Datadog instead of the log as received.This is a BREAKING CHANGE if regexes where defined for JSON escaped string.
Previously, filtering a the
statusfield of a JSON log was possible like thisNow, it will be like this
Motivation
The Regex was hard to reason about. Searching for the regex in a log file was working but not in the forwarder, this leaded to some support from customers.
Testing Guidelines
Additional Notes
Types of changes
Check all that apply