Skip to content

Malformed _X_AMZN_TRACE_ID value #89

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
robturner-kaluza opened this issue Mar 14, 2023 · 8 comments · Fixed by #90
Closed

Malformed _X_AMZN_TRACE_ID value #89

robturner-kaluza opened this issue Mar 14, 2023 · 8 comments · Fixed by #90

Comments

@robturner-kaluza
Copy link

Overnight we've suddently started to see Malformed _X_AMZN_TRACE_ID errors.

As an example the value we're seeing is: Root=1-46105bdf-04c13a9504458ebc539f5fba;Parent=240a548a42a88af4;Sampled=0;Lineage=12326a9d:0

Looks like lineage is valid (https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-tracingheader) however this library is throwing an error when it splits the traceId by ;. See here:

Not sure I can prevent the lineage property appearing in traceIds but should this library handle this case?

@DReigada
Copy link

We're also experiencing this.

Seems like AWS added a new field overnight to the tracing environment variable.
It is a valid value for the variable though, according to their docs

@astuyve
Copy link
Contributor

astuyve commented Mar 15, 2023

Hi, thanks for reaching out!

It does look like this is a change to AWS X-Ray.

That said, this library is now deprecated, and no longer needed.

There is an upgrade guide available in the documentation.

Please let me know if you have issues upgrading.

Thanks!

@GitMarco
Copy link

GitMarco commented Mar 15, 2023

I have a fix, sadly issues pushing a branch gives me a 403, tried many things. And since there's an upgrade possible, I'll just post the .patch file here, should anybody need it. This code requires at least 3 parts in the trace, any higher number is allowed too.

------- Patch starts below this line ------------------------------



Index: build.gradle
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/build.gradle b/build.gradle
--- a/build.gradle	(revision d8ef9596fe8430acd6117f640a4a40addc1fc3f1)
+++ b/build.gradle	(revision 5f2304c164ab7571e211352d90670cbe756e3ec2)
@@ -54,7 +54,7 @@
 targetCompatibility = 1.8
 
 group = 'com.datadoghq'
-version= '1.4.5'
+version= '1.4.10'
 archivesBaseName = "datadog-lambda-java"
 description = "datadog-lambda-java"
 
Index: src/main/java/com/datadoghq/datadog_lambda_java/Tracing.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/main/java/com/datadoghq/datadog_lambda_java/Tracing.java b/src/main/java/com/datadoghq/datadog_lambda_java/Tracing.java
--- a/src/main/java/com/datadoghq/datadog_lambda_java/Tracing.java	(revision d8ef9596fe8430acd6117f640a4a40addc1fc3f1)
+++ b/src/main/java/com/datadoghq/datadog_lambda_java/Tracing.java	(revision 5f2304c164ab7571e211352d90670cbe756e3ec2)
@@ -366,14 +366,15 @@
     String parentId;
 
     public XRayTraceContext(){
-        //Root=1-5e41a79d-e6a0db584029dba86a594b7e;Parent=8c34f5ad8f92d510;Sampled=1
+        // Example: Root=1-5e41a79d-e6a0db584029dba86a594b7e;Parent=8c34f5ad8f92d510;Sampled=1
+        // Example: Root=1-5e41a79d-e6a0db584029dba86a594b7e;Parent=8c34f5ad8f92d510;Sampled=0;Lineage=f627d631:0
         String traceId = System.getenv("_X_AMZN_TRACE_ID");
         if (traceId == null || traceId.equals("")){
             DDLogger.getLoggerImpl().debug("Unable to find _X_AMZN_TRACE_ID");
             return;
         }
         String[] traceParts = traceId.split(";");
-        if(traceParts.length != 3){
+        if(hasValidNumberOfParts(traceParts)){
             DDLogger.getLoggerImpl().error ("Malformed _X_AMZN_TRACE_ID value: "+ traceId);
             return;
         }
@@ -388,13 +389,17 @@
         this.traceIdHeader = traceId;
     }
 
+    private static boolean hasValidNumberOfParts(String[] traceParts) {
+        return traceParts.length < 3;
+    }
+
     /**
      * Test constructor that can take a dummy _X_AMZN_TRACE_ID value rather than reading from env vars
      * @param traceId
      */
     protected XRayTraceContext(String traceId){
         String[] traceParts = traceId.split(";");
-        if(traceParts.length != 3){
+        if(hasValidNumberOfParts(traceParts)){
             DDLogger.getLoggerImpl().error("Malformed _X_AMZN_TRACE_ID value: "+ traceId);
             return;
         }
Index: src/test/java/com/datadoghq/datadog_lambda_java/TracingTest.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/test/java/com/datadoghq/datadog_lambda_java/TracingTest.java b/src/test/java/com/datadoghq/datadog_lambda_java/TracingTest.java
--- a/src/test/java/com/datadoghq/datadog_lambda_java/TracingTest.java	(revision d8ef9596fe8430acd6117f640a4a40addc1fc3f1)
+++ b/src/test/java/com/datadoghq/datadog_lambda_java/TracingTest.java	(revision 5f2304c164ab7571e211352d90670cbe756e3ec2)
@@ -4,11 +4,15 @@
 import com.google.gson.reflect.TypeToken;
 import java.lang.reflect.Type;
 import java.util.HashMap;
-import org.junit.Assert;
+
 import org.junit.Test;
 
 import java.util.Map;
 
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+
 public class TracingTest {
 
     @Test
@@ -29,10 +33,10 @@
 
         Map<String, String> headers = t.makeOutboundHttpTraceHeaders();
 
-        Assert.assertEquals(3, headers.size());
-        Assert.assertEquals("2", headers.get(cxt.ddSamplingKey));
-        Assert.assertEquals("12344567890", headers.get(cxt.ddTraceKey));
-        Assert.assertEquals("6023947403358210776", headers.get(cxt.ddParentKey));
+        assertEquals(3, headers.size());
+        assertEquals("2", headers.get(cxt.ddSamplingKey));
+        assertEquals("12344567890", headers.get(cxt.ddTraceKey));
+        assertEquals("6023947403358210776", headers.get(cxt.ddParentKey));
     }
 
     @Test
@@ -53,7 +57,7 @@
 
         Map<String, String> headers = t.makeOutboundHttpTraceHeaders();
 
-        Assert.assertEquals(0, headers.size());
+        assertEquals(0, headers.size());
     }
 
     @Test
@@ -83,6 +87,24 @@
         expectedTracingInfo.put(cxt.ddParentKey, "6023947403358210776");
         expectedTracingInfo.put(cxt.ddSamplingKey, cxt.samplingPriority);
 
-        Assert.assertEquals(expectedTracingInfo, tracingInfo);
+        assertEquals(expectedTracingInfo, tracingInfo);
+    }
+
+    @Test
+    public void shouldSupportAtLeastThreePartsIn_X_AMZN_TRACE_ID() {
+        XRayTraceContext xRayTraceContextBefore2023March14 = new XRayTraceContext("Root=1-5e41a79d-e6a0db584029dba86a594b7e;Parent=8c34f5ad8f92d510;Sampled=1");
+        assertEquals("1-5e41a79d-e6a0db584029dba86a594b7e", xRayTraceContextBefore2023March14.getTraceId());
+        assertEquals("8c34f5ad8f92d510", xRayTraceContextBefore2023March14.getParentId());
+
+        XRayTraceContext xRayTraceContextSince2023March14 = new XRayTraceContext("Root=1-5e41a79d-e6a0db584029dba86a594b7e;Parent=8c34f5ad8f92d510;Sampled=0;Lineage=f627d631:0");
+        assertEquals("1-5e41a79d-e6a0db584029dba86a594b7e", xRayTraceContextSince2023March14.getTraceId());
+        assertEquals("8c34f5ad8f92d510", xRayTraceContextSince2023March14.getParentId());
+    }
+
+    @Test
+    public void shouldIgnoreLessThanThreePartsIn_X_AMZN_TRACE_ID() {
+        XRayTraceContext xRayTraceContextLessThanThreeParts = new XRayTraceContext("Root=1-5e41a79d-e6a0db584029dba86a594b7e;Parent=8c34f5ad8f92d510");
+        assertNull(xRayTraceContextLessThanThreeParts.getTraceId());
+        assertNull(xRayTraceContextLessThanThreeParts.getParentId());
     }
 }

@dataGeeek
Copy link

Hi astuyve,

we have currently a setup that prevents us from migrating to the new lambda extension solution.

  • We have to use a custom build image, since the bundle size is too big. Therefore, we cannot use the lambda layers
  • We use jib to build the docker image, which does not support the COPY --from=public.ecr.aws/datadog/lambda-extension:<TAG> /opt/. /opt/ command.

We therefore like to know, if there is an alternative way to obtain the files we need to install to /opts/? E.g. by downloading a .tar file?

@astuyve
Copy link
Contributor

astuyve commented Mar 17, 2023

Hi @dataGeeek!

Thanks for the question. For container-image based services, we have a different documentation page which should handle your use case.

Thanks!

@dataGeeek
Copy link

Hi @astuyve,

unfortunately the proposed solution for container-image based services does not work out-of-the-box for us, because jib does not support the COPY --from=public.ecr.aws/datadog/lambda-extension:<TAG> /opt/. /opt/ command.

Therefore I'm asking if there is a nother way to fetch the necessary files and place them in the docker container?

Thanks for your support!

@astuyve
Copy link
Contributor

astuyve commented Mar 21, 2023

Given a user cannot upgrade, we've decided it's best to backport a fix to this library.

I'll update this ticket when we perform the release, thanks everyone!

@astuyve
Copy link
Contributor

astuyve commented Mar 23, 2023

Hi folks,

This is fix is released in 1.4.10.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants