feat(replay): Adding OkHttp Request/Response bodies (#4796)

* Add HTTP Request Trigger to sentry-samples-android app

* [replay] Make DefaultReplayBreadcrumbConverter the default BeforeBreadcrumbCallback to extract NetworkRequestData from Hint

-> NetworkRequestData is landing on DefaultReplayBreadcrumbConverter via Hint.get(replay:networkDetails)

* [replay] Add data classes for NetworkDetails

These get set on the breadcrumb Hint, and then hang around in DefaultReplayBreadcrumbConverter until the replay segment has been sent off

See
https://github.com/getsentry/sentry-javascript/blob/632f0b953d99050c11b0edafb9f80b5f3ba88045/packages/replay-internal/src/types/performance.ts#L133-L140
https://github.com/getsentry/sentry-javascript/blob/develop/packages/replay-internal/src/types/request.ts#L12

* [DNL] Force dashboard to show request/response bodies

Manually set networkDetailHasUrls to pass this check
https://github.com/getsentry/sentry-javascript/blob/d1646c8a281dd8795c5a6a3b8e18f2e7069e7fa9/packages/replay-internal/src/util/handleRecordingEmit.ts#L134
https://github.com/getsentry/sentry/blob/master/static/app/views/replays/detail/network/details/getOutputType.tsx#L33
https://github.com/getsentry/sentry/blob/master/static/app/views/replays/detail/network/details/content.tsx#L55-L56

* Move BeforeBreadcrumbCallback initialization to after user config

added some unit tests:
 ./gradlew :sentry-android-replay:testDebugUnitTest --tests="*DefaultReplayBreadcrumbConverterTest*"

* bugfix: Update Breadcrumb #hashcode to be consistent with #equals

Breadcrumb.java has several timestamp fields:
`timestamp: Date`, `timestampMs: Long`, `nanos: Long`

`hashcode` was relying solely on `timestamp`, which can be null depending on which constructor was used.

=> Change to use getTimestamp as 1. this is what equals does (consistency) 2. getTimestamp initialises timestamp if null.

* Initial NetworkDetails extraction logic

Entrypoint is NetworkDetailCaptureUtils (initializeForUrl) called from SentryOkHttpInterceptor
- common logic to handle checking sdk options.
- Accept data from http client via NetworkBodyExtractor, NetworkHeaderExtractor interfaces
that can be reused in future (if needed)

Placeholder impl for req/resp bodies.

From https://docs.sentry.io/platforms/javascript/session-replay/configuration/
- networkDetailAllowUrls, networkDetailDenyUrls,
- networkCaptureBodies
- networkRequestHeaders, networkResponseHeaders

These SDKOptions don't exist yet => impl acts as if they do, but have not been enabled.

* DefaultReplayBreadcrumbConverter properly manages NetworkRequestData entries

Removes entry when creating RRWebSpanEvent
Uses syncrhonized LinkedHashMap with impl to cap size of map (avoid
memory bloat)

* Extract bodies of okhttp requests/responses

Replaces previous placeholder logic.
Now NetworkBodyParser uses io.sentry.JsonObjectReader
to extract body into JSONObject, JSONArray, with fallback to plain-text String (or nothing)

* Check-in sentry-android-replay.api

* Add additional http request types to sentry-samples app for testing

One-shot bodies
Form bodies
plain text bodies
binary bodies

* Add body too large http request types to sentry-samples app

* Properly handle content bodies that are too large

based on cursor[bot] feedback -
https://github.com/getsentry/sentry-java/pull/4796/files#r2466871208

* Disable Network Detail extraction

Initialize the values in FAKE_OPTIONS so that implementation acts as if dev never turned it on.

Leaves in the FAKE_OPTIONS variable to be easy to update reference to real `options` later on.

* Modify ReplayBreadcrumbConverter API to avoid breaking flutter/RN SDKs

use composition via optional constructor on DefaultReplayBreadcrumbConverter
Instead of modifying the parent interface ReplayBreadcrumbConverter to force all subclasses to implement BeforeBreadcrumbCallback (breaking / harder to manage change)

* Move ReplayBreadcrumbConverter initialization after SDKOptions initialization

DefaultReplayBreadcrumbConverter depends on SDKOptions as it now
 needs to delegate to any user-provided BeforeBreadcrumbCallback

* Custom #equals/#hashcode for http breadcrumbs

review thread - https://github.com/getsentry/sentry-java/pull/4796/files#r2461102218

* ./gradlew apiDump

---------

Co-authored-by: Sentry Github Bot <[email protected]>

Author: 43jay

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java

@@ -16,6 +16,7 @@
 import io.sentry.NoOpCompositePerformanceCollector;
 import io.sentry.NoOpConnectionStatusProvider;
 import io.sentry.NoOpContinuousProfiler;
+import io.sentry.NoOpReplayBreadcrumbConverter;
 import io.sentry.NoOpSocketTagger;
 import io.sentry.NoOpTransactionProfiler;
 import io.sentry.NoopVersionDetector;
@@ -253,6 +254,13 @@ static void initializeIntegrationsAndProcessors(
       options.setCompositePerformanceCollector(new DefaultCompositePerformanceCollector(options));
     }
 
+    if (options.getReplayController().getBreadcrumbConverter()
+        instanceof NoOpReplayBreadcrumbConverter) {
+      options
+          .getReplayController()
+          .setBreadcrumbConverter(new DefaultReplayBreadcrumbConverter(options));
+    }
+
     // Check if the profiler was already instantiated in the app start.
     // We use the Android profiler, that uses a global start/stop api, so we need to preserve the
     // state of the profiler, and it's only possible retaining the instance.
@@ -406,7 +414,6 @@ static void installDefaultIntegrations(
     if (isReplayAvailable) {
       final ReplayIntegration replay =
           new ReplayIntegration(context, CurrentDateProvider.getInstance());
-      replay.setBreadcrumbConverter(new DefaultReplayBreadcrumbConverter());
       options.addIntegration(replay);
       options.setReplayController(replay);
     }

sentry-android-replay/api/sentry-android-replay.api

@@ -9,6 +9,7 @@ public final class io/sentry/android/replay/BuildConfig {
 public class io/sentry/android/replay/DefaultReplayBreadcrumbConverter : io/sentry/ReplayBreadcrumbConverter {
 	public static final field $stable I
 	public fun <init> ()V
+	public fun <init> (Lio/sentry/SentryOptions;)V
 	public fun convert (Lio/sentry/Breadcrumb;)Lio/sentry/rrweb/RRWebEvent;
 }
 

sentry-android-replay/src/main/java/io/sentry/android/replay/DefaultReplayBreadcrumbConverter.kt

@@ -1,17 +1,33 @@
 package io.sentry.android.replay
 
 import io.sentry.Breadcrumb
+import io.sentry.Hint
 import io.sentry.ReplayBreadcrumbConverter
 import io.sentry.SentryLevel
+import io.sentry.SentryOptions
+import io.sentry.SentryOptions.BeforeBreadcrumbCallback
 import io.sentry.SpanDataConvention
+import io.sentry.TypeCheckHint.SENTRY_REPLAY_NETWORK_DETAILS
 import io.sentry.rrweb.RRWebBreadcrumbEvent
 import io.sentry.rrweb.RRWebEvent
 import io.sentry.rrweb.RRWebSpanEvent
+import io.sentry.util.network.NetworkRequestData
+import java.util.Collections
 import kotlin.LazyThreadSafetyMode.NONE
 
-public open class DefaultReplayBreadcrumbConverter : ReplayBreadcrumbConverter {
+public open class DefaultReplayBreadcrumbConverter() : ReplayBreadcrumbConverter {
+  private var options: SentryOptions? = null
+
+  public constructor(options: SentryOptions) : this() {
+    // We modify options, so keep it around to make that explicit.
+    this.options = options
+    this.options?.beforeBreadcrumb = ReplayBeforeBreadcrumbCallback(options.beforeBreadcrumb)
+  }
+
   internal companion object {
+    private const val MAX_HTTP_NETWORK_DETAILS = 32
     private val snakecasePattern by lazy(NONE) { "_[a-z]".toRegex() }
+
     private val supportedNetworkData =
       HashSet<String>().apply {
         add("status_code")
@@ -23,16 +39,68 @@ public open class DefaultReplayBreadcrumbConverter : ReplayBreadcrumbConverter {
       }
   }
 
+  /**
+   * Intercept the breadcrumb to process any Network Details data on the hint. Delegate to any
+   * user-provided callback to provide the actual breadcrumb to process.
+   */
+  private inner class ReplayBeforeBreadcrumbCallback(
+    private val delegate: BeforeBreadcrumbCallback?
+  ) : BeforeBreadcrumbCallback {
+    override fun execute(breadcrumb: Breadcrumb, hint: Hint): Breadcrumb? {
+      val resultBreadcrumb =
+        if (delegate != null) {
+          delegate.execute(breadcrumb, hint)
+        } else {
+          breadcrumb
+        }
+
+      resultBreadcrumb?.let { finalBreadcrumb ->
+        extractNetworkRequestDataFromHint(finalBreadcrumb, hint)?.let { networkData ->
+          httpNetworkDetails[finalBreadcrumb] = networkData
+        }
+      }
+
+      return resultBreadcrumb
+    }
+
+    private fun extractNetworkRequestDataFromHint(
+      breadcrumb: Breadcrumb,
+      breadcrumbHint: Hint,
+    ): NetworkRequestData? {
+      if (breadcrumb.type != "http" && breadcrumb.category != "http") {
+        return null
+      }
+
+      return breadcrumbHint.get(SENTRY_REPLAY_NETWORK_DETAILS) as? NetworkRequestData
+    }
+  }
+
   private var lastConnectivityState: String? = null
 
+  private val httpNetworkDetails =
+    Collections.synchronizedMap(
+      object : LinkedHashMap<Breadcrumb, NetworkRequestData>() {
+        override fun removeEldestEntry(
+          eldest: MutableMap.MutableEntry<Breadcrumb, NetworkRequestData>?
+        ): Boolean {
+          return size > MAX_HTTP_NETWORK_DETAILS
+        }
+      }
+    )
+
   override fun convert(breadcrumb: Breadcrumb): RRWebEvent? {
     var breadcrumbMessage: String? = null
-    var breadcrumbCategory: String? = null
+    val breadcrumbCategory: String?
     var breadcrumbLevel: SentryLevel? = null
     val breadcrumbData = mutableMapOf<String, Any?>()
+
     when {
       breadcrumb.category == "http" -> {
-        return if (breadcrumb.isValidForRRWebSpan()) breadcrumb.toRRWebSpanEvent() else null
+        return if (breadcrumb.isValidForRRWebSpan()) {
+          breadcrumb.toRRWebSpanEvent()
+        } else {
+          null
+        }
       }
 
       breadcrumb.type == "navigation" && breadcrumb.category == "app.lifecycle" -> {
@@ -42,6 +110,7 @@ public open class DefaultReplayBreadcrumbConverter : ReplayBreadcrumbConverter {
       breadcrumb.type == "navigation" && breadcrumb.category == "device.orientation" -> {
         breadcrumbCategory = breadcrumb.category!!
         val position = breadcrumb.data["position"]
+
         if (position == "landscape" || position == "portrait") {
           breadcrumbData["position"] = position
         } else {
@@ -53,8 +122,9 @@ public open class DefaultReplayBreadcrumbConverter : ReplayBreadcrumbConverter {
         breadcrumbCategory = "navigation"
         breadcrumbData["to"] =
           when {
-            breadcrumb.data["state"] == "resumed" ->
+            breadcrumb.data["state"] == "resumed" -> {
               (breadcrumb.data["screen"] as? String)?.substringAfterLast('.')
+            }
             "to" in breadcrumb.data -> breadcrumb.data["to"] as? String
             else -> null
           } ?: return null
@@ -67,6 +137,7 @@ public open class DefaultReplayBreadcrumbConverter : ReplayBreadcrumbConverter {
             ?: breadcrumb.data["view.tag"]
             ?: breadcrumb.data["view.class"])
             as? String ?: return null
+
         breadcrumbData.putAll(breadcrumb.data)
       }
 
@@ -75,18 +146,18 @@ public open class DefaultReplayBreadcrumbConverter : ReplayBreadcrumbConverter {
         breadcrumbData["state"] =
           when {
             breadcrumb.data["action"] == "NETWORK_LOST" -> "offline"
-            "network_type" in breadcrumb.data ->
+            "network_type" in breadcrumb.data -> {
               if (!(breadcrumb.data["network_type"] as? String).isNullOrEmpty()) {
                 breadcrumb.data["network_type"]
               } else {
                 return null
               }
-
+            }
             else -> return null
           }
 
         if (lastConnectivityState == breadcrumbData["state"]) {
-          // debounce same state
+          // Debounce same state
           return null
         }
 
@@ -105,6 +176,7 @@ public open class DefaultReplayBreadcrumbConverter : ReplayBreadcrumbConverter {
         breadcrumbData.putAll(breadcrumb.data)
       }
     }
+
     return if (!breadcrumbCategory.isNullOrEmpty()) {
       RRWebBreadcrumbEvent().apply {
         timestamp = breadcrumb.timestamp.time
@@ -120,29 +192,34 @@ public open class DefaultReplayBreadcrumbConverter : ReplayBreadcrumbConverter {
     }
   }
 
-  private fun Breadcrumb.isValidForRRWebSpan(): Boolean =
-    !(data["url"] as? String).isNullOrEmpty() &&
+  private fun Breadcrumb.isValidForRRWebSpan(): Boolean {
+    return !(data["url"] as? String).isNullOrEmpty() &&
       SpanDataConvention.HTTP_START_TIMESTAMP in data &&
       SpanDataConvention.HTTP_END_TIMESTAMP in data
+  }
 
-  private fun String.snakeToCamelCase(): String =
-    replace(snakecasePattern) { it.value.last().toString().uppercase() }
+  private fun String.snakeToCamelCase(): String {
+    return replace(snakecasePattern) { it.value.last().toString().uppercase() }
+  }
 
   private fun Breadcrumb.toRRWebSpanEvent(): RRWebSpanEvent {
     val breadcrumb = this
     val httpStartTimestamp = breadcrumb.data[SpanDataConvention.HTTP_START_TIMESTAMP]
     val httpEndTimestamp = breadcrumb.data[SpanDataConvention.HTTP_END_TIMESTAMP]
+
     return RRWebSpanEvent().apply {
       timestamp = breadcrumb.timestamp.time
       op = "resource.http"
       description = breadcrumb.data["url"] as String
-      // can be double if it was serialized to disk
+
+      // Can be double if it was serialized to disk
       startTimestamp =
         if (httpStartTimestamp is Double) {
           httpStartTimestamp / 1000.0
         } else {
           (httpStartTimestamp as Long) / 1000.0
         }
+
       endTimestamp =
         if (httpEndTimestamp is Double) {
           httpEndTimestamp / 1000.0
@@ -151,13 +228,54 @@ public open class DefaultReplayBreadcrumbConverter : ReplayBreadcrumbConverter {
         }
 
       val breadcrumbData = mutableMapOf<String, Any?>()
+
+      val networkDetailData = httpNetworkDetails.remove(breadcrumb)
+
+      // Add Network Details data when available
+      networkDetailData?.let { networkData ->
+        networkData.method?.let { breadcrumbData["method"] = it }
+        networkData.statusCode?.let { breadcrumbData["statusCode"] = it }
+        networkData.requestBodySize?.let { breadcrumbData["requestBodySize"] = it }
+        networkData.responseBodySize?.let { breadcrumbData["responseBodySize"] = it }
+
+        networkData.request?.let { request ->
+          val requestData = mutableMapOf<String, Any?>()
+          request.size?.let { requestData["size"] = it }
+          request.body?.let { requestData["body"] = it.value }
+
+          if (request.headers.isNotEmpty()) {
+            requestData["headers"] = request.headers
+          }
+
+          if (requestData.isNotEmpty()) {
+            breadcrumbData["request"] = requestData
+          }
+        }
+
+        networkData.response?.let { response ->
+          val responseData = mutableMapOf<String, Any?>()
+          response.size?.let { responseData["size"] = it }
+          response.body?.let { responseData["body"] = it.value }
+
+          if (response.headers.isNotEmpty()) {
+            responseData["headers"] = response.headers
+          }
+
+          if (responseData.isNotEmpty()) {
+            breadcrumbData["response"] = responseData
+          }
+        }
+      }
+
+      // Original breadcrumb http data
       for ((key, value) in breadcrumb.data) {
         if (key in supportedNetworkData) {
-          breadcrumbData[
-            key.replace("content_length", "body_size").substringAfter(".").snakeToCamelCase(),
-          ] = value
+          val formattedKey =
+            key.replace("content_length", "body_size").substringAfter(".").snakeToCamelCase()
+          breadcrumbData[formattedKey] = value
         }
       }
+
       data = breadcrumbData
     }
   }