Skip to content

[Migration Assistant] Add troubleshooting section for the replayer #11474

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[Migration Assistant] Add troubleshooting section for the replayer #11474

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion _migration-assistant/migration-phases/replay-captured-traffic.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@

Traffic Replayer sends requests in the same order that they were received from each connection to the source. However, relative timing between different connections is not guaranteed. For example:

- **Scenario**: Two connections exist:one sends a PUT request every minute, and the other sends a GET request every second.
- **Scenario**: Two connections exist: one sends a PUT request every minute, and the other sends a GET request every second.
- **Behavior**: Traffic Replayer will maintain the sequence within each connection, but the relative timing between the connections (PUTs and GETs) is not preserved.

Assume that a source cluster responds to requests (GETs and PUTs) within 100 ms:
Expand Down Expand Up @@ -179,6 +179,33 @@
These logs contain the contents of all requests, including authorization headers and the contents of all HTTP messages. Ensure that access to the migration environment is restricted, as these logs serve as a source of truth for determining what happened in both the source and target clusters. Response times for the source refer to the amount of time between the proxy sending the end of a request and receiving the response. While response times for the target are recorded in the same manner, keep in mind that the locations of the capture proxy, Traffic Replayer, and target may differ and that these logs do not account for the client's location.
{: .note}

## Troubleshooting

### Health check requests failing with 403s then passing with 200s

Check warning on line 184 in _migration-assistant/migration-phases/replay-captured-traffic.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.UnitsSpacing] Put a space between the number and the units in '403s '.

Raw Output:
{"message": "[OpenSearch.UnitsSpacing] Put a space between the number and the units in '403s '.", "location": {"path": "_migration-assistant/migration-phases/replay-captured-traffic.md", "range": {"start": {"line": 184, "column": 40}}}, "severity": "WARNING"}

Check failure on line 184 in _migration-assistant/migration-phases/replay-captured-traffic.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.StackedHeadings] Do not stack headings. Insert an introductory sentence between headings.

Raw Output:
{"message": "[OpenSearch.StackedHeadings] Do not stack headings. Insert an introductory sentence between headings.", "location": {"path": "_migration-assistant/migration-phases/replay-captured-traffic.md", "range": {"start": {"line": 184, "column": 1}}}, "severity": "ERROR"}

If you're seeing source requests that fail with 403 errors but then pass with 200 responses, you can configure the capture proxy to ignore these requests. These are typically health check requests that may fail authentication on the source cluster but succeed on the target cluster. You can ignore these requests by adding the following to the captureProxyExtraArgs configuration option and redeploying Migration Assistant:

```bash
--suppressCaptureForHeaderMatch user-agent .*ELB-HealthChecker.*
```

### No replayed traffic

If your dashboard shows a high 'time to catchup' metric (for example, more than 24 hours), this indicates that Traffic Replayer has large gaps of no traffic that will still be visible during playback. The speedup factor will only increase wall clock time by that factor, so even large gaps in traffic are preserved during replay.

Check failure on line 194 in _migration-assistant/migration-phases/replay-captured-traffic.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: catchup. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: catchup. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migration-assistant/migration-phases/replay-captured-traffic.md", "range": {"start": {"line": 194, "column": 41}}}, "severity": "ERROR"}

To resolve this issue, clear the captured traffic and restart:

```bash
# Stop the capture proxy
console replay stop

# Clear Kafka topics (this removes all captured data)
console kafka delete-topic

# Restart traffic capture
console replay start
```


### Example log entry

Expand Down
Loading