Auto update endpoint URL on redirect

[ericallam]

Closes #786

✅ Checklist

• I have followed every step in the contributing guide
• The PR title follows the convention.
• I ran and tested the code works

---

Testing

Tested against a URL redirecting using a 308 on Vercel (root -> www) and made sure the endpoint URL was updated and the indexing completed successfully.

---

Changelog

We will now detect redirects when doing an endpoint indexing request and update the endpoint URL to "follow" the redirect, and then retry the indexing. If that happens more than 5 times, we will abort. If the response is a redirect with no location URL, we will abort.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 6d1ddb2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[nicktrn]

Nice and simple 👍