Optional "rootUrl" option for crawling docs was useful and should be re-added

[TiagoTiago]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

Description

Without it, I can't for example index starting from https://www.gearblocksgame.com/apidoc/annotated.html without requiring a javascript-enabled crawler (actually, I haven't tested it with a crawler; but thru the browser, I can see that the base / URL has no useful links if javascript is disabled); while in older versions of Continue, I could add https://www.gearblocksgame.com/apidoc/ as the rootURL parameter and https://www.gearblocksgame.com/apidoc/annotated.html as the startURL and all links from that "annotated" page that didn't escape to a different site or out of the "/apidoc/" level of the URL got crawled.

To reproduce

No response

Log output

[shssoichiro]

Related issue that I discovered while creating #7007, it seems that when crawling, the actual docs root is not being chosen correctly, which results in entire sites being crawled when they shouldn't be. For example, for these Jest docs, it is crawling all the blog posts which are not part of the documentation path.

I'd expect the ideal behavior should be:

• If a rootUrl is specified, crawl pages that begin with that path
• If a rootUrl is not specified, default to the startUrl minus any trailing "index"/"index.htm"/"index.html"

I can work on this ticket next to resolve both.