Add docs.rs integration

[kureuil]

When a crate does not specify a documentation link, we query docs.rs to know
if the documentation for the requested version was successfully built. If
if is, we set the crate's documentation link to the corresponding docs.rs
page, if not nothing changes.

Fixes #419.

[carols10cents]

I am so excited about this!!! I've been waiting for someone to take care of this ❤️ ❤️ ❤️ I'm sorry for letting this sit for a few days!!!

So I tried this out locally against production's API by running yarn run start:live, and the following things work great!!

• I don't see an ajax request to docs.rs if a crate has a documentation link specified!!!
• I see an ajax request and then don't see a documentation link for crates that docs.rs wasn't able to build docs for, like http://localhost:4200/crates/a !!!
• I see a Documentation link appear for a crate without one specified when I'm on a crate page like http://localhost:4200/crates/abra !!!
• I see a Documentation link appear for a crate without one specified when I'm on a crate version page like http://localhost:4200/crates/sassers/0.13.4-maybe-not-broken !!!!

I think there's still a problem with crates that only have unstable versions on their crate page-- sassers is one of those, and I'm seeing the version number be undefined in the docs.rs request. I don't think that's a problem with your code though, I think that's basically a new symptom of #654/#510 :( So don't worry about that, and I'll try to fix that problem soon.

One question I have, though, is whether we should be doing this on crate list pages, like search result pages, /crates, category lists, keyword lists, etc since we have the documentation links there now too. I think it would be useful, and I hope doing a bunch of ajax requests to docs.rs on loads of those pages won't take too long or DDoS docs.rs, but we'll find out?

What do you think about trying to add that too?

[onur]

@carols10cents when I started adding builds.json endpoint to docs.rs we didn't have documentation links in crate list pages. I don't think it's a good idea to send multiple requests to docs.rs in this pages (docs.rs can handle thanks to rust but it will probably take too long for client).

I'll try to add another method to query multiple crates with only one request.

One more thing: please use: https://docs.rs/${crateName}/${crateVersion} link instead of https://docs.rs/${crateName}/${crateVersion}/${crateName}/. Last part of the link might be different than ${crateName}. https://docs.rs/calculator for example, using exp for lib.name and calculator for package.name.

[kureuil]

So, I just pushed some changes to the branch:

• The generated link to docs.rs now omits the lib.name (thanks @onur)
• No more undefined version numbers on a crate page with only unstable versions (I didn't see the query fetching all of the crates' versions was asynchronous)

@carols10cents I also think it would be very useful to have the auto-link on every page where a crate is referenced but I agree with @onur's argument, it might not be the best experience for end users to have up to N requests going to docs.rs for documentation links. Maybe we could fetch the build status in the backend and have the API return the docs.rs link if the crate did not specified a documentation link. This could also allow for more aggressive caching of the documentation link.
BTW, shouldn't the documentation link be stored in the Version model ? Why isn't the crate metadata (except for features) versionned ?

[carols10cents]

Thanks for the updates! Taking a look at the changes now. It might be a bit confusing why a documentation link appears sometimes when you click into a crate's page, but we can add the docs.rs link to lists of crates when either docs.rs has an endpoint to query multiple crates at once or we add caching to crates.io's backend.

BTW, shouldn't the documentation link be stored in the Version model ? Why isn't the crate metadata (except for features) versionned ?

I don't know exactly :) I could see a reason for the documentation links to be associated with a version, but I don't think anyone was hosting anything other than their most recent documentation when crates.io was created. Links like the homepage and repo either won't change or if they change, the old ones will be inaccessible, so there isn't really much reason to store links per version. Same with description, authors, README, LICENSE, etc.

[carols10cents]

Something's still not quite right navigating around versions and getting doc links for versions, and it might be due to our misuse of ember in general, i'm not sure... here's how to reproduce what I'm seeing:

• Run this code against the production api with yarn run start:live
• Visit http://localhost:4200/crates?page=2
• Click on the accelerate-provider crate to end up on http://localhost:4200/crates/accelerate-provider
• The documentation link on the crate page correctly goes to https://docs.rs/accelerate-provider/0.2.1/, 0.2.1 is the latest version
• Click on one of the other versions, like version 0.2.0
• You'll now be on http://localhost:4200/crates/accelerate-provider/0.2.0, but the documentation link on this page still goes to https://docs.rs/accelerate-provider/0.2.1/, using the latest version. I expected the docs.rs link to match the displayed version 0.2.0.
• If you reload http://localhost:4200/crates/accelerate-provider/0.2.0, then the documentation link correctly uses 0.2.0

Let me know if you have any questions or if this doesn't make sense or if you'd like help investigating why this is happening :)

[kureuil]

Good catch on the version change! I'll investigate this next week once I'm back from vacation.

[kureuil]

Finally took time to resolve this last issue. It should now work properly :)

Bonus feature: Any crate that has a docs.rs link as documentation automatically gets versioning on it (i.e: If I navigate to the lazy_static crate page and change version, the documentation is updated for the current version, because the crate specified https://docs.rs/crate/lazy_static as its documentation link)

[carols10cents]

WOOOO!!! This works great!!! Thank you thank you thank you!! ❤️

[kureuil]

I'll take care of updating the cargo docs in the coming days :)

[jplatte]

According to http://doc.crates.io/manifest.html#the-documentation-field-optional this should be in effect, right? I published my first crate today, but after about an hour I still don't see a documentation link on the crates.io page. docs.rs seems to have built the documentation just fine though.

[carols10cents]

@jplatte I'm seeing the documentation link on libproxy's page:

What browser are you using? Can you open developer tools to the network tab, reload the page, and see if there's a request to docs.rs as shown in this screenshot, and if that request succeeded or not?

Are you running any plugins in your browser that might be blocking the request, such as a privacy tool like EFF's privacy badger, an adblocker, anything like that?

[jplatte]

Ohhh, yeah my browser was blocking the request... I forgot that most of the crates.io interface is built on the client side.