Add doc coverage information

[GuillaumeGomez]

This PR adds the usage of the --show-coverage option of rustdoc. It currently looks like this:

A few things I think remains to be done (to be debated for some of them I guess?):

• Add tests
• Add this score in crates search results?
• Add color code (like if it's 100% it's green and the closer to 0, the whiter?)

[GuillaumeGomez]

Not sure to understand the issue with clippy here:

error: in an `if` condition, avoid complex blocks or closures with blocks; instead, move the block or closure higher and bind it with a `let`
##[error]   --> src/docbuilder/rustwide_builder.rs:550:43
    |
550 |               .process_lines(&mut |line, _| {
    |  ___________________________________________^
551 | |                 if line.starts_with('{') && line.ends_with('}') {
552 | |                     json = line.to_owned();
553 | |                 }
554 | |             })
    | |_____________^
    |
    = note: `-D clippy::blocks-in-if-conditions` implied by `-D warnings`
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#blocks_in_if_conditions

[Nemo157]

Looks like a false-positive from the lint.

[GuillaumeGomez]

And I think I forgot to add something in the migration tests, but not sure what exactly...

[GuillaumeGomez]

Updated!

[GuillaumeGomez]

Updated! I updated the screenshot as well.

[jyn514]

This says '50 out of 52 items documented' in the screenshot - does it have a subpage that shows which items are documented and which aren't? And if not, could you add that? Maybe /crate/:crate/:version/coverage or something, you could have it as one of the tabs after 'Builds'.

[jyn514]

The general approach seems good :)

[GuillaumeGomez]

Updated! I also added it into the "popup" as @jyn514 said. :) Screenshots updated.

[jyn514]

Blocked on rustwide publishing a new version.

[pietroalbini]

Pushed two commits to this branch, bumping rustwide to 0.10.0 and removing the macro.

[GuillaumeGomez]

🎊

[sunjay]

Hey @GuillaumeGomez, thanks for taking the time to add this feature. :)

I was wondering if we could move the Coverage section to a metrics page or something? I'm not sure if it makes sense to display this so prominently in the crate information dropdown. Documentation coverage is a great thing to know, but it doesn't indicate the quality of the crate or even the quality of its documentation. (It has the same downsides as using code coverage as the only metric for how good your tests are.)

I think having access to this information is good for library authors, but we should really find a proper place for it where we can make sure it's as useful as possible. If we have a separate page or tab or something, we can keep adding more things to it to help library authors while avoiding overpopulating the crate information dropdown.

I think it's great that you added it to the Crate tab as well as the dropdown. Maybe we could remove it from the dropdown for the timebeing while we figure out how to best go about displaying this information? :)

[GuillaumeGomez]

Thanks for the feedback @sunjay!

For now, the feature itself is pretty limited, and we agree: in some cases, the doc coverage isn't an important information (pure FFI bindings for examples). However, for everything else, we've been trying in the documentation team to enforce people to document all items by adding lints (missing_docs and missing_doc_code_examples) to help crates' maintainer in this task. Putting this information forward is just the next step and I really think that it's an important information for anyone that is comparing multiple crates. Documentation quantity isn't an insurance for good documentation, but no documentation is also a good factor.

For now it's only about how many items are documented, but I intend to modify this value by also adding more checks like "how much doc comments are actually providing examples". So this is just a first step, and I am curious to see how it'll be used by users. The "hidden" goal being to force people to document more of course. :3

[yaahc]

The "hidden" goal being to force people to document more of course. :3

I agree with this goal but not the approach. This feels like negative re-enforcement to me and I think this will have negative impacts on maintainers emotionally when they're being guilt-tripped into adding docs. I'm not sure it should be removed but I think you should tread carefully here and be sure you've considered the emotional impact of this change.

[GuillaumeGomez]

I wanted to add this information in the crate results directly, but for now I think it's too early for that and that we should let maintainers get to know how it works before putting everything in place. For now, I think the display is quite discrete: when browsing through docs.rs, it's "rare" to take a look at those places, which is why we think it's fine for the moment.

The goal isn't to shame anyone for their work. Everyone should be thankful for the incredible work of all rustaceans. We're just trying to help the crate maintainers by giving them tools to improve their crate even more, and to users to help them to have a crate that matches what they need better. Some crates are not meant to be used by "external" users, which is perfectly fine, but in this case, it's "normal" (still think it's a good thing to have documentation in any case, if not others at least for yourself :p) to have less documentation and it's a good indicator that this crate might not be the one you want.

However, once again: we didn't promote this display more specifically because we want to help people understand and adjust to the system. It'll also allow us to provide even more information and tools for both crate's owners and users.

[pietroalbini]

Maybe we could make it less prominent?

[GuillaumeGomez]

@pietroalbini Might be a bit hard to get what it's talking about, no? Not really sure... However, maybe we could move the documentation coverage block under "dependencies" and "versions" blocks. What do you think?

[yaahc]

For now, I think the display is quite discrete: when browsing through docs.rs, it's "rare" to take a look at those places, which is why we think it's fine for the moment.

I looked more carefully and you're right, I had mistakenly assumed it was in the sidebar on the landing page for the crate docs when you navigate to something like docs.rs/foo, so things are already not as bad as I'd assumed.

[GuillaumeGomez]

We can add it later on, but for now (and the next months in fact...) it's not scheduled. Next step will be to provide a page to help crates' maintainers to understand how this score is computed.

[pietroalbini]

Might be a bit hard to get what it's talking about, no?

@GuillaumeGomez tried moving it at the bottom and making the text more clear:

[GuillaumeGomez]

But still not keeping what the number is about? :p

[pietroalbini]

I don't think the exact number of items documented is that important. We can make that field a link to the about page though, which already shows the detailed data.

[GuillaumeGomez]

Saying what it is seems quite important from my point of view. But as you prefer...

[pietroalbini]

Opened #961 with my proposed change