make: stale diagnostics info in .json files can break unclean builds

[pnkfelix]

make: error E0038 from librustc' also found intmp/extended-errors/librustc_typeck.json' (from unclean build?)

That is, I did a git pull up to:

commit fa433875274ff4a7c4cab7d87c1284ba782ef643
Merge: 30a42fa b799cd8
Author: bors <[email protected]>
Date:   Wed May 13 06:43:05 2015 +0000

    Auto merge of #24619 - nrc:rc-coerce, r=nikomatsakis

    r? @nikomatsakis (note a few TODOs left in the code where I wasn't sure about stuff).

in a build directory that already had artifacts from a prior build lying around.

I re-ran configure (something I do sometimes when git pull'ing with an unknown set of changes):

/Users/fklock/Dev/Mozilla/rust.git/configure --enable-debug --enable-optimize     --enable-ccache --enable-clang --prefix=~/opt/rust-dbg --disable-optimize-tests --disable-llvm-assertions 

And then when attempting the build, I got this:

rustc: x86_64-apple-darwin/stage1/lib/rustlib/x86_64-apple-darwin/lib/librustc
/Users/fklock/Dev/Mozilla/rust.git/src/librustc/lib.rs:170:1: 170:52 error: error E0038 from `librustc' also found in `tmp/extended-errors/librustc_typeck.json'
/Users/fklock/Dev/Mozilla/rust.git/src/librustc/lib.rs:170 __build_diagnostic_array! { librustc, DIAGNOSTICS }
                                                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: aborting due to previous error

[pnkfelix]

(I am now double-checking whether a build from a fresh directory gets rid of the problem, as I assume it will. If so, then this is a relatively low priority ticket -- potentially even closable if the problem does not re-exhibit itself after that fresh build finishes...)

[pnkfelix]

(removed comment with totally bogus claims)

[pnkfelix]

Ah, I think I see the problem: our diagnostics plugin works by treating the json files as representing the error codes registered by all the crates.

But if there's a json file lying around from a previous build for a crate that we have not yet rebuilt, then we can erroneously load it in and report a conflict. One will only observe this occurring if the distribution of diagnostic codes has shifted between the various crates.

(Unfortunately in my experience make is not so great at encoding the production of side-products, e.g. these .json files, in its rules.)

Anyway, the simplest thing might be to add a make rule that just deletes the .json files if they predate any of the files from its origin crate. (I.e., work with the assumption that touching any of those files implies that the .json is out-of-date.) I will see what I can do about that.

[pnkfelix]

(deleted comment that was meant for #25380)

[pnkfelix]

See also PR #25592, which notes that in addition to the bug outlined here, there is also the issue that:

The code does not currently handle concurrent compilations where JSON files in the metadata directory may be half-written and hence contain invalid JSON when read.

[steveklabnik]

Triage: hopefully fixed by rustbuild?

[alexcrichton]

I think we've since fixed this, so closing.