x/vgo: broken json output in list -m -json when using replace directive

[dlsniper]

What version of Go are you using (go version)?

vgo @ 6cd5a417451d8ee907692eded07ef1b6b53825b1

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

Windows 10/amd64

What did you do?

Using the following example project: https://github.com/dlsniper/vgoawesomeProject I changed the go.mod file with the following content

module vgoawesomeProject

require github.com/dlsniper/vgodemo v1.0.0

replace github.com/dlsniper/vgodemo v1.0.0 => github.com/dlsniper/vgodemo v0.0.0-20180318080540-4356961a9

Then I ran vgo list -m -json all. I would expect to only get JSON output not any extra output about the replace directives and such.

What did you expect to see?

{
        "Path": "vgoawesomeProject",
        "Main": true
}
{
        "Path": "github.com/dlsniper/vgodemo",
        "Version": "v1.0.0",
        "Replace": {
                "Path": "github.com/dlsniper/vgodemo",
                "Version": "v0.0.0-20180704102233-4356961a96dc",
                "Time": "2018-07-04T10:22:33Z"
        },
        "Time": "2018-07-04T10:22:33Z"
}
{
        "Path": "github.com/gorilla/mux",
        "Version": "v1.6.2",
        "Time": "2018-05-13T03:22:33Z",
        "Dir": "D:\\go\\src\\mod\\github.com\\gorilla\\[email protected]"
}

What did you see instead?

vgo: finding github.com/dlsniper/vgodemo v0.0.0-20180704102233-4356961a96dc
{
        "Path": "vgoawesomeProject",
        "Main": true
}
{
        "Path": "github.com/dlsniper/vgodemo",
        "Version": "v1.0.0",
        "Replace": {
                "Path": "github.com/dlsniper/vgodemo",
                "Version": "v0.0.0-20180704102233-4356961a96dc",
                "Time": "2018-07-04T10:22:33Z"
        },
        "Time": "2018-07-04T10:22:33Z"
}
{
        "Path": "github.com/gorilla/mux",
        "Version": "v1.6.2",
        "Time": "2018-05-13T03:22:33Z",
        "Dir": "D:\\go\\src\\mod\\github.com\\gorilla\\[email protected]"
}

[bcmills]

Did the extra output go to stdout, or just to stderr?

(CC: @rsc @myitcv)

[seeruk]

I've just given this a try, and extra output goes to stderr. Additionally, it's not just triggered by using the replace directive, it's any time vgo needs to find a dependency, e.g. if you wipe out your vgo cache and then run vgo list -m -json it'll list every module your project depends (to stderr) then finally print the JSON to stdout when it's done.

[myitcv]

@bcmills this picks up on what we were discussing back in https://go-review.googlesource.com/c/vgo/+/118316

Would one solution here be to add (and document?) a -progress flag which takes a single argument, the path to a file? If provided, progress would be written to this file (the file would in all likelihood be one half of an os.Pipe()); else progress would be written to stderr (which works for the default terminal usage). Errors are written to stderr regardless, and cause a non-zero exit code (unless -e is provided, the current behaviour of go list). That way any tool consuming the output of vgo list (-m) -json would provide -process and be able to consume CombinedOutput, safe in the knowledge the it would contain only valid JSON unless an error was also returned, in which case CombinedOutput would also have error messages etc interleaved. This would also give tools the ability to relay progress to the user if they need to.

That said, I'm unclear if this pipe approach works on Windows...

[rsc]

This is working as intended.
All tools must separate stdout and stderr when reading JSON from any of the go commands.
Stdout will always be JSON.
But if an error happens, it will be printed to stderr, and probably stdout will be empty.
If the command exits with a non-zero status, use stderr and ignore stdout.

If the minor warning chatter here exposes tools that are not handling stdout/stderr correctly, that's unfortunate but not entirely terrible. Getting them fixed will make the tools more robust overall.