cmd/go: should default to use mod=vendor flag if the vendor folder exists

[JeremyLoy]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"

What did you do?

• Given a fresh install of go, there are no files downloaded in $GOPATH/pkg/mod.
• I cloned a repo that uses the new Go 1.11 modules with vendored dependencies
• I executed go run main.go to run the application.

What did you expect to see?

• the application ran without downloading external dependencies.

What did you see instead?

• The go tool downloaded all dependencies to $GOPATH/pkg/mod, despite the fact that the dependencies were vendored

I know this can easily be fixed by instead go run -mod=vendor main.go, but because the go tool is now being advertised as "context aware", I think this would be a good QoL change.

[thepudds]

@JeremyLoy note that while the default is to ignore the vendor directory, you can relatively easily change that behavior with an environmental setting GOFLAGS=-mod=vendor. You could use that in CI, or could set it in a .bashrc or similar for day-to-day use if you want.

This is a new environmental variable in Go 1.11, where one of the justifications for introducing it was specifically to make it easier to opt in to vendoring given some people would want that regularly (e.g., see #26585 or https://go-review.googlesource.com/c/go/+/126656).

From the doc:

GOFLAGS
	A space-separated list of -flag=value settings to apply
	to go commands by default, when the given flag is known by
	the current command. Flags listed on the command-line
	are applied after this list and therefore override it.

One other quick note is that those downloads end up in a shared cached (so even if you don't opt-in to vendoring, it cuts down on how many times you would end up downloading dependencies).

[rasky]

I strongly agree with this proposal. If a project committed a vendor folder, this is a strong indicator that vendoring should be used by default. I can’t think of a workflow in which a developer would commit a vendor folder but prefer other developers to ignore it by default.

In other words, if a project decides to use vendoring, then the whole go tool should default to use vendor first, and download/cache only if not there.

[FiloSottile]

By the way, this is part of what GOFLAGS was added for: https://go-review.googlesource.com/c/go/+/126656

[rasky]

GOFLAGS isn’t the right solution here. This is not a per-user / per-computer decision like “add debug flags”. This is a per-project decision that should affect all users building it. Currently, if a project uses vendoring, there is no way to let all builders use vendoring by default.

[JeremyLoy]

WhileGOFLAGS is a decent immediate workaround, I do want to stress the importance of the vendor folder as a part of this issue.

To put it bluntly, Github is not an artifact repository. At any given time, a repository owner can either

1. delete their account
2. delete a repository
3. force push deletions to master
4. remove a tag
5. change a tag

To avoid any of the above damaging your project, vendoring is a necessity, and should be treated as a first class citizen in the go toolchain.

There seems to be a rush to remove vendor from the go toolchain. It is still referenced as an "experiment" and there was a bit of swirl in even getting modules to support vendoring in its first release. Until Project Athens or equivalent becomes mainstream, vendoring should be fully supported (but never required).

EDIT: coincidentally, I found out the official Heroku build pack for Go uses this very workaround. It seems like this really should be the default case.

[myitcv]

@rasky

Currently, if a project uses vendoring, there is no way to let all builders use vendoring by default.

But we're talking about working in module mode here, correct? Or by "currently" do you mean GOPATH-mode? Because vendoring as a "first class citizen" continues to work in GOPATH mode, it's just that in module mode it is not the default.

@JeremyLoy

To avoid any of the above damaging your project, vendoring is a necessity, and should be treated as a first class citizen in the go toolchain.

There was a similar discussion over at https://twitter.com/_myitcv/status/1033824806930534400. Vendoring (as it was and still is in GOPATH mode) is one way of dealing with this problem. As I suggested, having a separate repo as a module download cache is another, which actually have a number of derivative benefits.

As you say, Athens (or indeed anything else that talks the published module protocol) are also solutions, solutions that in some cases try to solve even bigger problems.

There seems to be a rush to remove vendor from the go toolchain

I don't think this is quite accurate. In GOPATH mode (the default), vendoring works exactly as it has done up until now. In module mode (the opt in) the behaviour changes.

So for the record, in module mode I am against what is being proposed here, not least because of what I wrote above, but also because it confuses/complicates the rule that "vendor directories are totally ignored in module mode" which is particularly important when it to dependencies.

[rasky]

Because vendoring as a "first class citizen" continues to work in GOPATH mode, it's just that in module mode it is not the default.

Yes, I'm speaking about module mode. The point is that "not being the default" for projects that uses vendoring (that is, projects in which developers went through the process of calling go mod vendor and commit the vendor directory) is the wrong default. If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used. People that don't use vendoring don't need to have vendoring explicitly disabled; they simply won't have a vendor directory in their project. By switching the vendoring support off in module mode, go 1.11 is making using vendoring more difficult for no reasons I can understand, without making things simpler for those who don't need or want vendoring.

There was a similar discussion over at https://twitter.com/_myitcv/status/1033824806930534400. Vendoring (as it was and still is in GOPATH mode) is one way of dealing with this problem. As I suggested, having a separate repo as a module download cache is another, which actually have a number of derivative benefits.

Thanks, but that is largely suboptimal. It requires a custom installation step to be repeated for each user, and also handle and commit binary files into the SCM which are harder to handle (especially with git) and review.

As you say, Athens (or indeed anything else that talks the published module protocol) are also solutions, solutions that in some cases try to solve even bigger problems.

I fail to see how Athens solves the needs of people using vendoring. AFAICT Athens helps with the issue that people can remove/sell GitHub repositories to other parties, and this can break builds and/or even cause security problems. This is not what vendoring is after. Vendoring is required in workflows that need 100% of the source code for a project committed under a single SCM, so that it can be fully audited, versioned, and built completely offline, without any dependency on any third-party service or installation. I deliver code to enterprises that want the code to be complete, not depend on third party services, and fully versioned with their existing code tracking tools. Vendoring solves this problem; Athens doesn't seem to.

So for the record, in module mode I am against what is being proposed here, not least because of what I wrote above, but also because it confuses/complicates the rule that "vendor directories are totally ignored in module mode" which is particularly important when it to dependencies.

You seem against adding a feature for people that need it for the sake of orthogonality. If you don't want or need vendoring, there's nothing in vendoring support that creates problems to you. Just use modules, your download cache, Athens or whatever you prefer. You will never have a vendor directory in your repos and nothing will break or be suboptimal for you. But if I need vendoring support and commit a vendor directory, why should vendoring support be disabled by default for me, causing more difficult builds for all involved developers?

[bcmills]

If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used.

That's not at all “obvious” to me. In fact, the current structure (in 1.11 module mode) is exactly the opposite: if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.

Giving their users an option is not at all the same thing as providing an explicit recommendation.

Moreover, vendor directories can only reasonably work at the topmost level. If a developer makes edits within the vendor folder (intentionally or otherwise), they would effectively perform an implicit replace operation without surfacing that fact in the rest of the UI (such as in the output of go list).

[rasky]

If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used.

That's not at all “obvious” to me. In fact, the current structure (in 1.11 module mode) is exactly the opposite: if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.

I'm not sure that we can deduce anything from what Go 1.11. It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures. What we ended up with is basically non working for many vendoring use cases, hence this bug (and others).

Rather than discussing hypothesis made by people that don't vendor, I'd rather somebody that does use vendoring in production shows up and say that they absolutely don't want vendored modules to be used by default unless explicitly opted in (like Go 1.11 does), and explain us why with a small experience report. I for sure know this doesn't work for my use case as already explained in mine, and others have already chimed in.

Moreover, vendor directories can only reasonably work at the topmost level. If a developer makes edits within the vendor folder (intentionally or otherwise), they would effectively perform an implicit replace operation without surfacing that fact in the rest of the UI (such as in the output of go list).

That's perfectly fine for me at least. In fact, in my use case, vendoring support could go as far as completely subsuming the module download cache and/or go.sum. That is, it would be great that once you opt-in to "vendoring mode", all new modules are automatically expanded into the vendor directory, used by default, removed when they're not used, etc.

[bcmills]

I'd rather somebody that does use vendoring in production shows up and say that they absolutely don't want vendored modules to be used by default unless explicitly opted in (like Go 1.11 does), and explain us why with a small experience report.

That's kind of my point, though: you're requesting feedback from one particular subset (package maintainers that choose to use vendoring) and ignoring another (package consumers, who have no choice in the matter in GOPATH mode, but do now have a choice in module mode).

We need experience reports from everyone — including package maintainers and consumers, and including both open-source and enterprise codebases.