x/tools/go/packages: overlays cause loading of unneeded dependencies

[stamblerre]

Forked from #32457.

The similar issue reproduces in golangci-lint: after updating x/tools from 685fecacd0a0 to 755ce86c7629 go/packages started loading dependencies when not needed.
As I understand the regression was introduced in this overlays commit.
And this code adds all dependencies into needPkgsSet and after this to needPkgs, which are used to run go list on them:

	// Do another pass now that new packages have been created to determine the
	// set of missing packages.
	for _, pkg := range response.Packages {
		for _, imp := range pkg.Imports {
			pkgPath := toPkgPath(imp.ID)
			if _, ok := havePkgs[pkgPath]; !ok {
				needPkgsSet[pkgPath] = true
			}
		}
	}

/cc @matloob

[matloob]

Unfortunately, it's not always possible to avoid loading extra dependencies, and go/packages can't tell if they're unneeded unless the transitive set of dependencies is not changed at all.

I'll send a CL to avoid loading extra dependencies if the transitive set of dependencies is not changed. But I don't think this problem can be solved satisfactorily until go list starts handling overlays itself...

[gopherbot]

Change https://golang.org/cl/181917 mentions this issue: go/packages: avoid loading some packages in when computing overlays