x/tools/cmd/gopls: overlays cause go/packages to perform giant "go list" query

[muirdm]

I had ~60 go files open in gopls. None of them had any unsaved changes (i.e. bytes on disk matched bytes in gopls memory). I noticed when opening a new file in my editor gopls was taking a long time. I tracked it down to the overlay logic in go/packages.

Here is roughly what happened.

1. I opened file foo.go in a previously unopened package.
2. gopls does a go/packages file=foo.go query and passes an overlay that includes all 60 files.
3. In processGolistOverlay() it appends a new package to response.Packages for each overlay file's package
4. It marks all imports of response.Packages in needPkgsSet
5. addNeededOverlayPackages() performs a "go list" call for all items in needPkgs, which takes 3 seconds in my case.

I don't fully understand the package overlay stuff, but here are my observations:

• it doesn't seem like we need to treat "saved" files as overlays
• even if they are "real" overlays, go/packages seems to be doing too much work when querying only for foo.go (i.e. most overlay packages are not related to foo.go's package, so why do we list them and all their dependencies?)
• the overlay work is redone every time gopls invokes go/packages to look up a new file even if none of the overlays have changed

/cc @matloob @stamblerre

[muirdm]

I'd be happy to work on this if I could get some direction on the expected behavior in this case.

[jirfag]

The similar issue reproduces in golangci-lint: after updating x/tools from 685fecacd0a0 to 755ce86c7629 go/packages started loading dependencies when not needed.
As I understand the regression was introduced in this overlays commit.
And this code adds all dependencies into needPkgsSet and after this to needPkgs, which are used to run go list on them:

	// Do another pass now that new packages have been created to determine the
	// set of missing packages.
	for _, pkg := range response.Packages {
		for _, imp := range pkg.Imports {
			pkgPath := toPkgPath(imp.ID)
			if _, ok := havePkgs[pkgPath]; !ok {
				needPkgsSet[pkgPath] = true
			}
		}
	}

[gopherbot]

Change https://golang.org/cl/181123 mentions this issue: internal/lsp: add an implementation for textDocument/didSave

[stamblerre]

@muirrn: The change above should have reduced the overlays sent to go/packages to only the files that have not been saved.

I filed #32538 so that we can decide how to handle this in go/packages.