x/tools/gopls: format on save breaks imports when reordered

[zikaeroh]

What did you do?

In https://github.com/hortbot/hortbot/blob/master/internal/cli/flags/ircflags/ircargs.go, swap around some import lines and save.

My Go-related configs:

    "go.lintOnSave": "workspace",
    "go.lintTool": "golangci-lint",
    "go.formatTool": "goimports",
    "go.useLanguageServer": true,
    "go.alternateTools": {
        "go-langserver": "gopls"
    },
    "go.buildOnSave": "off",
    "go.vetOnSave": "off",
    "[go]": {
        "editor.snippetSuggestions": "none",
        "editor.formatOnSave": true,
        "editor.codeActionsOnSave": {
            "source.organizeImports": true
        },
        "editor.codeActionsOnSaveTimeout": 3000
    },
    "gopls": {
        "completeUnimported": true,
        "watchFileChanges": true,
        "deepCompletion": true,
        "go-diff": true,
        "local": "github.com/hortbot/hortbot"
    },

Though I'd normally configure local in the project's settings, since it doesn't make much sense globally.

What did you expect to see?

The imports are reordered back to the way they were (or re-sorted by the local feature).

What did you see instead?

The imports get formatted very wrong. Imports get deleted, duplicated, and sometimes turn out syntatically correct with misplaced quotes. I tried to do a screencap.

https://streamable.com/ffxjz

I also verified this without go-diff enabled to see if it was the diff algorithm, but there was no difference.

Also, the local option is applied at an unknown time. I want things to behave exactly as they do with VS Code + goimports, i.e. "hit save, imports are reordered and fixed", but can't seem to get that to work.

Build info

golang.org/x/tools/gopls master
    golang.org/x/tools/gopls@v0.1.8-0.20191105231337-689d0f08e67a h1:4tnHOn2ZDC1EOp6PyPlUfrxeuw1IzXmJ6Hh359rRlSI=
    github.com/BurntSushi/toml@v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/sergi/go-diff@v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
    golang.org/x/sync@v0.0.0-20190423024810-112230192c58 h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
    golang.org/x/tools@v0.0.0-20191105231337-689d0f08e67a h1:RzzIfXstYPS78k0QViPGpDcTlV+QuYrbxVmsxDHdxTs=
    golang.org/x/xerrors@v0.0.0-20190717185122-a985d3407aa7 h1:9zdDQZ7Thm29KFXgAX/+yaf3eVbP7djjWp/dXAppNCc=
    honnef.co/go/tools@v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=

Go info

go version go1.13.4 linux/amd64

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jake/.cache/go-build"
GOENV="/home/jake/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jake/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/jake/zikaeroh/hortbot/hortbot/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build676081404=/tmp/go-build -gno-record-gcc-switches"

[gopherbot]

Thank you for filing a gopls issue! Please take a look at the Troubleshooting guide, and make sure that you have provided all of the relevant information here.

[stamblerre]

Can you reproduce this by running the gopls imports command? Can you share your gopls logs from when this happens (https://github.com/golang/tools/blob/master/gopls/doc/troubleshooting.md#capturing-logs)?

[zikaeroh]

gopls imports makes no change to my file after I reorder my imports (-d flag shows no output).

Logs here: https://gist.github.com/zikaeroh/7eb4a0a7d7d21d929185315bd8cdcb45

Here is a screenshot of the bad quoting (since I couldn't get it in the screen recording):

[stamblerre]

Thanks! I'll take a look at the logs. Can you also share the output of running gopls imports with gopls -rpc.trace -v imports?

[stamblerre]

Interestingly enough, the only time textDocument/codeAction returns a value in your log is this:

[Trace - 20:38:56.202 PM] Received response 'textDocument/codeAction - (10)' in 1445ms.
Result: [{"title":"Organize Imports","kind":"source.organizeImports","edit":{"changes":{"file:///home/jake/zikaeroh/hortbot/hortbot/internal/cli/flags/ircflags/ircargs.go":[{"range":{"start":{"line":10,"character":3},"end":{"line":11,"character":3}},"newText":""},{"range":{"start":{"line":12,"character":53},"end":{"line":12,"character":53}},"newText":"\"\n\t\"github.com/hortbot/hortbot/internal/pkg/ctxlog"}]}}}]

All it's doing here is deleting a line and adding the "github.com/hortbot/hortbot/internal/pkg/ctxlog" important, which doesn't sound like it should cause the results you're seeing. Is it possible that both gopls and some other save hook are applying edits to your file?

[zikaeroh]

Output:

2019/11/05 20:51:19 Info:2019/11/05 20:51:19 Build info
----------
golang.org/x/tools/gopls master
    golang.org/x/tools/gopls@v0.1.8-0.20191105231337-689d0f08e67a h1:4tnHOn2ZDC1EOp6PyPlUfrxeuw1IzXmJ6Hh359rRlSI=
    github.com/BurntSushi/toml@v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/sergi/go-diff@v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
    golang.org/x/sync@v0.0.0-20190423024810-112230192c58 h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
    golang.org/x/tools@v0.0.0-20191105231337-689d0f08e67a h1:RzzIfXstYPS78k0QViPGpDcTlV+QuYrbxVmsxDHdxTs=
    golang.org/x/xerrors@v0.0.0-20190717185122-a985d3407aa7 h1:9zdDQZ7Thm29KFXgAX/+yaf3eVbP7djjWp/dXAppNCc=
    honnef.co/go/tools@v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=

Go info
-------
go version go1.13.4 linux/amd64

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jake/.cache/go-build"
GOENV="/home/jake/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jake/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/jake/zikaeroh/hortbot/hortbot/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build496587565=/tmp/go-build -gno-record-gcc-switches"
2019/11/05 20:51:20 Info:2019/11/05 20:51:20 go/packages.Load
	packages = 1
2019/11/05 20:51:20 Info:2019/11/05 20:51:20 go/packages.Load
	package = github.com/hortbot/hortbot/internal/cli/flags/ircflags
	files = [/home/jake/zikaeroh/hortbot/hortbot/internal/cli/flags/ircflags/ircargs.go]
// Package ircflags processes IRC-related flags.
package ircflags

import (
	"context"
	"database/sql"
	"time"

	"github.com/hortbot/hortbot/internal/birc"
	"github.com/hortbot/hortbot/internal/db/modelsx"
	"github.com/hortbot/hortbot/internal/db/redis"
	"go.uber.org/zap"
	"github.com/hortbot/hortbot/internal/pkg/apis/twitch"
	"github.com/hortbot/hortbot/internal/pkg/ctxlog"
	"github.com/hortbot/hortbot/internal/pkg/twitchx"
)

type IRC struct {
	Nick         string        `long:"irc-nick" env:"HB_IRC_NICK" description:"IRC nick" required:"true"`
	Pass         string        `long:"irc-pass" env:"HB_IRC_PASS" description:"IRC pass" required:"true"`
	PingInterval time.Duration `long:"irc-ping-interval" env:"HB_IRC_PING_INTERVAL" description:"How often to ping the IRC server"`
	PingDeadline time.Duration `long:"irc-ping-deadline" env:"HB_IRC_PING_DEADLINE" description:"How long to wait for a PONG before disconnecting"`
	Token        bool          `long:"irc-token" env:"HB_IRC_TOKEN" description:"Use a token from the database if available"`

	RateLimitSlow   int           `long:"irc-rate-limit-slow" env:"HB_IRC_RATE_LIMIT_RATE" description:"Message allowed per rate limit period (slow)"`
	RateLimitFast   int           `long:"irc-rate-limit-fast" env:"HB_IRC_RATE_LIMIT_RATE" description:"Message allowed per rate limit period (fast)"`
	RateLimitPeriod time.Duration `long:"irc-rate-limit-period" env:"HB_IRC_RATE_LIMIT_PERIOD" description:"Rate limit period"`
}

var DefaultIRC = IRC{
	PingInterval:    5 * time.Minute,
	PingDeadline:    5 * time.Second,
	RateLimitSlow:   15,
	RateLimitFast:   80,
	RateLimitPeriod: 30 * time.Second,
}

func (args *IRC) Pool(ctx context.Context, db *sql.DB, tw twitch.API) *birc.Pool {
	logger := ctxlog.FromContext(ctx)

	channels, err := modelsx.ListActiveChannels(ctx, db, args.Nick)
	if err != nil {
		logger.Fatal("error listing initial channels", zap.Error(err))
	}

	nick := args.Nick
	pass := args.Pass

	if args.Token {
		token, err := twitchx.FindBotToken(ctx, db, tw, nick)
		if err != nil {
			logger.Fatal("error querying for bot token", zap.Error(err))
		}
		if token != nil {
			logger.Debug("using token from database")
			pass = "oauth:" + token.AccessToken
		}
	}

	pc := birc.PoolConfig{
		Config: birc.Config{
			UserConfig: birc.UserConfig{
				Nick: nick,
				Pass: pass,
			},
			InitialChannels: channels,
			Caps:            []string{birc.TwitchCapCommands, birc.TwitchCapTags},
			PingInterval:    args.PingInterval,
			PingDeadline:    args.PingDeadline,
		},
	}

	return birc.NewPool(pc)
}

func (args *IRC) SendMessageAllowed(ctx context.Context, rdb *redis.DB, origin, target string) (bool, error) {
	return rdb.SendMessageAllowed(ctx, origin, target, args.RateLimitSlow, args.RateLimitFast, args.RateLimitPeriod)
}

I am not aware of any other save hook unless the VS Code extension is still trying to run an external tool even though I have not disabled gopls's formatting. For fun, I replaced my goimports binary with a script that just echo'd a string to try and get it to break or replace the buffer, but the behavior I gave in my original post persists.

[stamblerre]

Thanks for trying that out. I guess we can rule that out then. To confirm with the gopls imports command - this is the original file? Can you see what happens when you remove some imports? If you run it a couple times, does it give you consistent results?

[zikaeroh]

The above tests were done simply with the reorderings. If I only remove an import, I do see it trying to add it back. With gopls imports -d:

$ gopls imports -d ircargs.go                      
--- /home/jake/zikaeroh/hortbot/hortbot/internal/cli/flags/ircflags/ircargs.go.orig
+++ /home/jake/zikaeroh/hortbot/hortbot/internal/cli/flags/ircflags/ircargs.go
@@ -10,7 +10,8 @@
 	"github.com/hortbot/hortbot/internal/db/modelsx"
 	"github.com/hortbot/hortbot/internal/db/redis"
 	"github.com/hortbot/hortbot/internal/pkg/apis/twitch"
-	"github.com/hortbot/hortbot/internal/pkg/twitchx"
+	"github.com/hortbot/hortbot/internal/pkg/ctxlog"
+	"github.com/hortbot/hortbot/internal/pkg/twitchx"
 	"go.uber.org/zap"
 )

[stamblerre]

Can you try using the imports from gopls without the local setting? I would be curious to see if you experience the same behavior without it. Also, if you are able to share a log from when the imports behave strangely that would be helpful, even if it's very long - it's just hard to investigate since your previous log didn't seem to have any incorrect behavior in it.

[zikaeroh]

Hopefully this is what you're looking for: https://gist.github.com/zikaeroh/3300ac58971886fd3f217f9338da2c56

local is unset. The behavior is still there. I made a number of changes, ordering, removed imports, swapped lines, undid, etc. Experienced bad import replacements, quotes getting misplaced, as well as the imports even appearing outside of the import block.

[zikaeroh]

Just to try and eliminate the Go extension as the source of the problem, I configured it to run the formatting tool with a bad flag (-THISISBAD). The bad changes still happen, which tell me at least that the Go extension isn't running an external tool (since it'd otherwise give me an error, or at least do nothing). I have nothing else installed that could do any formatting.