proposal: cmd/go: add a sub command to manage authentication

[marwan-at-work]

The current state

The Go command utilitizes the ~/.netrc file, both implicitly and explicitly, to authenticate against remote servers when downloading Go code.

Go uses ~/.netrc implicitly when using "direct" downloads because git uses libcurl to fetch dependencies through https which in turn uses the ~/.netrc file to forward credentials.

Go uses ~/.netrc explicitly when authenticating against a GOPROXY server by looking for a matching "machine" URL with a valid login and password and forwards those credentials as a BasicAuth header.

Problem statement

A Go programmer who wants to set their credentials (whether against a proxy or VCS) must know how to create/edit the .netrc file in their home directory manually. This is has a few problems:

1. It is not well documented. The only place I can find a mention of the .netrc file is in the Go FAQ and that is because I was explicitly looking for it.

2. It is not a good UX: you have to learn/follow the netrc syntax to configure your credentials. Furthermore, Go does not validate the .netrc syntax for you. For example, machine myproxy.com login mytoken silently fails and does not send the credentials to myproxy.com unless I explicitly put machine myproxy.com login myuser password mytoken. On the other hand, machine github.com login mytoken works just fine for VCS authentication (since this is handled by git and libcurl directly and not by Go)

3. Most importantly, I found that newcomers to Go find this confusing and hard to deal with in comparison to other languages:

Many languages and tools abstract authentication management in their command line:

1. NodeJS has npm login, npm logout, and npm config set|get which all manage the ~/.npmrc file.

2. Ruby has gem signin and gem signout to manage credentials as well (~/.gem/credentials)

3. Dart's pub command line lets you manage ~/.pub-cache/credentials.json through an interactive browser that signs in to a Google account when running pub publish and it also has pub signout to remove those credentials.

4. Docker (though not a language but certainly has a registry) has docker login and docker logout

5. gcloud (though not a language) can also manage the credentials on the filesystem via gcloud auth login and can be static or interactive

And the list goes on.

Proposal

Go should provide a more pleasant, and less error prone, way to configure user's credentials for downloading private module dependencies.

Specifically, Go should be able to create and edit the ~/.netrc file without the user's direct manipulation of it.

Go should be able to add/edit/remove specific lines in the ~/.netrc through the Go command line.

Examples

Please note: The following syntax is arbitrary and can definitely be changed. This proposal is more about getting agreement that we should let the Go command manage the .netrc file and is not picky about what the syntax will look like.

That said, suggestion on what the command syntax would look like is welcome here.

1. Login to github.com

$ go auth login -host=github.com -user=marwan-at-work -password=myToken 

$ cat ~/.netrc
machine github.com login marwan-at-work password myToken

$ echo "myToken" | go auth login -host=myproxy.com -user=marwan --password-stdin

$ cat ~/.netrc
machine github.com login marwan-at-work password myToken
machine myproxy.com login marwan password myToken

2. Logout of github.com

$ go auth logout github.com

$ cat ~/.netrc
machine myproxy.com login marwan password myToken

3. List current authentications

$ go auth list # or go list auth

cc: @heschik @bcmills @jayconrod (I added the modules label, but I don't think it's exclusively for modules so I'm not sure what other label this might fit into)

Thank you!

[jayconrod]

To address a couple points:

1. Lack of documentation is certainly a problem. Hopefully, the upcoming module reference documentation will address that, specifically in CL 240686. We may want to add other how-to pages on that.
2. About silent failures with .netrc parsing: that seems like a bug we should fix in any case, regardless of what's decided here. Mind filing an issue?

That said, I'm not sure we should rely on .netrc as our primary authentication mechanism, long-term. Until we decide that, we should hold off adding more explicit support on the command line. The main problems are 1) no support for whitespace characters in passwords, 2) no support for other authentication methods (cookies, SSH keys, etc.), 3) no support for module-specific or repository-specific credentials.

#26232 is a feature request for an extensible authentication mechanism in the style of git-credential-helper. @bcmills was working on that last year, but it didn't make it into 1.14, and we didn't get back to it for 1.15.

That would provide a lot of flexibility, but I'm not sure it would make the user experience any simpler. Perhaps a go auth (or go mod auth) command could be built together with that?

[marwan-at-work]

About silent failures with .netrc parsing: that seems like a bug we should fix in any case, regardless of what's decided here. Mind filing an issue?

Done: #40215

That would provide a lot of flexibility, but I'm not sure it would make the user experience any simpler

Agreed. The GOAUTH env var would probably be even more complex for users to configure so a go auth command would hopefully mitigate the complexity here.

Perhaps a go auth (or go mod auth) command could be built together with that?

Could we not have a go auth command that supports both the current .netrc mechanism for the time being and can be adjusted to support GOAUTH once/if it gets implemented? This would provide a better experience for users who are trying to use this to authenticate now. I definitely understand that .netrc has its limitation, but to Go users, this is the only option at the moment.

Thanks!

[bcmills]

Under the original GOAUTH proposal, git users would be able to use git credential fill, which is approximately as complex (or simple) as cat >> ~/.netrc.

But it's true that git credential erase has no equally-simple analogue at the moment.

[bcmills]

That said, I would prefer to keep the go command orthogonal to its credential stores. For editing .netrc files, I would rather see a .netrc-specific tool than a go-specific one, given that .netrc files may in general contain credentials intended for use with not only the go command but also other tools (such as curl and ftp).

[marwan-at-work]

Under the original GOAUTH proposal, git users would be able to use git credential fill, which is approximately as complex (or simple) as cat >> ~/.netrc. But it's true that git credential erase has no equally-simple analogue at the moment.

From a UX perspective, I don't think is a good idea. A nice gain of having a GOPROXY is that we can tell users they don't need to have "git" and other VCS binaries in their build step because a GOPROXY will take care of module fetching for you. If we had to tell them they need to make sure they have "git" just to authenticate against a GOPROXY, it feels like a step in the wrong direction. But definitely correct me if I missed something.

That said, I would prefer to keep the go command orthogonal to its credential stores. For editing .netrc files, I would rather see a .netrc-specific tool than a go-specific one, given that .netrc files may in general contain credentials intended for use with not only the go command but also other tools (such as curl and ftp).

Agreed. But telling Go developers to install a third party tool just so they can make Go understand and pass credentials is not the most ideal. Though it's certainly better than telling Go developers to hand craft a ~/.netrc file.

Other than the above, I see two options:

1. Have the Go command be able to parse and edit the .netrc file. It can already do the former. This way, if the .netrc is also being used for libcurl/ftp, Go would intelligently edit the file keeping those tools happy.

2. Introduce a .gorc file similar to how most languages work (as I showcased in the issue description). This would ensure that .gorc files are not shared with other tools like curl/ftp and the Go command can take full ownership of it. When GOAUTH is implemented, the .gorc file can simply be one of the credential store options which can then sit next to a more flexible/secure way for managing credentials.

Thanks

[Sylvain2703]

I like the idea of a GOAUTH env variable, for at least a case: GCP Cloud Functions.

According to the Cloud Functions documentation for Go, the only way to use a private Go module is to vendor it (which is very constraining).

Having a GOAUTH env var will allow us to fetch modules from private repos in Google Cloud Functions.
Here is an example:
gcloud functions deploy FunctionName --set-env-vars GOAUTH=user:password@host,GOPRIVATE=gitlab.com/compagny/*"

[salewski]

That said, I'm not sure we should rely on .netrc as our primary authentication mechanism, long-term. Until we decide that, we should hold off adding more explicit support on the command line. The main problems are 1) no support for whitespace characters in passwords, 2) no support for other authentication methods (cookies, SSH keys, etc.), 3) no support for module-specific or repository-specific credentials.

Another problem, as it pertains to HTTP Basic Auth, is that (in accordance with RFC 7617) the user-id cannot contain a colon character (:) because that is the separator character between the <user-id> and <password> fields (and there is no escape mechanism to work around it).

[rsc]

To the extent that this issue is about maintaining ~/.netrc, it sounds like we should encourage prototyping a netrc manager outside the Go command.

It would be nice to have more of a problem statement about how people authenticate to their source repos and what they need. There are many things netrc does not address.

[marwan-at-work]

It would be nice to have more of a problem statement about how people authenticate to their source repos and what they need. There are many things netrc does not address.

@rsc below is my problem statement and more likely an experience report stemming from the last 3 companies I have worked at where we needed to authenticate and fetch private modules across different VCS repositories.

In all 3 companies, there were 2-3 areas when we needed to access private Go dependencies:

1. Local development
2. CI/CD (for example running unit tests before deploying)
3. Deployment environment (not relevant if your CI/CD knows how to download private dependencies and package the application, but probably more relevant if you're using products like Google Cloud Functions as mentioned above).

...

1. For local development, the workflow has always been:

• Make sure you have configured "git" either through SSH or through the .netrc file locally. And also don't forget to set your GOPRIVATE so that the checksum db doesn't yell at you. Or,

• Make sure you configure your GOPROXY to the internal company. If the internal GOPROXY is behind a VPN then you must always be logged in to the VPN to fetch modules. Otherwise, you must ensure to configure your .netrc correctly AND make sure to NOT configure GOPRIVATE but instead to configure GONOSUMDB (or keep GOPRIVATE but make sure to add GONOPROXY according to go help module-private). This can be quite confusing especially if a company introduces a GOPROXY after the employees have been using GOPRIVATE and vendoring their private modules.

Problem statement: This is by far the most frequently reported issues in our internal slack channels at work. People always run into issues trying to get/upgrade a private dependency and they have forgotten to update their GOPRIVATE or they have misconfigured their .netrc or git config.

Furthermore, people rarely know the difference between GOPRIVATE and GONOSUMDB. To make somewhat of a strong statement: If they are using a GOPROXY for their private modules, they really shouldn't be setting GOPRIVATE without a GONOPROXY and vice versa. But the underlying issue is the same: more often than not, my colleagues always got their configuration wrong and got frustrated until someone else helped them fix their configuration.

This also has the side effect of dividing our internal Go community into two categories: the super-users and the non-super-users. I ultimately don't want to make people feel that they don't have sufficient knowledge or that other colleagues are more knowledgable than them. Instead, I'd want them to understand and enjoy Go's simplicity the way I have because Go tends to abstract the really difficult parts with simple and easy to understand APIs.

2. For CI/CD:

• Always vendor your code and therefore no need to do anything special when it comes to fetching private dependencies in CI/CD.

Problem statement: I'd be more than happy to go into detail here. But I think the downsides of the vendor folder is quite well understood to the Go team and the Go community. To name some high level issues with vendoring: bloated repositories, stale/disappearing dependencies, inability to upgrade any module that has a stale/missing transitive dependency, poor lsp/editor integration (I'd be super happy to expand on this one) and more.

• Create an internal GOPROXY for a company to mitigate the issues that stem from vendoring.

Problem statement: a lot of companies might deploy their GOPROXY within VPN so that there wouldn't be a need for authentication, but then their CI/CD may not be within their VPN which deems this solution impossible.

On the other hand, if their internal GOPROXY was reachable from their CI/CD but required authentication, then they must make sure to configure GONOSUMDB correctly as well as programmatically creating a .netrc file in their CI/CD so that their private dependencies can be properly downloaded.

3. For deployment: if you are using an ecosystem like Google Cloud Functions w/ private module dependencies, the environment forces you to always vendor your dependencies and always have your function/go.mod file be at the root level.

Problem statement: because you cannot configure a .netrc file when you deploy a Google Cloud Function, GCF forces you to always vendor your dependencies which makes for a less ideal solution. We can definitely argue that this is a GCF issue so I won't focus on this too much but just thought I'd bring it up as another example where Go tooling can enable these platforms to be more friendly.

To summarize here are the pain points from a Go developer's perspective:

1. Figuring out how to set up authentication (whether by configuring git or a .netrc file), if using direct VCS private module fetching.
2. Figuring how to set up authentication (by configuring a GORPXOY + a .netrc file or ensuring you are always behind a VPN), if your company uses an internal GOPROXY.
3. Figuring out whether you need GOPRIVATE or GONOSUMDB, or a combination of both with GONOPROXY.
4. Figuring what to set those env vars to: private path pattern matching.

This brings me to my last point:

My core angle here is to address the user experience. I'd love for the Go team and the community to tackle this issue from a UX perspective.

The only reason this issue addresses the ".netrc" file is mainly because this is what we have today. But the underlying issue is the same: how can we make the user experience for fetching private modules wonderful, pleasant and also secure!

The last 3 companies I have been employed at all used private Go modules and servers spread out over multiple git repositories and all 3 companies have faced the same struggle due to the lack of a good user experience.

To solve the above issues, there are arguably infinite ways to approach this problem. However, here's one basic suggestion that I hope it really shouldn't be a hard sell:

• I'd love to see the Go command take ownership of authenticating with modules. If you see my examples in the issue above: Dart, npm, Ruby, Docker, and most relevant software out there allow you to do basic functionality like "login" and "logout" which abstracts the underlying system that stores and sends credentials. Therefore telling a Go developer to just run a Go command (i.e. "go auth login" or something along those lines) is an order of magnitude a better experience than telling them to to configure various environment variables, .netrc files, and VCS configurations.

If we can achieve the above, we would be in a much better position than we are today.

But if we want to go further and because the design space is quite vast, I think we can look into making the experience even better by doing a little more heavy lifting such as the following suggestion:

Maybe the GOPROXY Download Protocol can be extended so that it can declare what private module paths it supports. This way the Go command can know when to consult the sumdb for a particular module coming from a particular GOPROXY.

I think the new Swift Registry Proposal is considering such option where a registry server declares the package paths it supports.

Finally, regarding this:

To the extent that this issue is about maintaining ~/.netrc, it sounds like we should encourage prototyping a netrc manager outside the Go command.

I started building out a CLI that basically just "edits" the .netrc file. But then I realized this only goes as far as making sure the user doesn't make syntactical-netrc errors. What if that same tool can actually do more:

1. Validate that the given authentication is actually correct by consulting the source of truth (a central API or the GOPROXY itself) to see if the token is valid.
2. Have the response not only be a success, but also return the GOPROXY configuration so that the tool can adjust not only the .netrc file, but GONOSUMDB and GOPRIVATE.

I'm currently building a tool and an API that does exactly the above by consulting/authenticating with a central API which will return the GOPROXY URL as well as the GONOSUMDB configuration so that the user would not have to worry about those details and they can just run gofig login --token=<token> and then the CLI will do the heavy lifting of configuring the .netrc file and adding the private paths to GONOSUMDB and removing them from GOPRIVATE so that they don't get skipped when Go tries to fetch matched modules.

This is still early days and the API is not open source/available yet but I hope it shows some ideas in how we can make private modules a much better experience for Go developers.

Apologies for the long winded response but I wanted to make sure I addressed everything from the last 2.5 years of working with private Go modules and I'd be very happy to talk to anyone directly because writing all of the above in a GitHub issue may not be the best medium.

[ohir]

@marwan-at-work Ping!
Will you take a look at #45611, please. Would it suffice in your use-cases?
TiA.

[marwan-at-work]

Will comment there, thanks for the ping.

[rsc]

It's clear we need an answer for the underlying problem that both this issue and #45611 are trying to solve, but neither really solves it completely. We don't have a coherent plan right now. This is something the go command team hopes to look at in the next cycle. Putting on hold until then.

[rsc]

Placed on hold.
— rsc for the proposal review group

[seankhliao]

I believe this can be declined in favor of #26232