doc: restructure module documentation

[jayconrod]

Summary

Go module documentation is spread out over several sources. The canonical source is go help, but it's difficult to read in a terminal, and the HTML equivalent (golang.org/cmd/go) is hard to discover and lacks structure and examples. Many developers seem to have difficulty finding solutions to common problems and discovering reference material.

This issue proposes restructuring module documentation into three main areas.

• A specification should be added as one or more documents linked from the References section of golang.org/doc. The specification should be written in the main Go repository, and updates should be linked to the Go release cycle.
• A quick reference should remain in the go command with a brief description of each command and each flag. go help modules and other pages should be shortened, pointing to the specification where appropriate. The quick reference should be mirrored at golang.org/cmd/doc, and updates should be linked to the Go release cycle.
• A series of guide articles should be published to help developers learn how to start using modules and how to solve specific problems with modules. Guide articles should be linked from the Learning Go section of golang.org/doc. Guide articles may be published outside the Go release cycle and should be kept up to date with current features and best practices.

Specification

A complete reference for everything related to modules should be added as one or more structured documents linked from golang.org/doc under the References section. This is analogous to the Language specification or The Go memory model.

The specification should be written as HTML documents within the main Go repository. Updates on the website will be tied to the Go release cycle. This is desirable since things like the go.mod format and the proxy protocol are tied to the current release. The latest version of the documentation will always be visible from tip.golang.org.

The specification should include the following sections. Note that this is simply a rough list of headings to establish scope, not an outline, which will come later.

• Introduction
  • Motivation, problems solved (based on https://research.swtch.com/vgo-intro) - very brief description of what modules are and why they exist.
  • Preliminary module support (from go help modules) - how to enable or disable module aware mode.
  • Defining a module (from go help modules) - overview of using modules, with an example go.mod file and mention of go mod init.
  • The main module and the build list (from go help modules) - defining terminology.
  • Pseudo-versions (from go help modules) - definition and format.
  • Module queries (from go help modules) - list of @ suffixes accepted by go get, go mod download, etc.
  • Upgrading and downgrading with go get - more detailed description of go get flags and use of module queries. go help get will link here.
  • Module compatibility and semantic versioning - explanation of semantic versioning, need for major version suffixes, +incompatible version suffix.
• Mechanics of downloading, verifying, and storing modules
  • Module code layout (from go help modules) - how modules are extracted from vcs repo. Currently a pointer to https://research.swtch.com/vgo-module
  • Module download and verification (from go help modules) - description of how modules are downloaded and verified against go.sum.
  • Module replace semantics
  • Modules and vendoring (from go help modules) - go mod vendor and -mod=vendor. We will need to define vendoring without relying on prior knowledge or GOPATH documentation.
  • Minimal version selection (based on https://research.swtch.com/vgo-mvs) - how versions are chosen
  • Module authentication - go.sum and checksum database.
  • Private modules (from go help module-private) - hosting and accessing private modules. Currently, this is just about configuring the go command to avoid leaking information about private modules to the proxy and sumdb. We should include GOAUTH (when implemented; see cmd/go: define HTTP authentication extension mechanism #26232) and other mechanisms (.netrc, git config) for connecting to and authenticating with private servers.
• Formats and protocols
  • go.mod format (from go help go.mod)
  • go.sum format
  • GOPROXY protocol (from go help goproxy) - Should describe fallback behavior.
  • GOSUMDB protocol.

Quick reference

A quick reference for the Go command should be preserved in go help and golang.org/cmd/go/, which is generated from the same text. The quick reference should briefly describe each command and each flag. Quick reference pages may link to sections in the specification or occasionally guide articles, but they should not link to the wiki or external sources that may change or disappear.

As with the specification, updates to the quick reference on the website will be tied to the Go release cycle.

The following changes should be made to go help:

• environment - should provide a brief description of GOPROXY family variables, referring to 'go help goproxy' for more details.
• get (module-get) - Should be slightly more concise. Flags should be indented and descriptions should be shortened (like go help build) to make them easier to find quickly. Reference documentation should be linked for details.
• mod download - No change needed.
• mod edit - Flags should be indented (like go help build). Content is already concise.
• mod graph - No change needed.
• mod init - Should define the module path and explain what it should look like. Consider linking to an introductory guide article.
• mod tidy - No change needed.
• mod vendor - Should link to "Modules and vendoring" in reference documentation.
• mod verify - Should link to "Module authentication".
• mod why - Flags should be indented. Content is already concise.
• goproxy - Should provide a brief description of all proxy environment variables. Should link to reference documentation for proxy protocol, sumdb protocol, Module authentication, etc.
• modules - Should provide a concise introduction (~200-500 words) and link to reference documentation.
• modules-auth - Should link to "Module authentication" in reference documentation.
• modules-private - Should link to "Private modules" in reference documentation.

Guide articles

Guide articles should teach developers how to accomplish specific tasks. Each article should focus on a specific topic or small group of related topics. Articles may be more verbose than reference documentation and should include more realistic examples.
Unlike reference documentation, guide articles may be published outside of the Go release cycle. Unlike blog articles, guide articles should be updated after they are published to include new features and best practices. Blog articles may be converted to guide articles after they are published.

Guide articles should be linked from the Learning Go section of golang.org/doc. See the Hosting section below for possible hosting locations.

The following is a non-exhaustive list of articles to be written.

• Using Go Modules - An introduction to modules. Should be copied mostly as-is with dates and versions removed or updated.
• How to Write Go Code (website: Update "How to Write Go Code" with go modules #28215) - Currently is an introduction to GOPATH, aimed at new Go developers. Should be rewritten as an introduction to modules. This has some overlap with Using Go Modules, but the audience is new Go developers, rather than experienced Go developers who are new to modules.
• Publishing Go modules (blog post in progress) - a guide for releasing a new version of a module. Semantic versions, semantic import versioning, tag format for modules not at repository root. How to push new major version.
• Migrating to Go modules - how to migrate from dep and other systems. Discusses migration with go mod init and limitations. Considerations for v2+ modules. Compatibility with GOPATH (minimal module compatibility).
• v2 and beyond (blog post planned) - Migration and publishing considerations for v2+ modules.
• Tools for understanding modules - how to answer common questions using go list [-m] [-json], go mod why [-m], go mod graph.
• Debugging Go Modules - a guide for how to debug module failures. See https://github.com/golang/go/wiki/Modules#faqs--possible-problems for some existing topics.
• Multi module repositories - a guide for managing multi-module repositories. (see https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories)
• Maintaining module requirements - a guide for managing dependencies with modules, loosely based on go help modules. Explains how to accomplish common tasks with go get and go mod tidy in more detail than the above documents.
• How to upgrade to a new major version of a dependency - Should cover any details left out of Migrating to Go modules. Should keep up with tools that assist with this.
• Local modifications to dependencies - walks through creating a short-lived fork of a dependency, fixing an issue, submitting a PR, upgrading and unforking once the PR is merged. May be combined with Maintaining module requirements, depending on level of detail.
• Managing private modules - authentication with private repos, private proxies, -insecure flag.

Hosting

The golang.org website is served using golang.org/x/website/cmd/golangorg. Currently, this binary has some static content built into the content/static package. This includes some HTML, JS, CSS, and templates. Most content is extracted from the Go repository itself. For example, golang.org/doc is doc/docs.html.

The blog is served on a separate site, blog.golang.org using the binary golang.org/x/blog. Content is written in the content directory in golang.org/x/tools/present format.

At the moment, there is no place for non-blog content to be published outside the Go release cycle. We are considering a few locations:

• The doc/articles directory of the main Go repository. x/website would need to serve pages in this directory at master instead of from the latest release.
• A directory within golang.org/x/website itself. Currently, this repository has not content (just templates, CSS, and JS). This would reduce the need to serve pages from multiple versions of the Go repository, but we'd at least need to serve doc/docs.html from tip in order to update links outside the release cycle.
• A separate documentation site, to be determined.

Whereever we end up hosting articles, authors should be able to write in Markdown or a similar convenient format, rather than raw HTML.

[jayconrod]

cc @bcmills @jadekler @rsc @tbpg @thepudds

[mvdan]

@myitcv noticed that go help modules does mention GOPROXY's special off value, but go help goproxy doesn't. I think the latter help page is where one would look first.

[ianlancetaylor]

Is there any reason for this issue to go through the proposal process? It seems like something to be decided by the cmd/go maintainers.

[bcmills]

@ianlancetaylor, mostly I think we'd like feedback on where to put new documentation that isn't tied to the release cycle: in doc/articles in the main go repo, in x/website, or some new documentation repository (such as x/doc)?

I think our preference would be a new repository, so that the main go repo can remain synchronized to the release cycle and x/website can remain focused on mechanism rather than content.

[ianlancetaylor]

OK.

The latter point is related to #29206.

[jayconrod]

Removed Proposal label. We'll want to talk about modifying x/website and maybe creating a new repo (x/doc?) soon. Perhaps that doesn't need to go through the proposal committee though.