x/vgo: migrating the Go community.

[marwan-at-work]

I have a specific scenario that makes it hard for me to imagine how the entire Go open source community will have to adapt to vgo.

For those interested to participate in the discussion, please read through the scenario and answer in the context of the scenario so that we don't go off topic. Thanks!

This is my scenario:

I started a new project github.com/marwan-at-work/awesome-go-project.

Let's say my repo has 35 direct dependencies and 350 transitive dependencies.

Say 25 of my direct dependencies use github.com/dgrijalva/jwt-go where it’s currently tagged as v3 but does not have v3 in its import path. Therefore, jwt-go is a transitive dependency and I'm not 100 percent sure which 25 out of 35 direct dependencies use jwt-go.

When I run vgo build for the first time it breaks because of the import path rule. The error says: invalid module: github.com/dgrijalva/jwt-go should be v1, not v3 (v3.1.0)

It's important to note, that I am not trying to upgrade any packages, I am simply running vgo build for the first time.

So what’s the solution here? If I understand correctly, it’s a 3-step solution:

1. Mr or Ms dgrijalva will need to update their repo to include a go.mod in jwt-go’s root directory to specify a new import path that has v3 in it. Not that bad. Hopefully they’re online and around to accept PRs.

2. Every single one of the 25 dependencies that use jwt-go will need to update their source code to use github.com/dgrijalva/jwt-go/v3 (whether manually or through intended tooling). The update itself is not that hard, but having to wait on 25 authors to migrate might take a long while for me to get to step 3:

3. I can finally now run vgo build.

Now inflate this scenario to the entire Go community. How will we solve this?

Will package users like myself have to wait on my direct dependencies to update their source code, which they themselves will have to wait on their direct dependencies to migrate to vgo?

What is the strategy for this particular scenario?

Thanks

[kardianos]

One way this can be worked around in the current prototype is for a packages that have release v1+ is to put v2, v3, vN in subdirectories under master, with a modules file in each vN folder. Then for the next year and a half vgo and go can co-exist. The normal go command fetching packages like normal from the vN sub-directories, and vgo style looking for modules in the sub directories.

Once the world is mostly vgo based, we can use the ability to separate out the physical file path from the version import path.

[ChrisHines]

I don't think putting a copy of the code under a /vN subdirectory solves the problem described in this issue. Using go today we can import v3 of the repo with the import path github.com/dgrijalva/jwt-go. But vgo changes that import path to only refer to v1. No matter what, vgo requires all the dependencies to update their import paths to include /v3 before the main project will build with vgo.

Using /vN subdirectories can help maintain compatibility with non-module aware tools, but it's not straightforward.

If one of the dependencies updates its import paths to github.com/dgrijalva/jwt-go/v3 before jwt-go adds a /v3 subdirectory, then as far as I can tell the main project is unbuildable with any current tool, vgo or go. It's already unbuildable with vgo because the remaining 24 packages using github.com/dgrijalva/jwt-go will incorrectly get the v1 API, and it's unbuildable with go because the one dependency that switched to github.com/dgrijalva/jwt-go/v3 will not find any code there.

Let's suppose that github.com/dgrijalva/jwt-go adds the /v2 and /v3 subdirectories—hopefully getting all the history rewritten and tags preserved correctly. I believe the root directory of the jwt-go repo would still need to contain all of the historical code it always had so that projects that use dep style vendoring and go still build as before for dependents that haven't switched to github.com/dgrijalva/jwt-go/v3 import paths yet.

Maintaining v3 of this package and making future releases then requires duplicating code changes in the root directory and the /v3 directory until we stop supporting non-vgo tools. Maintaining v2 of this package for both vgo and go probably requires duplicating code changes in the root directory on a v2 branch and the /v2 directory on master.

Despite all of the above work to support both go and vgo by github.com/dgrijalva/jwt-go, the main project described above still cannot build with vgo until all transitive dependencies switch to using the github.com/dgrijalva/jwt-go/v3 import path.

[tpng]

What about editing go.mod of github.com/marwan-at-work/awesome-go-project and add jwt-go using pseudo-version v0.0.0-yyyymmddhhmmss-commit?

Say vgo reported that v3.1.0 is incorrect which I assume is the minimal version determined from the direct dependencies, then we will have to find the corresponding commit on jwt-go and vgo get github.com/dgrijalva/jwt-go@dbeaa933, which add jwt-go using pseudo-version.

[marwan-at-work]

@tpng this wouldn't work because jwt-go is a transitive dependency not a direct one.

[tpng]

@marwan-at-work From what I understand in https://research.swtch.com/vgo-tour, downgrading section, you can explicitly require a transitive dependency version.

[kardianos]

@tpgng this is correct. The consuming module gets final say in dependency versions. While transioning many projects will need to specify v0 versions and a hash. There are a few existing bugs in the implementation, but they are are fixable.

[marwan-at-work]

@kardianos @tpng i have made a quick example to clone and run vgo build from https://github.com/marwan-at-work/awesome-go-project where jwt is a transitive dependency. vgo build fails if you try to run it from the repo directory. As of now, what do I have to do to make vgo build succeed?

An important question still remains: If a workaround today does exist, how will we do this on a Go-community-scale? Will it be discouraging?

[tpng]

The problem is that vgo try to convert from other package manager to go.mod without considering the case that some packages tagged with v2+ are not imported using the semantic import path (/v2).

To solve this, I guess vgo can use the pseudo-version in the converted go.mod when encountering these packages. Then no user actions are required when they vgo build their packages.

[tpng]

I have a patched vgo version which use pseudo-version for converted go.mod.
With this version, I can vgo build https://github.com/marwan-at-work/awesome-go-project without doing anything.

[marwan-at-work]

that might be a good migration strategy @tpng -- do you see any downsides from it?

[tpng]

@marwan-at-work If one of your dependencies is using go.mod already and require a v0 or v1 version of jwt-go, then that version will be used instead of the pseudo-version of v3 required from converted package manager.

[fd0]

The issue #24056 is very similar, if not identical.

[marwan-at-work]

@fd0 yes it does. However, I'm asking a more broad question about how vgo should handle the migration of the entire open source community and not just fixing the broken build.

Q: what to do when your transitive dependency has a broken import compatibility.

But maybe even more community related questions like:

• What should we do when a dependency accidentally breaks import compatibility?
• How can tools/editors migrate to use vgo’s deps for autocomplete, imports etc.?