x/vgo: migrating the Go community.

[marwan-at-work]

I have a specific scenario that makes it hard for me to imagine how the entire Go open source community will have to adapt to vgo.

For those interested to participate in the discussion, please read through the scenario and answer in the context of the scenario so that we don't go off topic. Thanks!

This is my scenario:

I started a new project github.com/marwan-at-work/awesome-go-project.

Let's say my repo has 35 direct dependencies and 350 transitive dependencies.

Say 25 of my direct dependencies use github.com/dgrijalva/jwt-go where it’s currently tagged as v3 but does not have v3 in its import path. Therefore, jwt-go is a transitive dependency and I'm not 100 percent sure which 25 out of 35 direct dependencies use jwt-go.

When I run vgo build for the first time it breaks because of the import path rule. The error says: invalid module: github.com/dgrijalva/jwt-go should be v1, not v3 (v3.1.0)

It's important to note, that I am not trying to upgrade any packages, I am simply running vgo build for the first time.

So what’s the solution here? If I understand correctly, it’s a 3-step solution:

1. Mr or Ms dgrijalva will need to update their repo to include a go.mod in jwt-go’s root directory to specify a new import path that has v3 in it. Not that bad. Hopefully they’re online and around to accept PRs.

2. Every single one of the 25 dependencies that use jwt-go will need to update their source code to use github.com/dgrijalva/jwt-go/v3 (whether manually or through intended tooling). The update itself is not that hard, but having to wait on 25 authors to migrate might take a long while for me to get to step 3:

3. I can finally now run vgo build.

Now inflate this scenario to the entire Go community. How will we solve this?

Will package users like myself have to wait on my direct dependencies to update their source code, which they themselves will have to wait on their direct dependencies to migrate to vgo?

What is the strategy for this particular scenario?

Thanks

[kardianos]

One way this can be worked around in the current prototype is for a packages that have release v1+ is to put v2, v3, vN in subdirectories under master, with a modules file in each vN folder. Then for the next year and a half vgo and go can co-exist. The normal go command fetching packages like normal from the vN sub-directories, and vgo style looking for modules in the sub directories.

Once the world is mostly vgo based, we can use the ability to separate out the physical file path from the version import path.

[ChrisHines]

I don't think putting a copy of the code under a /vN subdirectory solves the problem described in this issue. Using go today we can import v3 of the repo with the import path github.com/dgrijalva/jwt-go. But vgo changes that import path to only refer to v1. No matter what, vgo requires all the dependencies to update their import paths to include /v3 before the main project will build with vgo.

Using /vN subdirectories can help maintain compatibility with non-module aware tools, but it's not straightforward.

If one of the dependencies updates its import paths to github.com/dgrijalva/jwt-go/v3 before jwt-go adds a /v3 subdirectory, then as far as I can tell the main project is unbuildable with any current tool, vgo or go. It's already unbuildable with vgo because the remaining 24 packages using github.com/dgrijalva/jwt-go will incorrectly get the v1 API, and it's unbuildable with go because the one dependency that switched to github.com/dgrijalva/jwt-go/v3 will not find any code there.

Let's suppose that github.com/dgrijalva/jwt-go adds the /v2 and /v3 subdirectories—hopefully getting all the history rewritten and tags preserved correctly. I believe the root directory of the jwt-go repo would still need to contain all of the historical code it always had so that projects that use dep style vendoring and go still build as before for dependents that haven't switched to github.com/dgrijalva/jwt-go/v3 import paths yet.

Maintaining v3 of this package and making future releases then requires duplicating code changes in the root directory and the /v3 directory until we stop supporting non-vgo tools. Maintaining v2 of this package for both vgo and go probably requires duplicating code changes in the root directory on a v2 branch and the /v2 directory on master.

Despite all of the above work to support both go and vgo by github.com/dgrijalva/jwt-go, the main project described above still cannot build with vgo until all transitive dependencies switch to using the github.com/dgrijalva/jwt-go/v3 import path.

[tpng]

What about editing go.mod of github.com/marwan-at-work/awesome-go-project and add jwt-go using pseudo-version v0.0.0-yyyymmddhhmmss-commit?

Say vgo reported that v3.1.0 is incorrect which I assume is the minimal version determined from the direct dependencies, then we will have to find the corresponding commit on jwt-go and vgo get github.com/dgrijalva/jwt-go@dbeaa933, which add jwt-go using pseudo-version.

[marwan-at-work]

@tpng this wouldn't work because jwt-go is a transitive dependency not a direct one.

[tpng]

@marwan-at-work From what I understand in https://research.swtch.com/vgo-tour, downgrading section, you can explicitly require a transitive dependency version.

[kardianos]

@tpgng this is correct. The consuming module gets final say in dependency versions. While transioning many projects will need to specify v0 versions and a hash. There are a few existing bugs in the implementation, but they are are fixable.

[marwan-at-work]

@kardianos @tpng i have made a quick example to clone and run vgo build from https://github.com/marwan-at-work/awesome-go-project where jwt is a transitive dependency. vgo build fails if you try to run it from the repo directory. As of now, what do I have to do to make vgo build succeed?

An important question still remains: If a workaround today does exist, how will we do this on a Go-community-scale? Will it be discouraging?

[tpng]

The problem is that vgo try to convert from other package manager to go.mod without considering the case that some packages tagged with v2+ are not imported using the semantic import path (/v2).

To solve this, I guess vgo can use the pseudo-version in the converted go.mod when encountering these packages. Then no user actions are required when they vgo build their packages.

[tpng]

I have a patched vgo version which use pseudo-version for converted go.mod.
With this version, I can vgo build https://github.com/marwan-at-work/awesome-go-project without doing anything.

[marwan-at-work]

that might be a good migration strategy @tpng -- do you see any downsides from it?

[tpng]

@marwan-at-work If one of your dependencies is using go.mod already and require a v0 or v1 version of jwt-go, then that version will be used instead of the pseudo-version of v3 required from converted package manager.

[fd0]

The issue #24056 is very similar, if not identical.

[marwan-at-work]

@fd0 yes it does. However, I'm asking a more broad question about how vgo should handle the migration of the entire open source community and not just fixing the broken build.

Q: what to do when your transitive dependency has a broken import compatibility.

But maybe even more community related questions like:

• What should we do when a dependency accidentally breaks import compatibility?
• How can tools/editors migrate to use vgo’s deps for autocomplete, imports etc.?

[dlsniper]

What should we do when a dependency accidentally breaks import compatibility?

Tell the author about it, send a PR, and don't use that version.

How can tools/editors migrate to use vgo’s deps for autocomplete, imports etc.?

This is not going to be such a big problem for tooling as the code will still be there, available for usage by the tooling. What will happen is that the tooling will support two modes, legacy and vgo, and it will work accordingly based on what it detects.

[marwan-at-work]

@dlsniper i wouldn't know who the author is because i don't know which of my 25 direct dependencies actually use the transitive dependency so that they can update their import paths that is right after jwt-go updates its go.mod.

[dlsniper]

@dlsniper i wouldn't know who the author is because i don't know which of my 25 direct dependencies actually use the transitive dependency so that they can update their import paths that is right after jwt-go updates its go.mod.

@marwan-at-work I'm not really sure I understand this one. The compiler will break giving you a trace to read and see which package API was broken, just like today. And if anything, having the major version in the import path would make it easier to figure out which version is broken.

Furthermore, there are prototype tools written to explore the space of API compatibility breaks between version bumps, and while they will not be 100% perfect, they'll be close enough that for most cases this won't matter.

Can you please explain where you see this as a problem in the vgo model but not in today's model?

[marwan-at-work]

@dlsniper perhaps it's better to see what im talking about more practically:

1. take a look (or git clone) this project: https://github.com/marwan-at-work/awesome-go-project
2. take a look at main.go, notice that we don't use jwt-go directly.
3. try vgo build and notice the error.
4. Notice how the errors don't tell you which of the five direct dependencies uses jwt-go.
5. Don't think about just the error or the potential solution, think about how vgo is going to migrate the entire community gracefully. Are we really going to dig through dependencies manually and open prs for everyone??
6. In this case, we have to open a PR for jwt-go first (to have v3 be part of its improt path), then open a pr for 3 out of the 5 direct dependencies (guess which ones...) so that they actually import the new path instead of the old broken one. Cool ,but now think about larger projects with tens or hundreds of dependencies and transitive dependencies, how will we untangle everything?

[dlsniper]

I don't think the question I've asked has been answered.

Are we really going to dig through dependencies manually and open prs for everyone??

I would hope that yes, you, me, and others will actually help the people that invest their spare time to make all these free, opensource projects we all use to make money with.

Cool ,but now think about larger projects with tens or hundreds of dependencies and transitive dependencies, how will we untangle everything?

Please show me such projects as I'd love to learn from them. Furthermore, if these projects use libraries that are maintained, I don't see where the problems is. There will be a transition period, there will be a new release in each of these projects, likely containing a major version bump, as they should be, and the development will continue.

As for testing this, the current vgo implementation is a proof of concept, not the final product, as far as I understand from the proposal or the 7 blog posts. I'll use your example in the coming days but, as I mentioned above, how is this different than today?

[marwan-at-work]

I don't think the question I've asked has been answered.

@dlsniper if your question is "how is this different than today" then: in today's world (aka dep) my "awesome-go-project" would build successfully because we don't have the Import Compatibility Rule. Nothing against the Import Compatibility Rule itself, but my concern here is about the migration process for the entire go community which leads me to your following statement:

I would hope that yes, you, me, and others will actually help the people that invest their spare time to make all these free, opensource projects we all use to make money with.

That's well said and is in awesome spirit :) I am quite excited to be active in this process as well. However, my opinion is that this will not be enough.

We should try our best not to have people be frustrated with potentially sloppy upgrades and long-term broken builds.

The Go team, and the active Go community as a whole, should plan a graceful migration process that inflicts the least amount of pain to users when upgrading.

The bold statement above is basically the essence of my concern and I hope this issue is a place to discuss such graceful process :)

Please show me such projects as I'd love to learn from them.

The easiest one to point out is the kubernetes manifest file -- but I bet there are many others in the wild (both open source and closed source).

[rsc]

This thread is duplicating a bit the discussion on the main proposal issue, which now contains:

#24301 (comment)

Proposed change: Define “new” code as code with a go.mod file in the same directory or a parent directory. The old go get must continue to download code exactly as it always has. I propose that the “go build” step adjust its handling of imports in “new” code. Specifically, if an import in new code says x/y/v2/z but x/y/v2/z does not exist and x/y/go.mod says “module x/y/v2”, then go build will read the import as x/y/z instead. We would push this update as a point release for Go 1.9 and Go 1.10.

I think that will address the objections raised in this issue, but I will leave this issue open for further discussion if people want to.

[rsc]

The change suggested in my previous comment was released in the most recent Go point releases (Go 1.10.3 and Go 1.9.7).