switch to git subtree for typeshed

[ddfisher]

git subtree is vastly nicer than git submodule. Let's use git subtrees instead for including typeshed.

Benefits:

• users won't have to run git submodule init or git submodule update
• generally much nicer to work with
• mypy experimental PRs can easily include typeshed changes before they're included in typeshed proper
• easy to extract work on typeshed made as part of working on mypy

See this blog post and the man page for more info on git subtree.

[JukkaL]

We discussed subtrees before switching to submodules (#882). I'm fine with reopening this as I don't particularly enjoy git submodules, though I've never used subtrees.

Before we can make a decision we should at least document how various common workflows would change, and what new git commands contributors would have to use.

Based on the previous discussion, these seem to be some of the worst things about subtrees:

• Something bad happens if you combine commits to typeshed and mypy (not sure what exactly). Is the commit impossible to merge back to typeshed?
• If we have local typeshed changes in the mypy repo, merging can become hard. Apparently there's a different kind of merge for subtrees. Anybody care to elaborate on how it differs from a regular merge? I'm especially interested if the UI has rough edges.
• The commands for pulling changes from typeshed and pushing changes to typeshed would likely be pretty unintuitive (i.e., people might have to look them up from docs). At least the submodule workflow is somewhat self-documenting now.

I'd love to hear feedback especially from people who have some experience with the current submodule workflow (and maybe even subtrees).

[matthiaskramm]

I've used subtree merges, but not the "new" git subtree command, which seems to be a UI for it. The main difference seems to be that git subtree also has the --squash feature.

After setting this up initially (git subtree add) I think these are the typical tasks for users:

• clone mypy: You don't have to do anything beyond git clone http://github.com/python/mypy (I.e., no git submodule init).
• modify mypy: You can edit files in both mypy/ and mypy/typeshed/. You can even put both into the same commit, even though the blog post David quoted says you shouldn't.
• create a PR for mypy and typeshed: You'll need to push your mypy changes and your typeshed changes. The former you push by doing standard git push, the latter you push by doing git subtree push --prefix=typeshed [email protected]:/your_user/your_typeshed_clone your_pull_request_branch
• update typeshed: You'll need to do git subtree pull --prefix typeshed http://github.com/python/typeshed master --squash. This will (with --squash) create one commit, which, for all practical purposes, is the same as if you had just manually copied all of typeshed into your repository. (So e.g. pickaxe, git log -S, will also pick up typeshed changes, and a mypy PR that contains a typeshed update will display the actual file changes in the latter.)

All in all, subtrees don't sound too horrible. The main thing I don't like is that the configuration of the subtree isn't copied onto your system via git clone. You have to know where typeshed lives and how to set up the proper subtree before you can e.g. do an update.

Unless users have carefully read all the documentation, it'll be typical that mypy gets a PR which contains commits touching both mypy and typeshed. Then, you'll either have to tell users to split the changes, or, if you don't mind a bit of commit-message redundancy, you have to do the corresponding git subtree push [...] yourself, afterwards.

I'm also a bit worried that there's going to be the temptation to "let typeshed float", i.e. not push changes to mypy/typeshed over to github.com/python/typeshed, or not doing so regularly. git submodule forces you into a habit of keeping upstream up to date, but with subtree, things work just fine without you updating upstream.

[ddfisher]

What's the current workflow for making a mypy change that requires typeshed changes? I don't think it's self documenting, particularly. Are you forced to upstream your typeshed changes first?

One downside of the submodule approach is that everyone that develops on mypy has to deal with it, even if they don't touch typeshed.

I agree that the temptation to "let typeshed float" is worrisome. Is there something we could do to prevent that? Is being disciplined enough?

[ddfisher]

Revisiting this (and rereading the subtree workflow that Matthias helpfully outlined!), I'm less convinced that it makes sense to switch to git subtree for managing typeshed. Here's how the tradeoffs look to me now (focusing on the pain points for both approaches):

submodule:

• a bit of a constant pain when developing: I continually have to call git submodule update when changing branches, etc.
• a slightly greater barrier to entry for new developers getting started with mypy/users who just want to install from source

subtree: (I think these basically echo Matthias's earlier concerns)

• harder to ensure that our copy of typeshed doesn't drift from the typeshed repo
• have to educate users about how exactly to contribute to the typeshed repo (and we're likely to get PRs where some of the changes should've been split to typeshed)

On balance, I'm starting to think that we may be better off just sticking with submodules. Maybe the real fix here (for me) is to start using a post-checkout hook that runs git submodule update.

Thoughts?

[ddfisher]

I added git submodule update as a post-checkout and post-merge hook. My life feels better already.