You should be able to pull with -b/-r

[grimmySwe]

Discussed in #325:
In version 0.4.0 of git-subrepo there is no good way to handle -b/-r flags for the subrepo pull operation.

As there is only one reference in the .gitrepo file, we can't determine the actual tree created by the merge and that would cause either:
a. We are cherry-picking without leaving a trace
b. We pull in old changes again because we have no easy way to determine how far we have pulled.

One possible solution would be to introduce another reference in the .gitrepo file. So that when we do pull -b/-r we actually store that in a separate entry, allowing us to recreate the structure when we push.

[grimmySwe]

A potential problem with using -b/-r is that you want to fetch commits from the remote based on SHA and that is not possible. Main problem is if you don't push the subrepo, clone a new main repo and then try to push. First step would then be to fetch the referenced commit from the .gitrepo file.

[svelez]

Thanks for establishing this issue.

I am a bit confused about the problem statement in #325 and the opening comment though. Is the lack of enough entries in .gitrepo assuming that the git subrepo pull command would not update the tracking branch or is this a general problem whether or not the tracking branch is updated? I thought I noticed the current behavior of pull was to update the tracking branch.

I am thinking that if the tracking branch is updated and future pulls are from that branch, then the current commit and parent entries should be completely adequate for those pulls? Though to be honest, I am not sure how merges in the superrepo are/should be handled.

I also think that if you supply -b again to pull/track the original branch once more the established superrepo history should allow a reasonable merge of that branch's head revision.

[grimmySwe]

@svelez Lets see if I can explain how I see this. Remember that this is my view of it, not the final truth.

In the standard use case (never using -b/-r) you will track one branch in the subrepo. .gitrepo will contain information on the remote and branch to track, it will also contain a commit reference (subrepo.commit) to the latest subrepo commit known in the main repo and finally a commit reference to the latest commit (subrepo.parent) in the main repo that was pushed to the subrepo.

When you pull in new subrepo changes, you update the subrepo.commit. This value is used to easily see if there are any new commits in the subrepo AND it's used to build the virtual structure when we merge in the changes. subrepo.commit will be converted into a parent reference in a commit allowing us to connect the history of our virtual subrepo commits and the actual subrepo commits.

If we add a -b option to pull, we need to decide what we want to do:

1. Don't update subrepo.commit, as we don't actually pull things from the tracking branch. This will allow us to keep business as usual, but there will be no possibility to create a trace of this operation in the subrepo.
2. Update subrepo.commit, this will actually let us create the correct merge operation in future pushes to subrepo BUT if we would perform a pull without -b we would need to retrace our steps and solve possible conflicts.
3. There is a possible third option that updates both subrepo.commit and subrepo.branch, but then we switch tracking branch and if we push, the previous pull operation would be reverse of what you expect.

So my current idea, would be to add a another possible reference in the .gitrepo file. So that we can keep the current tracking information AND also note if we pull in changes from other branches.

Did it make anything clearer?

[grimmySwe]

I find some holes in my design. Most of them are related to a use case where someone uses -b/-r to pull in changes from another repo/branch, pushes these changes to the main repo but NOT to the subrepo. Another user clones the main repo, sees that there are differences to the subrepo and tries to push those changes.

In that case it will need to fetch the correct commits to build up a history so that we can apply the correct ancestry on commits. So if you used pull -r, to pull in changes from a different remote, git-subrepo would need to know about this other remote and that it should fetch relevant commits from it.

I wonder if you actually need to have more data to solve this:

[tracking]
    remote = <tracking remote>
    branch = <tracking branch>
    commit = <last known commit on tracking branch>
    parent = <last commit pushed to subrepo>

[merge]
    remote = <remote used with -r>
    branch = <branch used with -b>
    commit = <parent for this virtual git-subrepo commit>

On the other hand, these operations might be better solved in the actual suprepo. Then you get the full strength of git, not just a subset of operations that is run through git-subrepo.

What would you expect would happen with git subrepo push -b <branch> -r <remote> <subrepo> should the subrepo actually know that we pushed in data from current tracking branch and consider it a merge? Or should it be a force push that simply overwrite the state of with current state?

[grimmySwe]

@ingydotnet Do you have some time thinking about the use cases for pull/push -r/-b combinations? For me it becomes complex operations where you try to handle the subrepo branch structure within another repo.

Git-subrepo already applies one virtual dimension, working with a specific tracking branch/remote and one subrepo. Allowing other branches/remotes to be merged/pushed seems like asking for trouble in the subrepo repository.

The easiest case would be, what is subrepo pull -b <branch> expected to do? Is it simply cherry picking in the latest state from without leaving a trace, or is it actually merging in so that subrepo knows about the operation.

[ingydotnet]

@grimmySwe,

git subrepo -b <branch> -f is effectively a cherry-pick replacement. Without -f it requires some kind of merge logic. I think -b ... -f is cool because you can try different HEADs. Nothing sticks until you push the subrepo.

I just had an idea. Maybe it would be useful if a .gitrepo could contain one or more pairs of commit refs, pairing a local commit to a subrepo commit. I'll think on this and make a new issue for it.

[grimmySwe]

@ingydotnet,

But aren't switching HEADs the subrepo clone job? Where you actually apply a complete state? Pull seems to be more of a merge tool to connect branches.

command/option	description	create local commit with .gitrepo update
clone	Clone state, can only be used if there is no previous directory	yes
-f	overwrite local state	yes
pull	pull new commits from tracking branch, requires HEAD in tracking history, merge local state	yes
-f	ignore previous HEAD requirement	yes
-b/-r	pull commits from other location	yes
-b/-r -u	switch tracking branch	yes
push	push unpushed changes to tracking branch, requires tracking HEAD in history	yes
-f	use force push	yes
-b/-r	push to other location	no
-b/-r -u	switch tracking branch	yes

[casnacaj]

I'm also missing '-f' option for the pull command. The missing HEAD problem may be also caused by the subrepo branch rebase (that may occur on WIP branch in my case). The alternative name for the 'pull [-b/-r] -f' command may be 'checkout'. The behaviour of checkout could be slightly different to pull - I mean 'git subrepo checkout' should ignore previous HEAD requirement but it should fail in case, that main project was changed since last pull/checkout (the changes must be merged by the user). 'git subrepo checkout -f' should also ignore changes in the history (changes will be thrown).

[grimmySwe]

Created a Wiki page where I try to list all different cases and how to handle them:
https://github.com/ingydotnet/git-subrepo/wiki/Command-summary-test-for-issue-328

There is now a branch based on release/0.4.0 called issue/328 where I have implemented support for -r/-b flags with pull and allowing push with -b/-r to create snap shots.

Added some tests and made sure all the old tests works as well. It probably needs some more polishing but it seems to do ok in my simple tests. Main feature is to allow pulling with -b/-r AND keeping the tracking valid. It will use a new merged property in .gitrepo file to track when there are things outside tracking branch introduced. That allows us to create correct merge nodes and don't mess up the subrepo.

So if anyone has some spare time, please feel free to test it out :-)

[casnacaj]

Hi.Thanks a lot for your work!

I'm working on your branch. I have the only problem. The '-f' option allows to pull in case that HEAD changed. But it requires manual merge. See the command output:

git subrepo pull src/bsp -b issue/PLM-26 -u -c -f -s
The "git merge" command failed:

  Auto-merging board/stm32h743-nucleo/cube-common/cube-common.ioc
  CONFLICT (content): Merge conflict in board/stm32h743-nucleo/cube-common/cube-common.ioc
  Auto-merging board/stm32h743-nucleo/cube-common/Src/main.c
  CONFLICT (content): Merge conflict in board/stm32h743-nucleo/cube-common/Src/main.c
  Automatic merge failed; fix conflicts and then commit the result.

You will need to finish the pull by hand. A new working tree has been
created at .git/tmp/subrepo/src/bsp so that you can resolve the conflicts
shown in the output above.

This is the common conflict resolution workflow:

  1. cd .git/tmp/subrepo/src/bsp
  2. Resolve the conflicts (see "git status").
  3. "git add" the resolved files.
  4. git commit
  5. If there are more conflicts, restart at step 2.
  6. cd /c/git/m1-2
  7. git subrepo commit src/bsp
See "git help merge" for details.

Alternatively, you can abort the pull and reset back to where you started:

  1. git subrepo clean src/bsp

See "git help subrepo" for more help.

My use case is:

1. pull main brach
2. make chanes
3. create issue branch
4. push issue branch
5. in the subrepo repository: rebase issue branch to the current main branch
6. pull updated issue branch

So I don't need to merge anything, I just want to make a snapshot of the current issue branch state from the subrepo. The only way which (as far as I know) is git clone, but clone requires to type http again.

[grimmySwe]

@casnacaj If you want to take a snap shot, git subrepo clone --force is the way to go. Otherwise you will need to solve you conflicts by selecting yours/theirs in the merge.

[casnacaj]

@grimmySwe Yes, I know. The problem I have with git subrepo clone is that I have to type url again. I'm missing some command which will allow me to 'reclone' subrepo without the need to retype url.

[grimmySwe]

@casnacaj Can you create a new issue for this? I feel that it would be better to add something to clone then updating pull with another function. Maybe the --force flag could work without additional argument