Additional info: if I manually restore the original remote and branch in the .gitrepo file everything works as expected and I am able to pull from the (canonical) subrepo, after merging the changes pushed to my topic branch in my subrepo fork.

I am thinking a little about what the expected behavior should be. Right now it seems that the .gitrepo file is always updated with new commit info, if you want to update the branch/remote you need to add the -u option.

For my own workflow, I would say that I prefer having the .gitrepo file update at all times, maybe introducing a flag that prevents update. So if I want to push stuff to another branch/repo nothing will happen locally.

What do you think about that?

I may be wrong, I'm definitely new to git subrepo, but I tried 0.3.1 first and it seems to me that in that case the -u option was actually needed to update the .gitrepo info. This is also what I would expect from reading the docs:

--update (-u)
If -b or -r are used, and the command updates the .gitrepo file, include these values to the update.

If I am correct, 0.4.0 would be introducing a BC break wrt 0.3.1 and adding a new flag to skip the update would work but would introduce an inconsistency with the rest of the commands.

Unless, obviously, the docs need to be updated and all commands are actually working the same way in 0.4.0 (didn't try that), that is always acting as -u was specified. This might be totally fine and a new flag to skip the update would make sense in this case, although it would still be a behavior change wrt the previous minor.

If we want to restore the original behavior, maybe there's a way to specify the -u option by default via the git configuration to address your use case? That way your .gitrepo file would always be updated, but the default behavior would still be compatible with 0.3.1 (and the current docs).

First: @plach79, good comments!

For 0.3.1 the difference is that you don't rely on the information in the .gitrepo, instead you parse the tree to find the information. So in that case you don't have a default behavior of updating the .gitrepo file. It was only done with -u.

In the 0.4.0 subrepo structure is based on the information in the .gitrepo file, parent show when something was last pushed and commit is used as a soft pointer to connect main repo commits with the sub repo commits. So the default behavior is to update .gitrepo when you pull/push.

Question is what actually happens to consistency in the different cases:

1. No -u, no -b/-r, "normal" case, .gitrepo will be updated with latest commit and parent
2. No -u, -b/-r, .gitrepo will be updated with latest commit and parent BUT the changes are actually stored in another repo/branch. So when you push the next time, parent/commit has changed and not everything will be included in the changes.
3. -u, -b/-r As 2 but you will have an up to date version.

@ingydotnet What do you say? If I remember correctly you have been an advocate for having "no additional commit" on push as the default? My personal taste is that I want to be able to have a "push-push" workflow, allowing me to keep on working without any additional operations.

Makes sense, I'll just outline my workflow here to clarify why I'd need not to update the .gitrepo file. Maybe I'm just doing it wrong and there are better ways to leverage 0.4.0 :)

I have a private github setup with two main projects that are sharing a set of libraries. We would like to be able to work on the libraries from both projects, but still be able to keep them in-sync without having to manually copy over changes.

Since the two projects are using the PR workflow we would like to be able to do the same with the libraries subrepo(s). This is how we are currently planning to work:

• Import the library in the project repo:

  git subrepo clone upstream-lib1 lib/lib1 -b master
  

• Fork the library (sub)repo and add a remote in your local project repo:

  git remote add fork-lib1 <fork_url.git>
  

• Pull changes from the canonical library (sub)repo as usual

  git subrepo pull lib/lib1
  

• To work on the library from the main project's development branch:

  git checkout -b topic_branch
  git commit -am "lib1 changes"
  git subrepo push lib/lib1 -r fork-lib1 -b topic_branch
  

• Create a PR for the main project repo and a child PR in the subrepo. The development branch in the project repo still needs to point to the canonical subrepo, not the fork, hence the no-update requirement :)
• Review the subrepo changes first, merge them
• Review the project repo changes afterwards, merge them
• When appropriate a gate keeper would pull changes from the subrepo and start over the process

I understand that a PR workflow is based on this circular approach:
Main repo clones against a subrepo, pushes to the subrepo fork. Then the subrepo fork pushes to the subrepo, so finally the main repo will pull in the accepted changes.

Would it be possible to change that so that you instead have the subrepo fork as your pull/push from the main repo? Then you can have a gatekeeper review the changes in the subrepo fork and work against the subrepo when you want to share them with other projects?

Why? because one of the main features in subrepo is that we need to keep track on where we are relative to the subrepo, using the parent/commit data in .gitrepo will accomplish this. If you use the circular approach I think there will be conflicts along the way. If you try the more linear approach, I think it will be much easier to handle changes. And I can't see that it adds in complexity, you still need to handle actual subrepo changes in it's own fork.

@plach79 , What do you think?

Would it be possible to change that so that you instead have the subrepo fork as your pull/push from the main repo?

I'm not sure I understand your suggestion, sorry, can you provide an example?

Then you can have a gatekeeper review the changes in the subrepo fork and work against the subrepo when you want to share them with other projects?

We would like to be able to work on the libraries from the project repo because these are exposed as plugins and so are hard to test in isolation (only unit testing, no integration testing available).

Example:

• Fork subrepo ask subrepo-fork
• Clone subrepo-fork into your main repo
• pull/push without -r, using the subrepo-fork
• When you want to sync with subrepo, you do this from the subrepo-fork (But as you said, it might be troublesome verifying things without a complete context.)

Another thought if you want to verify the subrepo in the context of main repo:

• Clone subrepo into main repo
• When you push subrepo, create main repo branch and then push it to a new subrepo branch
• Create PR:s, review both subrepo and main repo
• When everything is ok, merge the main repo and then push the changes into subrepo master. Or merge the subrepo first and then clone in the change into your master to skip any conflicts.

Was that any helpful?

Thanks, @grimmySwe, this was helpful!

Suggestion 1 wouldn't work for us because each developer has a different subrepo fork. We could use a remote alias to clone the subrepo and have everyone configure their own fork url with the same remote alias, but this would just reverse the problem. When the GK pulls changes from the upstream repo, .gitrepo would be updated with the upstream subrepo remote. This would be a good option if we could skip the update ;)

Suggestion 2 was exactly the other option I was considering, the only problem with it would be that the discussion around the library issue won't be in the subrepo, but scattered through out the project repos. However this looks more viable to me.

@grimmySwe, just to be sure, we are just waiting for @ingydotnet's feedback on #313 (comment), right?

@grimmySwe, I'm reviewing the release/0.4.0 branch and just looking through issues.

Looking briefly into this one and then working with #318 earlier, I am now
aware that a push adds a commit to the parent repo. I wasn't aware of this.

The 0.3.1 release and the master branch don't update gitrepo for a pull, and I
don't want to do it unless it is truly needed.

Can you explain why and when it got added into the release/0.4.0 branch?

In my mind, a parent repo shouldn't care when some other repo was updated.

FWIW, -u is intended to update the .gitrepo file when the branch or remote changes, but it could also be used just to update the commit and parent.

I think it's important to be explicit when you want to update the parent/commit because you could do a subrepo push that updates the .gitrepo file, and then never push that new commit.

In regular git, you can do a push-push workflow. As long as no one else changes a branch you don't need to pull. Every time you push in regular git, the origin reference is moved to a new location to remember the state.

In git subrepo the 0.4.0 design is based on the .gitrepo, we use "commit" to know the latest commit in the subrepo and "parent" as reminder to how much we already pushed. So in a sense this is the reference that moves for each time you push. Unfortunately you can't do this without actually making a local commit with a new .gitrepo. If we don't update the .gitrepo file, we have no track of what is actually done and that will force us to perform a pull before we do our next push. And we can't differentiate our own commits form any other commits so we will have to go through the entire process of recreating the history from the previous .gitrepo information.

The odd use cases are when you want to push things into new branches/remotes. Then you might want to avoid updating the .gitrepo file as it will turn everything into a mess unless you update the entire .gitrepo file. Worst case is that you only update commit/parent but you actually pushed into another branch/remote, then you will end up in trouble when you continue on your previous branch/remote.

So there are maybe three different use cases:

1. Regular work, regular pushes to the same branch/remote
2. Temporary update, one time push to another branch/remote
3. Switching branch, change branch/remote and keep on working there

In both 1 and 3, I see that you update the .gitrepo, for 2 you should not update the .gitrepo.

I think it's important to be explicit when you want to update the parent/commit because you could do a subrepo push that updates the .gitrepo file, and then never push that new commit.

Yes, that is true, but if we don't update the .gitrepo file we don't have any trace on what we have done. There will be a discrepancy between the state of the repo and the subrepo. If I were to clone the repo I would get the latest state of the subrepo but my .gitrepo file would still point at a previous state.

For me it's just a help that I get an extra commit directly, then I just need to push the parent repo. If I don't get that, I would have to pull the subrepo and then push the update .gitrepo.

From my own workflow, I don't want git-subrepo to read the tree and try to find an ancestor
Because my subrepo have commit coming from 2 different git tree (and so ancestor search failed most of time)

So I would like to be able to force it to the use the .gitrepo file as reference for commit.

From my own workflow, I don't want git-subrepo to read the tree and try to find an ancestor
Because my subrepo have commit coming from 2 different git tree (and so ancestor search failed most of time)

So I would like to be able to force it to the use the .gitrepo file as reference for commit.

@FanchTheSystem So you do like the extra commit updating the .gitrepo?

Yes I like it :)

This discussion is definitely above my head, I can only say that if the current behavior is going to be the new normal, there should be a way to opt-out from it. I've ben testing #314 with the workflow described in #313 (comment) and it has been working flawlessly so far.

@plach79 thanks for your input! Even if you think the discussion is above your head, your input is extremely valuable.

It seems as though adding the commit is just a helper for git-subrepo to do
future work more efficiently. It also seems like some people can benfit from
this kind of workflow. It also seems like some people would have their optimal
workflow disrupted.

My stance here is that we make adding this commit something that is opt-in. We
can use the -u flag per push, or set it to automatically update using config,
or set a parent repo policy that it always update by setting a value in the
.gitrepo file.

I have always expected that subrepo would do a pull before a push. I also have
always made subrepo have as little effect on history as possible. Using a tool
like this should not make your history complicated, unless you want it to be.

Adding a commit when no content was changed locally feels bad to me. I wouldn't
want other tools to add commits when I used them. Unless of course I expected
it and wanted it to happen for some reason.

This gets worse in the case of #318, where push -a adds 19 commits for just a
routine housekeeping operation. Anf again, maybe that's what a user wants, but
as a default I think it's undesirable.

I'm assuming here that if we don't have the information, we can still figure it
out from a pull. If not, we could also add an option to specify the commit info
from the last pull manually, when things get hairy.

Thoughts?

In regular git, you can do a push-push workflow. As long as no one else changes a branch you don't need to pull. Every time you push in regular git, the origin reference is moved to a new location to remember the state.

@grimmySwe I'm not entirely certain what you mean by push-push workflow. Whatever it means, git is certainly not adding commits when this info is updated, right? Maybe you want to add a subrepo last push reference here?

Starting with the easy part:

This gets worse in the case of #318, where push -a adds 19 commits for just a
routine housekeeping operation. Anf again, maybe that's what a user wants, but
as a default I think it's undesirable.

I totally agree that 19 commits is way to much, there must be a better way (as the one you suggested in that issue)

Personally I think that the .gitrepo commit created by push saves me time, there will be a commit anyway by the pull that I need to do before the next push, so for me I can't see that there is a real difference. When thinking about it, I actually prefer the push commit as it tells me that I do something, if it was always pull commits, then I don't know if it was something I did or just a regular update.

Comparing it with regular git, I wouldn't want to pull in my own changes each time I push.

My stance here is that we make adding this commit something that is opt-in. We can use the -u flag per push, or set it to automatically update using config, or set a parent repo policy that it always update by setting a value in the .gitrepo file.

With the current design I would think that it's more logical to have the commit created by default and have an opt-out option that you can use in the cases when you don't want it. I understand that it is opposite to previous versions, but I think that it will generate less support.

Personally I think that the .gitrepo commit created by push saves me time,

I respect that and I'm here to help! :)

What I am suggesting is that you can get this behavior in several ways:

• Manually using push -u

• Using git config subrepo.push-update true

  This makes it always happen for you on a particular parent repo

• Using git config -g subrepo.push-update true

  This makes it always happen for you on any repo or subrepo

• Adding update = true to your .gitrepo file

  This makes it update for anyone pushing to that subrepo. It's a parent repo
  "policy".

there will be a commit anyway by the pull that I need to do before the next
push, so for me I can't see that there is a real difference.

That's not actually true. There will be a new commit if something changed in
the subrepo. In that case it's absolutely necessary and expected, because now
it's a change to the parent repo.

There are situations (like subrepo iteself for instance) where the changes to
the subrepo could mostly come from the parent repo. So the pull is rarely going
to add commits.

Comparing it with regular git, I wouldn't want to pull in my own changes each
time I push.

2 things here:

1. I am not suggesting that you actually do a manual git subrepo pull first.
   I am suggesting that the push command tries a pull first. I thought that's what
   it already did (or used to do anyway).

2. (regular) Git is not updating files or adding commits to track its state.
   It's just keeping things in local refs. Subrepo should do the same.

So in one sense, I could come to the sane conclusion that there are never push
commits. But some people like them, so I'm inclined to support them. But
they should definitely be opt-in.

@grimmySwe can you find the commit that started adding the commit on push?

I'll do some more thinking on this and report back later today.

So in one sense, I could come to the sane conclusion that there are never push
commits.

Does it means .gitrepo file may never be updated ?

So If I use it only for push, and never use update.
The internal git operation to compare git tree (merge, branch, and rebase), will be longer and longer after thousand of commit. (And maybe I will have to resolve the same old conflict each time ?)

I may be wrong, I don't know well how git subrepo use the .gitrepo file

The gitrepo file will be updated on a pull or on a push -u and possibly a new update command.

We also might want to detect a conflict resolved by hand and tell people they should do an update or actually prompt them to do an update.

The thing we are missing here is real test situations on public repos. THat's a shame because these situations are easy to recreate even if they happen a year ago. If anyone can point me to public repos and commits that cause various conflicts it would be much appreciated. I'm also going to write some brute force scripts to find some of these today.

Can anyone think of a good way to search github to find public subrepo usage? I just tried but with little success.

I'll report back later today with my findings.

I have a real situation here:
https://github.com/sil-project/Platform
All the src//Bundle/ are subrepo

Our Platform where created without subrepo and commit where done on both side
So before enable subrepo I had to merge both git tree into one branch on each remote repo.
(I have wrote a little how to for my teamate in my native language I may translate it in english and share it if I find some time)

Due to this special situation i have had some conflict, and I was unable to push well with git-subrepo 0.3, using the current 0.4 branch solve my issue, and now I am able to push. I think it is because 0.4
use more the .gitrepo file.

@FanchTheSystem thanks. I'll play around with this.

I was just thinking... If people are having problems with their private repos and they trust me to look at them, you can let me know and I will create a private repo called ingydotnet/<your-github-user-name> and give you access. You can then push the relevant parent and sub repos each as a branch whose name is your repo name. Then we will use this private repo's issues forum to discuss it further, and I'll delete the repo when we are done.

I think we don't need private repo for this. All our work are with GPL or LGPL Licence, you can fork as you like ;)

Also to make good real test, it need 21 repository, one for main repo and one for each subrepo :)

But It is easy for me to create a full copy of our Platform on any github account where I have full access (I have some script for this)

So maybe I can create this as public on my own account and give you access.

(It will do it tomorow, if you confirm you are ok with this)

@ingydotnet So I took a good night sleep and thought about this some more. I can't help it, but I really think that push should update .gitrepo and create a new commit as the real default. Why?

In subrepo 0.4.0 the .gitrepo:commit attribute is used as a virtual reference between commits. So when it creates the internal branch structure it will detect the correct state and make it in simple way. If we leave this out, it will have to redo everything AND most likely you will need to reapply changes causing conflicts along the way. I would compare the .gitrepo file with the "origin/ref".

I do understand that there are cases where you might want to avoid creating this commit but that should only be relevant if you use the -r/-b flags, pushing to another branch/remote. In "all" other cases that I can think of I really want the .gitrepo to reflect the current state, not a previous state.

Having subrepo do pull before each push, would not mitigate this as you will still end up where your .gitrepo file doesn't reflect the current connection between subrepo and parent repo.

I also get back to the regular git comparison, for me I think it's important that subrepo tries to be as close to git as possible. Yes there are some differences, but the standard push/pull workflow should be as similar as possible. Ok, regular git push doesn't create an actual commit BUT it does update your local repo so it is not a read only operation.

Finally I truly believe that it's much easier for a new subrepo user to understand the flow of how things work if the update is there from the start. Yes, there could be ways to disable it, but for me that is super user actions, as afterwards you really need to know what you are doing.

Version 0.2.X used .gitrepo, but for me it had some shortcomings in that it didn't allow you to work push-push and it could potentially lose some changes along the way.
Version 0.3.0 tried to remedy this by comparing the tree hashes and building the virtual structure behind the scene. In that case you didn't need the .gitrepo file, you got the "correct" connections by this and it allowed you to have your states in sync. It though had a lot of other bad features based on the fact that tree hashes wasn't the solid entity that I first thought.
Version 0.4.0 goes back to the .gitrepo, this time allowing me as a user to keep pushing things into the subrepo. It mimics standard git workflow in most cases and behind the scenes it tries to minimize the tricky operations. Yes, it might not have been seriously tested in the field, but so far I have felt much more secure using it for my self.

With all that said, for me personally, I have no problem with an option because I know how it works. My biggest concern is that without having the .gitrepo update automatically on pushes, there will be an endless stream of questions from new subrepo users trying to understand why their pull operations caused conflicts.

Regarding the -r/-b options there should be some more thought there:

1. Currently they are supported for the subrepo pull but they don't really make any sense there. There are most likely no common ancestor to connect the operation with, so you can't calculate what changes you want to apply. It most likely subrepo clone you want to do and if not, the actual merge is best done in the subrepo itself to avoid strange conflicts.
2. With subrepo push you have two cases, first you have the push and forget, where you don't want to update the .gitrepo at all. Secondly you want to push and change tracking. Again if you use -r/-b you don't have any previous relation to that tree so you just push your state as the new one.

Note that you should never only update the remote and branch in the .gitrepo file, then your commit/parent attributes will most likely be invalid.

So I see three use cases:
a. I want to work on another branch/remote, use subrepo clone --force repo -b
b. I want to push my current state to another branch/remote, use subrepo push -r/-b (--no-update)?
c. I want to push my current state to another branch/remote and keep on working there, use subrepo push -r/-b (--update)?

Question is what the default behavior should be, depending on the conclusion of the push .gitrepo commit here, I guess we should align this with that.

@grimmySwe thanks for the input. My executive stance at this point in time is:

• Don't add push commits by default (for 0.4.0)
• Make it dead simple to turn that feature on
• Start making a set of real word tests/stories that document actual workflows
• Leave this idea open until we become certain of the right way

I have a feeling that we can make the default behavior more nuanced; ie have it know when it is appropriate to add a commit, or at least tell the user they should push -u in that case. More research needs to be done.

Adding a push commit by default throws conflicts with my goal that subrepo stays almost completely out of sight except when absolutely necessary, and that's something I don't want to erode.

Hopefully today I'll have time to make some repeatable situations and put them in a place we can discuss.

@grimmySwe 2 questions:

1. What do you mean by push-push? I haven't heard that term before.
2. Why are we requiring git 2.7? I can't seem to find the commit for that.

Cheers.

What do you mean by push-push? I haven't heard that term before.

That is me describing simple work where I only push. No one else is working on the subrepo/branch, so I just implement things and push. For me that is a common use case. At work we have a great CI system so we encourage to push often and get early feedback.

Why are we requiring git 2.7? I can't seem to find the commit for that.

It is in f98012a, reason is that I use worktree listand that comes first in 2.7.

Adding a push commit by default throws conflicts with my goal that subrepo stays almost completely out of sight except when absolutely necessary, and that's something I don't want to erode.

I totally agree that subrepo should stay out of sight if possible. And it would be great if we don't need the push commits. I'll see if I can test some alternatives and see if there is a way around it.

Made some simple tests, first disabling the push commit. If I didn't do anything between the subrepo push and the subrepo pull, it worked without conflict, if I did anything related locally first, before the subrepo pull it often ended up with a conflict. I guess that the reason for that is that the merge behind the scene is a three point merge, tip of subrepo, tip of local subrepo and then the common ancestor. So we don't actually know/care that the tip of the subrepo is actually part of our local history.

@grimmySwe Sorry, I had a big family day yesterday and today is a travel day, but I like where things are going. Let's dig in and keep working through this.

I thought of an idea for this. We can make it the default to add a .gitrepo commit after a push that involves a conflict resolution.

The last step of a conflict resolution is: git subrepo push <subfolder> subrepo/<subfolder>

We can make that form add a commit by default, because that's when it is probably really needed. You can still add the commit with -u or by default by setting a permanent config setting.

Thoughts?

PS Tomorrow I'll spend some time on 0.4.0 release planning. I'd like to spend at least one hour a day on this until we can get it out.

It will prevent some conflict due to the fact you update .gitrepo and that is always good. But the large source of conflicts are what you do after you pushed, so I don't see it as a fool proof solution.

From a debugging perspective it might even be a little harder to have different behaviors. Then it's harder to explain why things happen "sometimes".

I have tried to come up with something to improve the situation, but I have failed so far. The main culprit as I see it, is that the merge is a three point operation, and if you don't use the right three points you end up with a conflict.

Worst thing by not updating the .gitrepo file, is that it is easy to get conflict even if I am the only one working on the subrepo. Why? Because I don't record the changes locally so every time I push things to the subrepo it will be as anyone has made changes.

Hi @admorgan, as asked, here is a use case of this.