subrepo pull with merge method using release/0.4.0 fails to introduce merge markers

[chrisdtaylor]

I have a multirepo and a subrepo where the subrepo content of each has diverged significantly and I want to "sync" the subrepo content between them again. Both sides have made a number of commits since they last were in sync. I have been using the latest 0.3.1 to exchange changes until now, but at this point the rebasing of 0.3.1 is too much - I have over 300 commits I would need to rebase from the multirepo back to the subrepo. So I started playing with various other branches including release/0.4.0 to try the merge approach. Here is the command I ran:

git subrepo pull ext/foo -r path/to/subrepo -b master --verbose --debug --method merge

At the end of that, it tells me the merge failed (no surprise - I expected to need to hand-merge some things), but the instructions don't point me at something I can merge. Here's the output of the tool:

• Merge in changes from refs/subrepo/ext/foot/fetch

git merge refs/subrepo/ext/foo/fetch
The "git merge" command failed:

You will need to finish the pull by hand. A new working tree has been
created at .git/tmp/subrepo/ext/foo so that you can resolve the conflicts
shown in the output above.

This is the common conflict resolution workflow:

1. cd .git/tmp/subrepo/ext/foo
2. Resolve the conflicts (see "git status").
3. "git add" the resolved files.
4. git commit
5. If there are more conflicts, restart at step 2.
6. cd path/to/multirepo
7. git subrepo commit ext/foo
   See "git help merge" for details.

Alternatively, you can abort the pull and reset back to where you started:

1. git subrepo clean ext/foo

See "git help subrepo" for more help.

If I cd to .git/tmp/subrepo/ext/foo and run git status, I have many files with changes and many untracked files (not expected), but none of the files have merge conflict markings. Note also that the output states that there should have been some output after the 'The "git merge" command failed:' line, but there's nothing there. It's like something just went horribly wrong and the tool gave up.

Or maybe I'm missing something? Any ideas or tips are greatly appreciated! Only other thought I had was that the use of 0.3.1 for several "syncs" up to this point might not be compatible with this release/0.4.0 branch with merging instead of rebasing.

[chrisdtaylor]

I've been toying with this some more and tried an experiment where I reset the multi-repo back to an earlier point and only then did the subrepo pull (using release/0.4.0 branch) with merge configuration. I get a sensible merge (resolvable conflicts, merge markers, etc.) in the subrepo worktree.

So this leads me to believe that there is some issue with the scope or complexity of doing a merge after some amount of divergence in both multi-repo and subrepo. I still don't know how to obtain more debug information about why git apparently just gave up and left my worktree in a bad state.

Any ideas?

[chrisdtaylor]

I believe I've root caused the problem. There is a component of the subrepo that uses a special merge tool (setup in a .gitattributes file). The environment I'm using to run git-subrepo does not have the proper setup to be able to call this special mergetool, so the git-merge croaks when there is a conflict in this component. I discovered this while trying to reproduce the error by hand. Running git-merge by hand produced the error message, but the git-merge that is part of git-subrepo-pull did not display this error - it apparently just ate it and left me with a broken worktree.

Might be something to improve in the tool.

[grimmySwe]

@chrisdtaylor Good working tracking this issue down. I'll see if I can find some time next week to look into this. Just be clear on what you are saying:
One of the subrepos has an internal setup specified in .gitattributes, wanting a certain merge tool. When you run this in the "parent" scope it fails to find this.

When you do the subrepo operations you will be in a parent repo scope, so there might be things that you miss. One thing that I know of is if you have git hooks for a subrepo, those wont run when you do the subrepo operations.

[chrisdtaylor]

The issue is actually that our production environment is using an older version of git (pre git 2.5), so it's not compatible with release/0.4.0 because of the use of git worktree. So I'm using my own custom environment setup to run git subrepo as we work on updating the production environment to use git 2.8.4. The unforeseen side effect here is that there's a component of my subrepo where a .gitattributes file specifies a special merge tool. Something like this:

*.foo.bar merge=special_merge_tool

Because I ran things outside the production environment, I didn't have the necessary path to find the special_merge_tool.

When I followed the steps to resolve the merge manually (the steps printed by a failed subrepo merge) I got an error message indicating that special_merge_tool could not be found. To be clear, this happened in the tmp worktree where subrepo/ext/foo was checked out and it was attempting to merge refs/subrepo/ext/foo/fetch.

So I think the subrepo tool just ate the stderr output of git in this case. Would be helpful to pipe that all to the screen, especially if debug switch is on.

[grimmySwe]

@chrisdtaylor Added output if commands fail.

I have encountered this multiple times, don't know why I haven't fixed it before!

[chrisdtaylor]

@grimmySwe I'm still not getting stderr piped all the way to the screen in my case. I was able to confirm that the git-merge failure text I referenced above does go out on stderr when git-merge fails, so somehow git-subrepo is still eating it. I looked briefly at the code below in lib/git-subrepo, but didn't get too far since I'm not great with bash scripting, but I wondered if stderr should be captured to something like $err and printed out next to or after $out.

RUN() {
  $debug_wanted && $SAY && say '>>>' $*
  if $EXEC; then
    "$@"
    return $?
  fi

  OK=true
  set +e
  local rc=
  local out=
  if $debug_wanted && $TTY && interactive; then
    "$@"
  else
    if $OUT; then
      out="$("$@" 2>/dev/null)"
    else
      out="$("$@" 2>&1)"
    fi
  fi
  rc=$?
  set -e

  if [[ $rc -ne 0 ]]; then
    OK=false
    $FAIL && error "Command failed: '$*'.\n$out"
  fi
  output="$out"
}