Fix status for old-style submodules with commondir #1402

sceptical-coder · 2022-11-03T13:07:45Z

In some setups, old-style submodules (i.e. the ones with .git directory within theirs worktrees) with commondir can be of tremendous help. For example, commondir link can be used to avoid duplication of objects and also to keep branches in sync with multiple copies of the repo's worktree, while keeping the .git directory inside the worktree can be (ab?-)used to exploit the sharing of the same submodule worktree across different projects (this at least works on Windows with submodule directory being a directory junction, but having a junction is not relevant for reproducing the bug described below).

Unfortunately, at the moment, when git status is run in the root repo of such a setup, it gives an output akin to this:

fatal: unable to access '�??\1?/config': Invalid argument
fatal: 'git status --porcelain=2' failed in submodule commonlibs

where �??\1? part of '�??\1?/config' varies from run to run, and commonlibs is the name of submodule's directory.

Currently, when Git discovers old-style submodule , it spawns subprocess to get its status, like this one:

cd commonlibs; unset GIT_PREFIX; GIT_DIR=.git git status --porcelain=2

Unsurprisingly, the following output is also quite unexpected:

fatal: unable to access '`??L&?/config': Invalid argument

The core reason for these is that global repository field for commondir is not being cleared to NULL after being free()'d in repo_set_commondir(), which is precisely what this commit fixes.

Regarding the further details of the case of investigation, this value of struct pointed by the global the_repository pointer is checked for being not-NULL down in the callstack in compatibility layer for MinGW in a function that is called by repo_set_commondir() before the free()'d value gets assigned in its body (i.e. the body of repo_set_commondir()).

Backtrace from the check is:

#0  mingw_open (filename=0x<address-25> ".git/commondir", oflags=0)
    at compat/mingw.c:784
#1  0x<address-27> in strbuf_read_file (sb=0x<address-26>,
    path=0x<address-25> ".git/commondir", hint=0) at strbuf.c:758
#2  0x<address-24> in get_common_dir_noenv (sb=0x<address-23>,
    gitdir=0x<address-22> ".git") at setup.c:313
#3  0x<address-21> in repo_set_commondir (repo=0x<address-19> <the_repo>,
    commondir=0x0) at repository.c:57
#4  0x<address-20> in repo_set_gitdir (repo=0x<address-19> <the_repo>,
    root=0x<address-15> ".git", o=0x<address-18>) at repository.c:76
#5  0x<address-17> in setup_git_env (git_dir=0x<address-15> ".git")
    at environment.c:179
#6  0x<address-16> in set_git_dir_1 (path=0x<address-15> ".git")
    at environment.c:334
#7  0x<address-14> in update_relative_gitdir (name=0x0,
    old_cwd=0x<address-13> "C:/Users/%username%/<root-repo-name>/commonlibs",
    new_cwd=0x<address-11> "C:/Users/%username%/<root-repo-name>/commonlibs", data=0x0) at environment.c:348
#8  0x<address-12> in chdir_notify (
    new_cwd=0x<address-11> "C:/Users/%username%/<root-repo-name>/commonlibs") at chdir-notify.c:72
#9  0x<address-10> in setup_work_tree () at setup.c:428
#10 0x<address-9> in run_builtin (p=0x<address-8> <commands+2856>,
    argc=2, argv=0x<address-2>) at git.c:458
#11 0x<address-7> in handle_builtin (argc=2, argv=0x<address-2>)
    at git.c:721
#12 0x<address-6> in run_argv (argcp=0x<address-5>, argv=0x<address-4>)
    at git.c:788
#13 0x<address-3> in cmd_main (argc=2, argv=0x<address-2>) at git.c:921
#14 0x<address-1> in main (argc=6, argv=0x<address-0>)
    at common-main.c:56

Backtrace from the death is:

#0  die_errno (fmt=0x<address-42> <result_type+2002> "unable to access '%s'")
    at usage.c:210
#1  0x<address-41> in access_or_die (
    path=0x<address-40> "`\001\r��\004/config", mode=4, flag=0)
    at wrapper.c:667
#2  0x<address-39> in do_git_config_sequence (opts=0x<address-35>,
    fn=0x<address-37> <git_config_include>, data=0x<address-36>)
    at config.c:2142
#3  0x<address-38> in config_with_options (
    fn=0x<address-37> <git_config_include>, data=0x<address-36>,
    config_source=0x0, opts=0x<address-35>) at config.c:2198
#4  0x<address-34> in repo_read_config (repo=0x<address-19> <the_repo>)
    at config.c:2524
#5  0x<address-33> in git_config_check_init (
    repo=0x<address-19> <the_repo>) at config.c:2543
#6  0x<address-32> in repo_config_get_bool (
    repo=0x<address-19> <the_repo>,
    key=0x<address-30> <pad+3116> "windows.appendatomically",
    dest=0x<address-29> <append_atomically>) at config.c:2612
#7  0x<address-31> in git_config_get_bool (
    key=0x<address-30> <pad+3116> "windows.appendatomically",
    dest=0x<address-29> <append_atomically>) at config.c:2714
#8  0x<address-28> in mingw_open (
    filename=0x<address-25> ".git/commondir", oflags=0) at compat/mingw.c:785
#9  0x<address-27> in strbuf_read_file (sb=0x<address-26>,
    path=0x<address-25> ".git/commondir", hint=0) at strbuf.c:758
#10 0x<address-24> in get_common_dir_noenv (sb=0x<address-23>,
    gitdir=0x<address-22> ".git") at setup.c:313
#11 0x<address-21> in repo_set_commondir (repo=0x<address-19> <the_repo>,
    commondir=0x0) at repository.c:57
#12 0x<address-20> in repo_set_gitdir (repo=0x<address-19> <the_repo>,
    root=0x<address-15> ".git", o=0x<address-18>) at repository.c:76
#13 0x<address-17> in setup_git_env (git_dir=0x<address-15> ".git")
    at environment.c:179
#14 0x<address-16> in set_git_dir_1 (path=0x<address-15> ".git")
    at environment.c:334
#15 0x<address-14> in update_relative_gitdir (name=0x0,
    old_cwd=0x<address-13> "C:/Users/%username%/<root-repo-name>/commonlibs",
    new_cwd=0x<address-11> "C:/Users/%username%/<root-repo-name>/commonlibs", data=0x0) at environment.c:348
#16 0x<address-12> in chdir_notify (
    new_cwd=0x<address-11> "C:/Users/%username%/<root-repo-name>/commonlibs") at chdir-notify.c:72
#17 0x<address-10> in setup_work_tree () at setup.c:428
#18 0x<address-9> in run_builtin (p=0x<address-8> <commands+2856>,
    argc=2, argv=0x<address-2>) at git.c:458
#19 0x<address-7> in handle_builtin (argc=2, argv=0x<address-2>)
    at git.c:721
#20 0x<address-6> in run_argv (argcp=0x<address-5>, argv=0x<address-4>)
    at git.c:788
#21 0x<address-3> in cmd_main (argc=2, argv=0x<address-2>) at git.c:921
#22 0x<address-1> in main (argc=6, argv=0x<address-0>)
    at common-main.c:56

Signed-off-by: Andrey Zabavnikov [email protected]

gitgitgadget · 2022-11-03T13:08:50Z

Welcome to GitGitGadget

Hi @zwerdna, and welcome to GitGitGadget, the GitHub App to send patch series to the Git mailing list from GitHub Pull Requests.

Please make sure that your Pull Request has a good description, as it will be used as cover letter. You can CC potential reviewers by adding a footer to the PR description with the following syntax:

CC: Revi Ewer <[email protected]>, Ill Takalook <[email protected]>

Also, it is a good idea to review the commit messages one last time, as the Git project expects them in a quite specific form:

the lines should not exceed 76 columns,
the first line should be like a header and typically start with a prefix like "tests:" or "revisions:" to state which subsystem the change is about, and
the commit messages' body should be describing the "why?" of the change.
Finally, the commit messages should end in a Signed-off-by: line matching the commits' author.

It is in general a good idea to await the automated test ("Checks") in this Pull Request before contributing the patches, e.g. to avoid trivial issues such as unportable code.

Contributing the patches

Before you can contribute the patches, your GitHub username needs to be added to the list of permitted users. Any already-permitted user can do that, by adding a comment to your PR of the form /allow. A good way to find other contributors is to locate recent pull requests where someone has been /allowed:

Search: is:pr is:open "/allow"

Both the person who commented /allow and the PR author are able to /allow you.

An alternative is the channel #git-devel on the Libera Chat IRC network:

<newcontributor> I've just created my first PR, could someone please /allow me? https://github.com/gitgitgadget/git/pull/12345
<veteran> newcontributor: it is done
<newcontributor> thanks!

Once on the list of permitted usernames, you can contribute the patches to the Git mailing list by adding a PR comment /submit.

If you want to see what email(s) would be sent for a /submit request, add a PR comment /preview to have the email(s) sent to you. You must have a public GitHub email address for this. Note that any reviewers CC'd via the list in the PR description will not actually be sent emails.

After you submit, GitGitGadget will respond with another comment that contains the link to the cover letter mail in the Git mailing list archive. Please make sure to monitor the discussion in that thread and to address comments and suggestions (while the comments and suggestions will be mirrored into the PR by GitGitGadget, you will still want to reply via mail).

If you do not want to subscribe to the Git mailing list just to be able to respond to a mail, you can download the mbox from the Git mailing list archive (click the (raw) link), then import it into your mail program. If you use GMail, you can do this via:

curl -g --user "<EMailAddress>:<Password>" \
    --url "imaps://imap.gmail.com/INBOX" -T /path/to/raw.txt

To iterate on your change, i.e. send a revised patch or patch series, you will first want to (force-)push to the same branch. You probably also want to modify your Pull Request description (or title). It is a good idea to summarize the revision by adding something like this to the cover letter (read: by editing the first comment on the PR, i.e. the PR description):

Changes since v1:
- Fixed a typo in the commit message (found by ...)
- Added a code comment to ... as suggested by ...
...

To send a new iteration, just add another PR comment with the contents: /submit.

Need help?

New contributors who want advice are encouraged to join [email protected], where volunteers who regularly contribute to Git are willing to answer newbie questions, give advice, or otherwise provide mentoring to interested contributors. You must join in order to post or view messages, but anyone can join.

You may also be able to find help in real time in the developer IRC channel, #git-devel on Libera Chat. Remember that IRC does not support offline messaging, so if you send someone a private message and log out, they cannot respond to you. The scrollback of #git-devel is archived, though.

gitgitgadget · 2022-11-03T13:08:53Z

There are issues in commit a7a3fe1:
Fix status for old-style submodules with commondir
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget · 2022-11-03T13:12:14Z

There are issues in commit c58cff2:
Fix status for old-style submodules with commondir
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget · 2022-11-03T13:15:26Z

There are issues in commit a3b7143:
Fix status for old-style submodules with commondir
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

dscho · 2022-11-04T08:05:03Z

/allow

gitgitgadget · 2022-11-04T08:06:08Z

User zwerdna is now allowed to use GitGitGadget.

In some setups, old-style submodules (i.e. the ones with .git directory within theirs worktrees) with commondir can be of tremendous help. For example, commondir link can be used to avoid duplication of objects and also to keep branches in sync with multiple copies of the repo's worktree, while keeping the .git directory inside the worktree can be (ab?-)used to exploit the sharing of the same submodule worktree across different projects (this at least works on Windows with submodule directory being a directory junction, but having a junction is not relevant for reproducing the bug described below). Unfortunately, at the moment, when `git status` is run in the root repo of such a setup, it gives an output akin to this: ```sh fatal: unable to access '�??\1?/config': Invalid argument fatal: 'git status --porcelain=2' failed in submodule commonlibs ``` where `�??\1?` part of '�??\1?/config' varies from run to run, and `commonlibs` is the name of submodule's directory. Currently, when Git discovers old-style submodule , it spawns subprocess to get its status, like this one: ```sh cd commonlibs; unset GIT_PREFIX; GIT_DIR=.git git status --porcelain=2 ``` Unsurprisingly, the following output is also quite unexpected: ``` fatal: unable to access '`??L&?/config': Invalid argument ``` The core reason for these is that global repository field for commondir is not being cleared to `NULL` after being `free()`'d in `repo_set_commondir()`, which is precisely what this commit fixes. Regarding the further details of the case of investigation, this value of struct pointed by the global `the_repository` pointer is checked for being not-NULL down in the callstack in compatibility layer for MinGW in a function that is called by `repo_set_commondir()` before the `free()`'d value gets assigned in its body (i.e. the body of `repo_set_commondir()`). Backtrace from the check is: #0 mingw_open (filename=0x<address-25> ".git/commondir", oflags=0) at compat/mingw.c:784 git-for-windows#1 0x<address-27> in strbuf_read_file (sb=0x<address-26>, path=0x<address-25> ".git/commondir", hint=0) at strbuf.c:758 git-for-windows#2 0x<address-24> in get_common_dir_noenv (sb=0x<address-23>, gitdir=0x<address-22> ".git") at setup.c:313 git-for-windows#3 0x<address-21> in repo_set_commondir (repo=0x<address-19> <the_repo>, commondir=0x0) at repository.c:57 git-for-windows#4 0x<address-20> in repo_set_gitdir (repo=0x<address-19> <the_repo>, root=0x<address-15> ".git", o=0x<address-18>) at repository.c:76 git-for-windows#5 0x<address-17> in setup_git_env (git_dir=0x<address-15> ".git") at environment.c:179 git-for-windows#6 0x<address-16> in set_git_dir_1 (path=0x<address-15> ".git") at environment.c:334 git-for-windows#7 0x<address-14> in update_relative_gitdir (name=0x0, old_cwd=0x<address-13> "C:/Users/%username%/<root-repo-name>/commonlibs", new_cwd=0x<address-11> "C:/Users/%username%/<root-repo-name>/commonlibs", data=0x0) at environment.c:348 git-for-windows#8 0x<address-12> in chdir_notify ( new_cwd=0x<address-11> "C:/Users/%username%/<root-repo-name>/commonlibs") at chdir-notify.c:72 git-for-windows#9 0x<address-10> in setup_work_tree () at setup.c:428 git-for-windows#10 0x<address-9> in run_builtin (p=0x<address-8> <commands+2856>, argc=2, argv=0x<address-2>) at git.c:458 git-for-windows#11 0x<address-7> in handle_builtin (argc=2, argv=0x<address-2>) at git.c:721 git-for-windows#12 0x<address-6> in run_argv (argcp=0x<address-5>, argv=0x<address-4>) at git.c:788 git-for-windows#13 0x<address-3> in cmd_main (argc=2, argv=0x<address-2>) at git.c:921 git-for-windows#14 0x<address-1> in main (argc=6, argv=0x<address-0>) at common-main.c:56 Backtrace from the death is: #0 die_errno (fmt=0x<address-42> <result_type+2002> "unable to access '%s'") at usage.c:210 git-for-windows#1 0x<address-41> in access_or_die ( path=0x<address-40> "`\001\r��\004/config", mode=4, flag=0) at wrapper.c:667 git-for-windows#2 0x<address-39> in do_git_config_sequence (opts=0x<address-35>, fn=0x<address-37> <git_config_include>, data=0x<address-36>) at config.c:2142 git-for-windows#3 0x<address-38> in config_with_options ( fn=0x<address-37> <git_config_include>, data=0x<address-36>, config_source=0x0, opts=0x<address-35>) at config.c:2198 git-for-windows#4 0x<address-34> in repo_read_config (repo=0x<address-19> <the_repo>) at config.c:2524 git-for-windows#5 0x<address-33> in git_config_check_init ( repo=0x<address-19> <the_repo>) at config.c:2543 git-for-windows#6 0x<address-32> in repo_config_get_bool ( repo=0x<address-19> <the_repo>, key=0x<address-30> <pad+3116> "windows.appendatomically", dest=0x<address-29> <append_atomically>) at config.c:2612 git-for-windows#7 0x<address-31> in git_config_get_bool ( key=0x<address-30> <pad+3116> "windows.appendatomically", dest=0x<address-29> <append_atomically>) at config.c:2714 git-for-windows#8 0x<address-28> in mingw_open ( filename=0x<address-25> ".git/commondir", oflags=0) at compat/mingw.c:785 git-for-windows#9 0x<address-27> in strbuf_read_file (sb=0x<address-26>, path=0x<address-25> ".git/commondir", hint=0) at strbuf.c:758 git-for-windows#10 0x<address-24> in get_common_dir_noenv (sb=0x<address-23>, gitdir=0x<address-22> ".git") at setup.c:313 git-for-windows#11 0x<address-21> in repo_set_commondir (repo=0x<address-19> <the_repo>, commondir=0x0) at repository.c:57 git-for-windows#12 0x<address-20> in repo_set_gitdir (repo=0x<address-19> <the_repo>, root=0x<address-15> ".git", o=0x<address-18>) at repository.c:76 git-for-windows#13 0x<address-17> in setup_git_env (git_dir=0x<address-15> ".git") at environment.c:179 git-for-windows#14 0x<address-16> in set_git_dir_1 (path=0x<address-15> ".git") at environment.c:334 git-for-windows#15 0x<address-14> in update_relative_gitdir (name=0x0, old_cwd=0x<address-13> "C:/Users/%username%/<root-repo-name>/commonlibs", new_cwd=0x<address-11> "C:/Users/%username%/<root-repo-name>/commonlibs", data=0x0) at environment.c:348 git-for-windows#16 0x<address-12> in chdir_notify ( new_cwd=0x<address-11> "C:/Users/%username%/<root-repo-name>/commonlibs") at chdir-notify.c:72 git-for-windows#17 0x<address-10> in setup_work_tree () at setup.c:428 git-for-windows#18 0x<address-9> in run_builtin (p=0x<address-8> <commands+2856>, argc=2, argv=0x<address-2>) at git.c:458 git-for-windows#19 0x<address-7> in handle_builtin (argc=2, argv=0x<address-2>) at git.c:721 git-for-windows#20 0x<address-6> in run_argv (argcp=0x<address-5>, argv=0x<address-4>) at git.c:788 git-for-windows#21 0x<address-3> in cmd_main (argc=2, argv=0x<address-2>) at git.c:921 git-for-windows#22 0x<address-1> in main (argc=6, argv=0x<address-0>) at common-main.c:56 Signed-off-by: Andrey Zabavnikov <[email protected]>

dscho · 2022-11-04T08:11:47Z

@zwerdna I allowed myself to change the commit message so that it follows Git's conventions. The culprit was this line:

#0  die_errno (fmt=0x<address-42> <result_type+2002> "unable to access '%s'")

It is longer than 76 characters and is not indented. The "trick" was to indent the stack traces ;-)

dscho · 2022-11-04T08:17:03Z

@zwerdna please note that the entire initial PR comment will be sent as "cover letter", but it repeats the commit message. Maybe you can replace it (i.e. edit it) to say something interesting about the background?

You could, for example, reference f9b7573 (repository: free fields before overwriting them, 2017-09-05), explaining how it tried to release memory before overwriting it but then 357a03e (repository.c: move env-related setup code back to environment.c, 2018-03-03) changed the code so that it would not always be overwritten, and you could then add the footers "Cc: Jeff King [email protected]" and "Cc: Nguyễn Thái Ngọc Duy [email protected]".

What do you think?

sceptical-coder · 2022-11-04T17:38:52Z

@zwerdna please note that the entire initial PR comment will be sent as "cover letter", but it repeats the commit message. Maybe you can replace it (i.e. edit it) to say something interesting about the background?

You could, for example, reference f9b7573 (repository: free fields before overwriting them, 2017-09-05), explaining how it tried to release memory before overwriting it but then 357a03e (repository.c: move env-related setup code back to environment.c, 2018-03-03) changed the code so that it would not always be overwritten, and you could then add the footers "Cc: Jeff King [email protected]" and "Cc: Nguyễn Thái Ngọc Duy [email protected]".

What do you think?

Just checked. The bug described is not present in upstream, so I'll close this again (sorry for not describing the reason in the first place).

gitgitgadget bot added the new user label Nov 3, 2022

sceptical-coder force-pushed the upstream-fix-uncleared-commondir branch from a7a3fe1 to c58cff2 Compare November 3, 2022 13:11

sceptical-coder force-pushed the upstream-fix-uncleared-commondir branch from c58cff2 to a3b7143 Compare November 3, 2022 13:14

sceptical-coder closed this Nov 3, 2022

dscho reopened this Nov 4, 2022

dscho force-pushed the upstream-fix-uncleared-commondir branch from a3b7143 to b0f1a41 Compare November 4, 2022 08:09

sceptical-coder closed this Nov 4, 2022

sceptical-coder deleted the upstream-fix-uncleared-commondir branch November 4, 2022 17:39

dscho mentioned this pull request Nov 5, 2022

Fix global repository field not being cleared git-for-windows/git#4083

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix status for old-style submodules with commondir #1402

Fix status for old-style submodules with commondir #1402

Uh oh!

sceptical-coder commented Nov 3, 2022 •

edited

Loading

Uh oh!

gitgitgadget bot commented Nov 3, 2022

Uh oh!

gitgitgadget bot commented Nov 3, 2022

Uh oh!

gitgitgadget bot commented Nov 3, 2022

Uh oh!

gitgitgadget bot commented Nov 3, 2022

Uh oh!

dscho commented Nov 4, 2022

Uh oh!

gitgitgadget bot commented Nov 4, 2022

Uh oh!

dscho commented Nov 4, 2022

Uh oh!

dscho commented Nov 4, 2022

Uh oh!

sceptical-coder commented Nov 4, 2022

Uh oh!

Uh oh!

Fix status for old-style submodules with commondir #1402

Fix status for old-style submodules with commondir #1402

Uh oh!

Conversation

sceptical-coder commented Nov 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gitgitgadget bot commented Nov 3, 2022

Welcome to GitGitGadget

Contributing the patches

Need help?

Uh oh!

gitgitgadget bot commented Nov 3, 2022

Uh oh!

gitgitgadget bot commented Nov 3, 2022

Uh oh!

gitgitgadget bot commented Nov 3, 2022

Uh oh!

dscho commented Nov 4, 2022

Uh oh!

gitgitgadget bot commented Nov 4, 2022

Uh oh!

dscho commented Nov 4, 2022

Uh oh!

dscho commented Nov 4, 2022

Uh oh!

sceptical-coder commented Nov 4, 2022

Uh oh!

Uh oh!

sceptical-coder commented Nov 3, 2022 •

edited

Loading