Some existing and all new PRs for a given repository are broken

[Chippit]

Description

One of our Gitea repositories has found itself in a strange state where:

• New PRs cannot be opened. The task fails with a 500 Internal Server Error when submitting the new PR.
• Some (but not all) pre-existing PRs display the following message (I think these are ones that have had new commits pushed to them since whatever event happened that caused the repository to enter this state.):

This pull request is broken due to missing fork information.

• At least one of the non-broken PRs was still able to be merged successfully despite this state.

Some supplementary information based on a minor investigation I did:

• We can still open and continue working on PRs in other repositories.
• We can still pull, push and clone the broken repository in question.
• Other repositories that have had PRs contain several subdirectories within their refs/pull directory.
• The failing repository contains a refs/pull directory but it is empty.
  • PR Refs still appear in info/refs
  • I have attempted to remove the PR entries and the empty refs/pull directory with no change in behaviour.
• Repositories that have never had a PR do not seem to have a refs/pull.

The attached logs were captured in trace mode and are all entries printed around when a new PR was opened (and the user receives a 500 error)

Gitea Version

1.21.1

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

https://gist.github.com/Chippit/be5a1616958d74a4aa61954bd520e32b

Screenshots

No response

Git Version

git version 2.25.1

Operating System

Ubuntu 20.04.6 LTS

How are you running Gitea?

Our instance is deployed from binary distributions retrieved via github, and is managed via a Systemd unit, on virtualised hardware.

Database

MySQL/MariaDB

[lunny]

Push failed because of some reason. Can you check whether the repository is normal? /var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git

Apr 09 12:20:55 <redacted system hostname> gitea[2767615]: 2024/04/09 12:20:55 ...ervices/pull/pull.go:507:pushToBaseRepoHelper() [E] Unable to push PR head for <redacted>/<redacted project name>#20 (<Repository 60:<redacted>/<redacted project name>>:refs/pull/20/head) due to Error: push failed: exit status 1 - Invalid remote name "/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git": Invalid remote name: "/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git"
Apr 09 12:20:55 <redacted system hostname> gitea[2767615]: error: failed to push some refs to '/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git'
Apr 09 12:20:55 <redacted system hostname> gitea[2767615]:  - Invalid remote name "/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git": Invalid remote name: "/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git"
Apr 09 12:20:55 <redacted system hostname> gitea[2767615]: error: failed to push some refs to '/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git'
Apr 09 12:20:55 <redacted system hostname> gitea[2767615]:  - Invalid remote name "/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git": Invalid remote name: "/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git"
Apr 09 12:20:55 <redacted system hostname> gitea[2767615]: error: failed to push some refs to '/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git'

[Chippit]

Yeah, I don't see anything wrong with it. Pushing and pulling from clients works fine too. All the expected bare repo files are intact at that path, and I can execute stuff like git log from within it and it works as expected. File permissions are all okay, as well.

Is there anything else I should check?

[lunny]

Maybe you can check the permission of refs/pull/*.*.

[Chippit]

Unfortunately, as I mentioned in my description above, refs/pull is completely empty. (Though corresponding items still existed in info/refs)

[lunny]

So maybe you can check the permission of refs/pull Maybe the dirs/files cannot be created under that directory.

[Chippit]

Permissions all seem fine:

$ echo $USER
gitea
$ pwd
/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted-project-name>.git/refs/pull
$ ls -la
total 8
drwxr-xr-x 2 gitea gitea 4096 Apr  9 14:06 .
drwxr-xr-x 5 gitea gitea 4096 Apr  9 13:34 ..
$ touch test
$ ls -la
total 8
drwxr-xr-x 2 gitea gitea 4096 Apr 10 09:07 .
drwxr-xr-x 5 gitea gitea 4096 Apr  9 13:34 ..
-rw-rw-r-- 1 gitea gitea    0 Apr 10 09:07 test

It also seems like this issue has now extended to another repository as well. That one still has all its prior entries in refs/pull but now presents this new issue. Some existing PRs there are broken, and no new ones can be made, with the same error in the log and the same 500 error in the frontend.

[lunny]

How does your Gitea instance run? With docker or systemd?

[Chippit]

As mentioned in the ticket, we are running with systemd.

I think I have solved the problem however.

I tried running the command from the trace log interactively. It fails the same way.

$ git push --dry-run -- /var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git mb-fix-inputfield-nullref:refs/pull/20/head
Invalid remote name "/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git": Invalid remote name: "/var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git"

However, executing it with --no-verify worked fine.

$ git push --no-verify --dry-run -- /var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git mb-fix-inputfield-nullref:refs/pull/20/head
To /var/lib/gitea/data/gitea-repositories/<redacted>/<redacted project name>.git
 * [new branch]      mb-fix-inputfield-nullref -> refs/pull/20/head

I went to check and the failing repos all contained git-lfs hook files. These are not present in other repos.

$ ls
applypatch-msg.sample      post-checkout  post-receive        pre-applypatch.sample    prepare-commit-msg.sample  pre-rebase.sample  pre-receive.sample  update.sample
commit-msg.sample          post-commit    post-receive.d      pre-commit.sample        pre-push                   pre-receive        update
fsmonitor-watchman.sample  post-merge     post-update.sample  pre-merge-commit.sample  pre-push.sample            pre-receive.d      update.d

After removing them, it seems the PRs once again work correctly.

$ mv post-checkout post-checkout.bak
$ mv post-commit post-commit.bak
$ mv post-merge post-merge.bak
$ mv pre-push pre-push.bak

This ticket and this one made me a bit suspicious of the hooks. While I did try reset all the hooks from the admin interface when this issue first came up, as prompted by the first issue, I didn't scrutinize the ones present inside the broken repos particularly. (I'm not that familiar with which ones are expected and which ones aren't so I'm not sure I'd have spotted anything wrong at first glance anyway). But it seems the lfs hooks were causing the pushes to fail.

After doing this, we are once again able to open new ones. And previously broken ones work again after new commits are pushed to their branches.

---

It does raise the question, though: how did these get here? Are they SUPPOSED to be there? We have been using this gitea instance for about a year now with no issues, until yesterday, so something must have triggered this behaviour.

Our system runs the latest available git-lfs from the debian/ubuntu repositories, but it does seem like there is a much newer version of git lfs available via packagecloud repos. Should we install those instead?

$ git lfs --version
git-lfs/2.9.2 (GitHub; linux amd64; go 1.13.5)

[lunny]

From the design of Gitea, those hooks should not be there.

[Chippit]

Okay, well it seems this is a valid resolution then.

While I am concerned about how this happened in the first place, and it would be nice to know how to ensure that it doesn't happen again both for us and for other users, I suppose it's okay to close this issue. Thank you for your help.

[Chippit]

Okay, I have figured out how to reproduce this.

If I use this feature to download an archive of a repository as a zip file, then those hooks are recreated in the repo. Afterwards, this issue recurs.

[lenianiva]

I have a similar problem, but the hooks mentioned above don't exist:

$ ls repositories/<repo-name>/hooks
applypatch-msg.sample	   post-receive.d	  pre-merge-commit.sample    pre-receive	 proc-receive.d		    update.d
commit-msg.sample	   post-update.sample	  prepare-commit-msg.sample  pre-receive.d	 push-to-checkout.sample    update.sample
fsmonitor-watchman.sample  pre-applypatch.sample  pre-push.sample	     pre-receive.sample  sendemail-validate.sample
post-receive		   pre-commit.sample	  pre-rebase.sample	     proc-receive	 update

[lunny]

Maybe fixed by #31931