Gitea stopped working after a git push, hook crashed?

[shaoran]

• Git version: 1.11.4
• Operating system: Debian 10
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist: see below

Description

When I made a git push I got this an error when processing the internal gitea hooks:

$ git push
Enumerating objects: 22, done.
Counting objects: 100% (22/22), done.
Delta compression using up to 8 threads
Compressing objects: 100% (15/15), done.
Writing objects: 100% (15/15), 1.80 KiB | 1.80 MiB/s, done.
Total 15 (delta 13), reused 0 (delta 0), pack-reused 0
remote: . Processing 1 references
remote: Gitea: Internal Server Error
remote: Unable to contact gitea: Post http://localhost:3000/api/internal/hook/post-receive/mondas/mondas: read tcp [::1]:36456->[::1]:3000: i/o timeout

I had to press Ctrl+C to stop the push, it hanged. I tried to open the web interface but the page could not be loaded. Doing a curl https://mygiteapage delivered HTML content, but curl https://mygiteapage/user/login?redirect_to= also hanged.

Systemd's journal of the Gitea services was not showing anything unusual, I could see the login attempts that I myself tried

May 27 21:22:55 gitea gitea[468]: [Macaron] 2020-05-27 21:22:55: Completed GET /vendor/assets/font-awesome/css/font-awesome.min.css 200 OK in 312.83µs
May 27 21:23:16 gitea gitea[468]: [Macaron] 2020-05-27 21:23:16: Started GET /user/login?redirect_to= for xx.xx.xx.xx
May 27 21:23:17 gitea gitea[468]: [Macaron] 2020-05-27 21:23:17: Started GET / for xx.xx.xx.xx
May 27 21:25:09 gitea gitea[468]: [Macaron] 2020-05-27 21:25:09: Started GET /user/login?redirect_to= for xx.xx.xx.xx
May 27 21:25:16 gitea gitea[468]: [Macaron] 2020-05-27 21:25:16: Started GET /user/login?redirect_to= for xx.xx.xx.xx
May 27 21:25:34 gitea gitea[468]: [Macaron] 2020-05-27 21:25:34: Started GET /user/login?redirect_to= for xx.xx.xx.xx
May 27 21:25:47 gitea gitea[468]: [Macaron] 2020-05-27 21:25:47: Started GET /user/login?redirect_to= for xx.xx.xx.xx

so I decided to stop the process. The systemctl stop gitea.service took at least one minute to end and then I inspected the journal again, now I got this:

$ journalctl -e -u gitea
...
May 27 21:26:46 gitea systemd[1]: Stopping Gitea (Git with a cup of tea)...
May 27 21:27:47 gitea gitea[468]: [Macaron] 2020-05-27 21:27:47: Completed POST /api/internal/ssh/3/update/1 500 Internal Server Error in 6m49.225738586s
May 27 21:27:47 gitea gitea[468]: panic: sync: negative WaitGroup counter
May 27 21:27:47 gitea gitea[468]: goroutine 496105 [running]:
May 27 21:27:47 gitea gitea[468]: sync.(*WaitGroup).Add(0xc001d49d10, 0xffffffffffffffff)
May 27 21:27:47 gitea gitea[468]:         /usr/local/go/src/sync/waitgroup.go:74 +0x139
May 27 21:27:47 gitea gitea[468]: sync.(*WaitGroup).Done(...)
May 27 21:27:47 gitea gitea[468]:         /usr/local/go/src/sync/waitgroup.go:99
May 27 21:27:47 gitea gitea[468]: code.gitea.io/gitea/modules/graceful.wrappedConn.Close(0x3347d40, 0xc001d2d930, 0xc001d49ce0, 0xc0086fa008, 0x330ace0, 0xc012064e10)
May 27 21:27:47 gitea gitea[468]:         /go/src/code.gitea.io/gitea/modules/graceful/server.go:253 +0x83
May 27 21:27:47 gitea gitea[468]: net/http.(*conn).close(0xc010256000)
May 27 21:27:47 gitea gitea[468]:         /usr/local/go/src/net/http/server.go:1662 +0x42
May 27 21:27:47 gitea gitea[468]: net/http.(*conn).serve.func1(0xc010256000)
May 27 21:27:47 gitea gitea[468]:         /usr/local/go/src/net/http/server.go:1771 +0xbf
May 27 21:27:47 gitea gitea[468]: net/http.(*conn).serve(0xc010256000, 0x3335ae0, 0xc0073aa6c0)
May 27 21:27:47 gitea gitea[468]:         /usr/local/go/src/net/http/server.go:1900 +0xa49
May 27 21:27:47 gitea gitea[468]: created by net/http.(*Server).Serve
May 27 21:27:47 gitea gitea[468]:         /usr/local/go/src/net/http/server.go:2928 +0x384
May 27 21:27:47 gitea systemd[1]: gitea.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 27 21:27:47 gitea systemd[1]: gitea.service: Failed with result 'exit-code'.
May 27 21:27:47 gitea systemd[1]: Stopped Gitea (Git with a cup of tea).

I made a backup of the database and I inspected the sqlite3 file and it was OK, it doesn't seem to be corrupted or anything.

So I restarted Gitea and now is working fine again, I can access the web interface again. I can see that the commits were pushed but the database was not updated (it was the hook which failed). I tested pushing again:

$ git commit --amend
$ git push
....
Writing objects: 100% (8/8), 918 bytes | 918.00 KiB/s, done.
Total 8 (delta 7), reused 0 (delta 0), pack-reused 0
remote: 
remote: Create a new pull request for 'dev_new_scheduler_loop':
remote:   https://mygiteapage/myuser/myproject/compare/master...somebranch
remote: 
remote: . Processing 1 references
remote: Processed 1 references in total
To mygitea:user/project
 + b47d03e73...c2d9f8e79 somebranch -> somebranch (forced update)

It seems to be fine again. So the question is, what happened? Should I worry? Is this something you've already seen?

[zeripath]

We need the logs around the time of:

remote: Gitea: Internal Server Error
remote: Unable to contact gitea: Post http://localhost:3000/api/internal/hook/post-receive/mondas/mondas: read tcp [::1]:36456->[::1]:3000: i/o timeout

Then we can try to find some reason as to why there was a hang - potentially there was a deadlock in SQLite but without some more info as to where it got stuck we're just guessing,

The logs around panic: sync: negative WaitGroup counter are not anything to be worried about - we've put a PR in to prevent these. (In 1.12)

[shaoran]

Hi, is there a way to give you the last 1000 lines of the logs in a more secure & private way? I don't want to put the log in a public settings (don't want to expose private information like IP, users, repositories, etc.).

[zeripath]

You can message me directly on our discord

[shaoran]

How can I send you directly? Is there a way in github to send a private message?

[shaoran]

Hi,I've created an account on discord and tried to send a message to you, but I got the error Your message could not be delivered because you don't share a server with the recipient or you disabled direct messages on your shared server, recipient is only accepting direct messages from friends, or you were blocked by the recipient.. I've never used discord, I don't know what do do.

[zeripath]

https://discord.gg/Gitea

[shaoran]

I've never used discord, so I don't really know what to do there or how to send you privately the log file without everybody else seeing it. I looked at previous post on the discord page and found you, tried to send you a message but got the error mentioned above. Perhaps if you tell me how to do that in discord, like I said, never used it, I'm completely unfamiliar with it :(

[stale]

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

[stale]

This issue has been automatically closed because of inactivity. You can re-open it if needed.