Debian 9.6 AMD64: Segfault on start at os/user.Current (1.7.0+)

[hendrikbl]

• Gitea version (or commit ref): 1.6.4
• Git version: 2.11.0
• Operating system: Debian 9.6
• Database (use [x]):
  • PostgreSQL
  • MySQL (MariaDB)
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

After upgrading from gitea version 1.6.4 to 1.7.0 (replacing gitea binary with new one) I'm unable to start gitea. The gitea service fails with the following message (service gitea status):

● gitea.service - Gitea (Git with a cup of tea)
   Loaded: loaded (/etc/systemd/system/gitea.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2019-05-03 12:27:24 CEST; 1s ago
  Process: 31656 ExecStart=/usr/local/bin/gitea web -c /etc/gitea/app.ini (code=exited, status=2)
 Main PID: 31656 (code=exited, status=2)

If I try to run it directly via gitea web I get the following:
Gist

More logs:

Mai 03 12:18:43 gitea systemd[1]: Started Gitea (Git with a cup of tea).
Mai 03 12:18:43 gitea gitea[29491]: fatal error: unexpected signal during runtime execution
Mai 03 12:18:43 gitea gitea[29491]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]
Mai 03 12:18:43 gitea gitea[29491]: runtime stack:
Mai 03 12:18:43 gitea gitea[29491]: runtime.throw(0x19c3d99, 0x2a)
Mai 03 12:18:43 gitea systemd[1]: gitea.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mai 03 12:18:43 gitea systemd[1]: gitea.service: Unit entered failed state.
Mai 03 12:18:43 gitea systemd[1]: gitea.service: Failed with result 'exit-code'.

I get this error with every version greater than 1.6.4. All permissions were set the same with every version I tried. I haven't tried building gitea myself since this is something I only want to do if there really is no other way.

[zeripath]

Are you running Gitea on a raspberry pi? If so there is a known problem with the arm7 binary and you need to use arm6.

If not what kernel version do you have? If it's very old you're likely to need to build Gitea yourself as we build on Ubuntu 18.04.

Finally, please don't run 1.7.0 there are a number of security issues in that version and we recommend 1.8 if not at least 1.7.6 (there are fixes in 1.8 that cannot be ported back to 1.7.6)

[hendrikbl]

I'm on a virtual machine running Debian 9.6 x86_64 with kernel version 4.9.0-4-amd64.

I tried version 1.7.0, 1.7.6 and 1.8. They all give me the same error.

[zeripath]

Ok.

Under what user are you trying to run Gitea?
And where? Things don't make sense.

The segfault looks like it's happening when Gitea tries to get the current username. That implies that you're running Gitea as a weird user.

Although you state:

/usr/local/bin/gitea web -c /etc/gitea/app.ini

Gitea seems to be trying get its config:

/usr/local/bin/custom/conf/app.ini

Not in /etc...

What does your service file look like?

Does Gitea run when you just run it as a normal user from a normal directory?

[hendrikbl]

I followed the documentation when first installing gitea. So gitea service is running as git user:

root@gitea:~# cat /etc/passwd | grep git
git:x:109:116:Git Version Control,,,:/home/git:/bin/bash

The binary is located at /usr/local/bin. The working directory is /var/lib/gitea/. As seen in the documentation.

Here's my service file: gitea.service

I tried running gitea as root and git from the corresponding home directory but i still get a segfault.

Although you state:

/usr/local/bin/gitea web -c /etc/gitea/app.ini

Gitea seems to be trying get its config:

/usr/local/bin/custom/conf/app.ini

Not in /etc...

Thats my fault. The log I gave you was from running gitea without the -c argument. Here are the three things I tried:

• User: git, Directory: /usr/local/bin - Gist
• User: git, Directory: /home/git - Gist
• User: root, Directory: home/root - Gist

Does Gitea run when you just run it as a normal user from a normal directory?

So the answer to this question is: No, I can't run gitea anywhere. The strange thing is that version 1.6.4 runs everytime on every circumstance. Everything thats newer than that fails.

Edit: Everything here was tested with version 1.8 of gitea.

[sapk]

This thread on discourse seem related: https://discourse.gitea.io/t/solved-segfault-due-to-misconfigured-unix-user/1104

[zeripath]

Hmm. It looks like this is:

golang/go#24787

Which if I'm reading correctly causes panic on os/user.Current or Lookup for uid <1000.

Now it seems like it's suggested that we need to add more flags to our static build:

golang/go#24787 (comment)

But I bet a simple workaround is to have a uid>=1000. Now you may actually need that in any case because of the SSH requirements - in that if you want to clone over SSH you need to be able to login as that user and run authorized_keys or an authorized keys command which I'm not certain you can do as a uid <1000.

[sapk]

@zeripath It is a pity that we need an other tag (osusergo) like netgo but if it ensure consistency across env we should add it.

[hendrikbl]

I tried running gitea with a new user with a uid>1000:

root@gitea:/usr/local/bin# grep gitea /etc/shadow
gitea:!:18022:0:99999:7:::
root@gitea:/usr/local/bin# grep gitea /etc/passwd
gitea:x:1003:1002:Gitea,,,:/home/gitea:/bin/bash

Still no luck. I get exactly the same. Current version 1.6.4 runs perfectly, everything above that fails.

Can I use the gitea dump command on my current version, create a fresh install of version 1.8 somewhere else and restore the dump? If that's the case I will try that.

[lunny]

@hendrikbl That's dump and restore are not guaranteed.

[zeripath]

@hendrikbl I assume that by a uid > 1000 you haven't marked the user as a system user?

OK, so I wonder how we test this. I've just downloaded the docker debian:latest and that doesn't replicate the issue with 1.8.0

[zeripath]

and I've just tried docker pull debian:9.6 and haven't got this issue either and 1.8 and 1.7.6

[hendrikbl]

@zeripath correct, I just used adduser and followed the setup without setting a password.

Hmm, have you done everything according to the installation guide? Because that's what I did. I'll see if I can try this on a fresh debian install this week. I'm not sure when I have time for that though.

[zeripath]

OK, I mean I can start from the command line - I think getting systemd started on a debian docker is fairly difficult.

[zeripath]

Oh I did add a password though... Nope that doesn't seem to matter

[zeripath]

root@ef9a8c7df397:/home/gitsys# adduser git2
Adding user `git2' ...
Adding new group `git2' (1001) ...
Adding new user `git2' (1001) with group `git2' ...
Creating home directory `/home/git2' ...
Copying files from `/etc/skel' ...
Enter new UNIX password: 
Retype new UNIX password: 
No password supplied
Enter new UNIX password: 
Retype new UNIX password: 
No password supplied
Enter new UNIX password: 
Retype new UNIX password: 
No password supplied
passwd: Authentication token manipulation error
passwd: password unchanged
Try again? [y/N] 
Changing the user information for git2
Enter the new value, or press ENTER for the default
	Full Name []: 
	Room Number []: 
	Work Phone []: 
	Home Phone []: 
	Other []: 
Is the information correct? [Y/n] y
root@ef9a8c7df397:/home/gitsys# sudo -u git2 /gitea-1.7.6-linux-amd64 -c /home/git2/custom/conf/app.ini
2019/05/06 13:43:10 [W] Custom config '/home/git2/custom/conf/app.ini' not found, ignore this if you're running first time
2019/05/06 13:43:10 [T] AppPath: /gitea-1.7.6-linux-amd64
2019/05/06 13:43:10 [T] AppWorkPath: 
2019/05/06 13:43:10 [T] Custom path: custom
2019/05/06 13:43:10 [T] Log path: log
2019/05/06 13:43:10 [I] Gitea v1.7.6 built with go1.12.4 : bindata, sqlite, sqlite_unlock_notify
2019/05/06 13:43:10 [I] Log Mode: Console(Info)
2019/05/06 13:43:10 [I] XORM Log Mode: Console(Info)
2019/05/06 13:43:10 [I] Cache Service Enabled
2019/05/06 13:43:10 [I] Session Service Enabled
2019/05/06 13:43:10 [I] SQLite3 Supported
2019/05/06 13:43:10 [I] Run Mode: Development
2019/05/06 13:43:10 [I] Listen: http://0.0.0.0:3000
2019/05/06 13:43:10 Serving [::]:3000 with pid 5021
^C2019/05/06 13:43:12 Exiting pid 5021.

[zeripath]

Damn if we can't replicate this we can't be sure that osusergo will actually fix it.

[zeripath]

Hmm what's the result of:

dpkg -l libc6

mine is: 2.24-11+deb9u3 and on my ubuntu box it's 2.27-3ubuntu, on unstable it's 2.28 or so.

[zeripath]

So we're fairly certain that #6862 will fix this. We'll get it backported to the 1.8 line but that won't fix the other releases at present.

[hendrikbl]

@zeripath @sapk You're all a bunch of geniuses! The new 1.8 build works perfectly. Thank you for solving this in only one day :)