[Summary] USER_UID/uid related problems (snap/docker)

[wxiaoguang]

Background

1. Discord: "v1.19.0 installed via snap on Ubuntu server doesn't launch. all is OK with 1.18.5 , but with the last one i got" : setting.go:322:loadRunModeFrom() [F] Expect user 'gitea' but current user is: root.
   • "no i didn't change anything, automatic snap update last night to 1.19, break my gitea. i just rollback to 1.18 and all is ok again without touch my config", the chat link
   • Personally I guess it is likely a misconfiguration problem, but I can't see the details nor get more clues from the reporter at the moment.
2. Docker config: USER_UID=1000 & USER_GID=1000
   • It's unclear what they are used for, or what they affect.
3. Can not verify your SSH key: SSHKeyGenParsePublicKey failed: exit status 255 No user exists for uid 997 #19920
   • Although it was closed, actually it didn't get a fix, it's closed because no time spending on it.
4. The pterodactyl / parkervcp:
   • I used pterodactyl which uses docker with this docker image: ghcr.io/parkervcp/yolks:debian
   • And installs from the config generated from this file: https://github.com/parkervcp/eggs/blob/master/software/gitea/egg-gitea.json
   • Then: AddPublicKey, calcFingerprintSSHKeygen: 'ssh-keygen -lf /tmp/....' failed with error 'exec(64c52017-2:AddPublicKey) failed: exit status 255(<nil>) stdout: stderr: No user exists for uid 999 ': No user exists for uid 999

TODO

I haven't spent time on these problems. Some brief thoughts:

1. If Gitea runs with root, it should be able to setuid/seteuid to switch to the "gitea" user.
   • However, "setuid" correctly in a complex Go program like Gitea is very difficult.
2. Gitea checks the current running user by its name, it's not ideal, because in some rare cases, the UID could be used without an existing user name.
   • Update: eg: Builtin SSH Server & Docker-rootless only needs a virtual SSH_USER.
3. The USER_UID/USER_GID config for docker-root/docker-rootless are unclear, and it might cause problems if something mismatches or changes.

I guess Gitea needs to spend some time on these problems, work them out, clarify the behaviors and improve documents.

[lunny]

2.Gitea checks the current running user by its name, it's not ideal, because in some rare cases, the UID could be used without an existing user name.

For the ssh protocol of git, the run user name is also the name which be used in the SSH URL when install with Open SSHD.

[wxiaoguang]

2.Gitea checks the current running user by its name, it's not ideal, because in some rare cases, the UID could be used without an existing user name.

For the ssh protocol of git, the run user name is also the name which be used in the SSH URL.

That's not true for Builtin SSH Server, eg: docker-rootless. Only SSH_USER is necessary, the RUN_USER shouldn't be mixed there.

---

I updated the description, added this context: "Builtin SSH Server & Docker-rootless only needs a virtual SSH_USER."

[bd4]

I have this problem with gitea/gitea:latest-rootless. No changes to configuration, I have been using user: "1016:1017" in the docker-compose.yml, RUN_USER git in app.ini, and it used to work. Now it no longer works, I get errors like this:

gitea     | 2023/04/26 22:10:29 ...s/setting/setting.go:322:loadRunModeFrom() [F] Expect user 'gitea' but current user is: 

[wxiaoguang]

Could you help to provide more details? What are the operations before the bug occurs?

Or, you you provide detailed reproducible setup steps? Then I can try to reproduce it on my side.

[bd4]

I did not test carefully last time I upgraded, but when I rolled back to gitea/gitea:1.18-rootless, the problem goes away. I have a user/group gitea on host system, with uid/gid 1016/1017, and my docker-compose looks like this:

version: "3"

networks:
  gitea:
    external: false

services:
  server:
    image: gitea/gitea:latest-rootless
    container_name: gitea
    user: "1016:1017"
    restart: unless-stopped
    networks:
      - gitea
    volumes:
      - ./data:/var/lib/gitea
      - ./config:/etc/gitea
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3001:3000"
      - "2222:2222"

Maybe it is enough to just have user: set in docker-compose and upgrade to 1.19? In config/app.ini, I have:

APP_NAME = Gitea: Git with a cup of tea
RUN_USER = git
RUN_MODE = prod

Not sure if RUN_USER has anything to do with it.

[bd4]

To be more clear, doing docker-compose pull followed by docker-compose up -d to upgrade to 1.19 from 1.18 reproduces, and rolling back to 1.18 by changing the image from gitea/gitea:latest-rootless to gitea/gitea:1.18-rootless fixes it.

[bd4]

It is running on Ubuntu 22.04, but no snaps - just have docker and docker-compose installed and manage some docker instances manually.

[bota87]

I have probably the same issue as reported in #24385
I just tried with the latest release 1.19.2-rootless and the issue persist, if I remove user: "1016:1017" it works

[wxiaoguang]

I have a question, why did you set user: "1016:1017" ?

[bota87]

because user 1000 it's already used for another service and I want to keep permissions separated

[wxiaoguang]

I'm not sure whether it's related (haven't looked into the problem at the moment):

Gitea's docker-rootless hard-coded 1000:

gitea/Dockerfile.rootless

Lines 40 to 49 in ad03c6e

	RUN addgroup \
	-S -g 1000 \
	git && \
	adduser \
	-S -H -D \
	-h /var/lib/gitea/git \
	-s /bin/bash \
	-u 1000 \
	-G git \
	git

gitea/Dockerfile.rootless

Line 62 in ad03c6e

USER 1000:1000

[bd4]

My understanding is that the USER from the dockerfile is the default, but it's it can be overridden by the user config in docker compose or appropriate docker run args: https://docs.docker.com/compose/compose-file/05-services/#user

[wxiaoguang]

Yup, but Gitea can't find the user's name for UID=10xx (non-1000), then bug occurs ..... actually it's also related another problem (#24389), the IsRunUserMatchCurrentUser logic is fragile.

[bd4]

Ok so the issue happens when using a host UID that does not exist in the docker image? And it stopped working in 1.19 because the code is trying to determine the user name, and before it wasn't?

[wxiaoguang]

Yes (I guess so)

[wxiaoguang]

Sorry for bothering, one more question, do you have "DISABLE_SSH=false" and "START_SSH_SERVER=true" in your app.ini , aka, have you enabled Gitea's builtin SSH server?

[bd4]

Sorry for bothering, one more question, do you have "DISABLE_SSH=false" and "START_SSH_SERVER=true" in your app.ini , aka, have you enabled Gitea's builtin SSH server?

Yes I have both those set in the config, so is using the builtin server.

[wxiaoguang]

If you could build your own docker image, welcome to try this fix #24435. Or I think it could be merged in one or two days soon then you can try the 1.19 nightly image.

[wxiaoguang]

https://hub.docker.com/r/gitea/gitea/tags?page=1&name=1.19-dev these images contain #24435

[bota87]

1.19-dev-rootless isn't updated and didn't worked for me, 1.19-dev-linux-amd64-rootless works

[wxiaoguang]

1.19-dev-rootless isn't updated and didn't worked for me, 1.19-dev-linux-amd64-rootless works

@techknowlogick

[lonix1]

@wxiaoguang as discussed on discord, here is my analysis of the problem.

Assumptions:

• I followed the "SSHing Shim (with authorized_keys)" docs. Is it possible I made a mistake? Yes, but I spent a lot of time on this so I doubt it.
• gith is user on host and gitc is user in container

Problem 1: uid:gid for gith and gitc differ

• gith will have some random values when created
• gitc is hardcoded to 1000:1000, and that's impossible to change via environment: in the docker-compose.yml (as suggested by the docs, incorrectly)
• So the uid:gid of gith and gitc are diffferent, and that's the core problem
• Note that it's possible to specify uid:gid for gith upon creation, but on debian/ubuntu distros (maybe others too?) the primary user already has 1000:1000 by default, so there's no way to match these values

Problem 2: ssh rules

From the man page for sshd:

~/.ssh/authorized_keys
...The content of the file is not highly sensitive, but the recommended permissions are read/write for the user, and not accessible by others. If this file, the ~/.ssh directory, or the user's home directory are writable by other users, then the file could be modified or replaced by unauthorized users. In this case, sshd will not allow it to be used...

~/.ssh/authorized_keys
...similar

~/.ssh/known_hosts
...similar

So these paths must have proper permissions on the host:

path	owner:group	permissions
/home/gith/	gith:gith	755
/home/gith/.ssh/	gith:gith	700
/home/gith/.ssh/authorized_keys	gith:gith	600

Results

On the host:

• if those paths are correct, then it will be possible to ssh to the host (but the tunnel to the container will fail)
• this can be tested from your local machine: $ ssh -T gith@server; that will connect but the tunnel will fail, which means ssh on the host is correct

In the container:

• if those paths are correct on the host, then they will be incorrect in the container - because the uid:gid of gith differ to those of gitc
• that is why when performing any git clone/whatever ... action, it always fails with

  Permission denied (publickey).
  fatal: Could not read from remote repository.
  Please make sure you have the correct access rights and the repository exists.

• if one changes them to the gitc user's 1000:1000 to fix that problem, then of course you won't be able to ssh to the host (because that is incorrect for the host's gith user).

Recommendations

• I spent a lot of time on this because it was documented and I assumed if there's a problem it's just me; but I have my doubts that this works as documented. I also have a suspicion that the instructions did work, but not for the latest version (was reported in some other issue). I think a simpler alternative should be proposed that does not involve ssh tunnelling.
• Alternative 1: in container, change gitc user's hardcoded uid:gid from 1000:1000 to something else (e.g. 1010:1010). That also isn't perfect but will eliminate 99% of problems out here, because as explained above, 1000:1000 is already used for the default user on the host. If you use 1010:1010, then when creating gith we could specify 1010:1010 for that user and thus match the values, and so eliminate the problem.
• Alternative 2: create a custom Dockerfile and set uid:gid manually for gitc user.
• Alternative 3: do not use ssh tunnelling, rather have a separate port on the host, e.g. port 22 for host's normal sshd process, and port 2222 for container's sshd process (and compose file would have ports: 2222:22).
• Alternative 4: add another IP address to host, and bind container's port 22 to that, so both sshd processes work on port 22
• Alternative 5: I have a strong suspicion that all the "ssh passthrough" stuff can be simplified using a regular ssh tunnel (ssh -L ...) but I don't have time to test that. It could be MUCH simpler than we think, but someone else must investigate that. There are many examples on StackOverflow, so if someone has time, start there.

Whatever you guys decide, one thing that's important to do right way, is put a "warning" box in that section on the docs stating that this is a very advanced scenario and that most users would probably not be able to get it to work. (I think it's impossible, but I may be wrong.) Don't let people waste time on this.

[lonix1]

Another thing, I set ENABLE_SSH_LOG = true to be able to diagnose ssh problems in the container's sshd, but no sshd errors were written to the log file (which is working for other cases). I wonder if that setting is working?

[wxiaoguang]

Another thing, I set ENABLE_SSH_LOG = true to be able to diagnose ssh problems in the container's sshd, but no sshd errors were written to the log file (which is working for other cases). I wonder if that setting is working?

That's another longstanding problem: difficult to debug SSH part with logs. "ENABLE_SSH_LOG" only sends a few (usually the last error log) to the web server, then the error log will be shown in Gitea server's log with prefix "ssh: ". It's not useful in most cases.

[lonix1]

TL;DR

The problem is here - the UID and GID are hardcoded in the Dockerfile.

• They should be based on environment variables; so user can specify them in Dockerfile (ENV), docker-compose.yml (environment:) or via docker cli (docker run -e).
• There should be default values, e.g. 1001 (not to clash with default 1000 uid/gid on most distros)

Most of this issue (and MANY others) are based on this one problem. Please consider it as a major issue - because of this, we cannot use passthrough.

[krokosik]

@lonix1 great work, I also spent lots of time with the documentation, thinking I'm plain stupid. We are currently using Alternative 3 from your recommendation, but would love a simpler solution

[tomholford]

@lonix1 thanks for sharing your findings. I also spent many hours trying to debug this issue. A configurable UID / GID via docker env vars would be really useful

[tomholford]

@wxiaoguang @lonix1 @wkrasnicki I started on a potential solution in #27405. Could use your feedback to see if this direction makes sense, and what other changes need to be made to resolve this issue.

[wxiaoguang]

Thank you for inviting me to review. At the moment I don't have full understanding for this problem. So feel free to propose a complete solution (with explanation and documents), maybe other users&maintainers could help and suggest.

[maykelvink]

I encountered the same issue, but resolved it by linking the passwd file in the Docker Compose configuration. I adjusted the user ID to match the ones on my system. I opted for SSH Container pass through to make use of my yubikey / security key. Everything operates smoothly now.

This is the extracted passwd file from the gitea/gitea:1.20.5-rootless docker image. You can see as mentioned above it's hard coded as 1000:1000 thus can't run correctly if you run it as an user with a different UID even if you run it as a specific user.

root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/mail:/sbin/nologin
news:x:9:13:news:/usr/lib/news:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
man:x:13:15:man:/usr/man:/sbin/nologin
postmaster:x:14:12:postmaster:/var/mail:/sbin/nologin
cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
ftp:x:21:21::/var/lib/ftp:/sbin/nologin
sshd:x:22:22:sshd:/dev/null:/sbin/nologin
at:x:25:25:at:/var/spool/cron/atjobs:/sbin/nologin
squid:x:31:31:Squid:/var/cache/squid:/sbin/nologin
xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin
games:x:35:35:games:/usr/games:/sbin/nologin
cyrus:x:85:12::/usr/cyrus:/sbin/nologin
vpopmail:x:89:89::/var/vpopmail:/sbin/nologin
ntp:x:123:123:NTP:/var/empty:/sbin/nologin
smmsp:x:209:209:smmsp:/var/spool/mqueue:/sbin/nologin
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin
catchlog:x:100:101:catchlog:/:/bin/false
git:x:1000:1000:Linux User,,,:/data/git:/bin/bash

When starting your docker container you can overwrite this file with using an bind mapping and store your custom passwd file next to your docker-compose.yml. I've updated the passwd to reflect my local git UID and GID

version: "2"

services:
  gitea:
    image: gitea/gitea:1.20.5-rootless
    environment:
      - GITEA__database__DB_TYPE=postgres
      - GITEA__database__HOST=db:5432
      - GITEA__database__NAME=gitea
      - GITEA__database__USER=gitea
      - GITEA__database__PASSWD=gitea
    restart: always
    user: "1001:1003" #git
    volumes:
      - /home/git/.ssh/:/home/git/.ssh
      - ./data:/var/lib/gitea
      - ./config:/etc/gitea
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - type: bind
        source: ./passwd
        target: /etc/passwd
    ports:
      - "3000:3000"
      - "2222:2222"
    depends_on:
      - db

  db:
    image: postgres:14
    restart: always
    user: "1001:1003" #git
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=gitea
    volumes:
      - ./postgres:/var/lib/postgresql/data
      - type: bind
        source: ./passwd
        target: /etc/passwd

e.g. i've updated the last line in the passwd file

git:x:1001:1003:Linux User,,,:/data/git:/bin/bash

[tuomotalvitie]

After spending some time wondering about this, I've been looking at the difference between
gitea/docker/root/usr/bin/entrypoint
and
gitea/docker/rootless/usr/local/bin/docker-entrypoint.sh

The first one has sed magic for changing the USER_GID and USER_ID which the second one lacks. If the second one is actually used by rootless, this might have something to do with the issue.

[maykelvink]

Ideally you want to overwrite the UID/GID that's being used to start the Gitea binary as environment arguments. You cannot add arguments to the Dockerfile if you are not going to build it locally. e.g. I'm pulling a pre-build image from dockerhub.

In order to start the binary as a certain user you could add a PUID/PGID as environment variables in docker compose and adjust the dockerfile to start the binary with for example S6 Overlay.

exec s6-setuidgid "$PUID:$PGID" gitea

[jmptbl]

Hello! I was surprised to see this issue persisting for so long. The simplest fix would be to just give this Docker image a less common UID and GID, with the expectation that a random UID+GID would be unlikely to clash with an existing user account on the host system. It seems strange to release a public image that uses UID and GID 1000!

Anyhow, the easiest workaround without building a custom image is to follow @maykelvink /etc/passwd bind mounting method. I combined it with an /etc/group bind mount as well, so that I could change the UID and GID. Like this I've gotten SSH pass-through working with the Docker shell method.