search indexer broken: ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 29: fork/exec /usr/bin/git: argument list too long

[eveslage]

Description

the search indexer is not working. the gitea.log gets filled with error messages:

2023/12/22 11:27:05 ...dules/git/command.go:291:Run() [D] git.Command.RunDir(/home/git/gitea-repositories/think-cell/code.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= ls-tree --full-tree -l df61ef8421d86acd35e2a2f98341646c03b87a3f -- .gitignore .gitmodules [[[[ followed by a endless list of files, part of the repository]]]]

2023/12/22 11:27:33 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 29: fork/exec /usr/bin/git: argument list too long

Also when we try to search in the repository we receive a 500 error message

Gitea Version

1.21.3 (and earlier)

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.39.2

Operating System

Debian 6.1.67-1

How are you running Gitea?

we run the latest downloadable version.

Database

None

[lunny]

Which index type are you using?

[eveslage]

Does this answer your question? or is there another setting. I think we went for the default here

REPO_INDEXER_ENABLED = true
REPO_INDEXER_PATH = indexers/repos.bleve

[MaxWaldorf]

same here

Search broken... Same settings...

[wxiaoguang]

The problem is that the code tries to add every file into the "git" command argument list, which makes the OS unhappy.

That code is quite old, so it is a longstanding problem, it only occurs if you have a repo which has a lot of files.

[MaxWaldorf]

Any recommendations for large repos for indexing tools?

[wxiaoguang]

I guess no until the bug could be fixed.

[MaxWaldorf]

So, no 3rd party integrations like elastic?

[lunny]

Elastic search is supported, but the problem is git command line has length limitation but the file names are too much.

[schmurtzm]

Looking forward the fix on this issue. Any workaround ?
My repo has numerous files indeed.
My search doesn't seems to get error 500 (not sure that it stills indexing however) but I have a lot of these events :

2024/03/20 00:27:02 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 2: fork/exec C:\Program Files\Git\cmd\git.exe: Nom de fichier ou extension trop long.
2024/03/20 00:27:02 ...queue/workergroup.go:105:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
2024/03/20 00:27:06 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 2: fork/exec C:\Program Files\Git\cmd\git.exe: Nom de fichier ou extension trop long.
2024/03/20 00:27:06 ...queue/workergroup.go:105:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
2024/03/20 00:27:09 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 2: fork/exec C:\Program Files\Git\cmd\git.exe: Nom de fichier ou extension trop long.
2024/03/20 00:27:09 ...queue/workergroup.go:105:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
2024/03/20 00:27:12 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 2: fork/exec C:\Program Files\Git\cmd\git.exe: Nom de fichier ou extension trop long.
2024/03/20 00:27:12 ...queue/workergroup.go:105:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds

[lunny]

Looking forward the fix on this issue. Any workaround ? My repo has numerous files indeed. My search doesn't seems to get error 500 (not sure that it stills indexing however) but I have a lot of these events :

2024/03/20 00:27:02 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 2: fork/exec C:\Program Files\Git\cmd\git.exe: Nom de fichier ou extension trop long.
2024/03/20 00:27:02 ...queue/workergroup.go:105:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
2024/03/20 00:27:06 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 2: fork/exec C:\Program Files\Git\cmd\git.exe: Nom de fichier ou extension trop long.
2024/03/20 00:27:06 ...queue/workergroup.go:105:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
2024/03/20 00:27:09 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 2: fork/exec C:\Program Files\Git\cmd\git.exe: Nom de fichier ou extension trop long.
2024/03/20 00:27:09 ...queue/workergroup.go:105:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
2024/03/20 00:27:12 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 2: fork/exec C:\Program Files\Git\cmd\git.exe: Nom de fichier ou extension trop long.
2024/03/20 00:27:12 ...queue/workergroup.go:105:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds

Looks like you have a different error log from this issue.