Repo indexer is a memory hog

[mvrhov]

• Gitea version (or commit ref): 1.5.0
• Git version: 2.18
• Operating system: Ubuntu Xenial
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

If one enables repo indexer to be able to search for a code the Gitea becomes a real memory hog.

This is the excerpt from the Dashboard with repo indexer on:
This of course goes on and on.. until the OS kills the app due to OOM

Server Uptime    6 minutes, 18 seconds
Current Goroutines    31
Current Memory Usage    1.4GB
Total Memory Allocated    23GB
Memory Obtained    1.8GB
Pointer Lookup Times    10848
Memory Allocations    184233578
Memory Frees    175630887
Current Heap Usage    1.4GB
Heap Memory Obtained    1.7GB
Heap Memory Idle    276MB
Heap Memory In Use    1.4GB
Heap Memory Released    0B
Heap Objects    8602691

The same but with repo indexer turned off:

Server Uptime    6 minutes, 20 seconds
Current Goroutines    26
Current Memory Usage    20MB
Total Memory Allocated    140MB
Memory Obtained    69MB
Pointer Lookup Times    822
Memory Allocations    482121
Memory Frees    298375
Current Heap Usage    20MB
Heap Memory Obtained    62MB
Heap Memory Idle    38MB
Heap Memory In Use    25MB
Heap Memory Released    0B
Heap Objects    183746

There is ~ 50 repos which totals approximately ~500MB in size

[sps-jlwalker]

I've encountered something similar on Windows as well. Gitea crashes once the indexer grows too large.

[lafriks]

Yes that is why it is marked as experimental feature. Once we have finished notification refactoring we should also refactor indexing to allow other source indexing options (googlecode search lib is one option, external elastic instance could be other)

[lunny]

Maybe related with #4450 ?

[mvrhov]

It definitely not related!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[lunny]

I think this has been resolved sometimes. I will close this and please feel free to reopen it.