git LFS and storage type minio: Massive memory usage

[jochenhz]

• Gitea version (or commit ref): 1.14.0-rc2
• Git version: 2.30.2
• Operating system: docker
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Log gist (heap, allocs, gitea.log):
  Logs.zip

Description

This is a follow up issue from the discord general channel.
Running git lfs push with very small LFS files (330kb in size total) and storage type minio causes a massive use of memory of gitea (around 2.2 GB). Pushing another round of small files quickly exceeds 5GB.

I’ve attached a zip containing a docker compose and test data so that you can reproduce the issue easily.
docker.zip

Reproduction steps:

1. Extract docker.zip
2. Adjust IP address for server.root_url and storage.minio.minio_endpoint in config/app.ini
3. Run: docker-compose -f docker-compose.yml up -d
4. Go to localhost and setup gitea
5. Create a new repo + initialize
6. Run: git clone
7. Cd into cloned repo
8. Run: git lfs install
9. Run: git lfs track *.png
10. Copy provided PNG images to repository
11. Run: git add .
12. Run: git lfs status - and verify that LFS files are tracked correctly
13. Run: git commit and push
14. Observe gitea memory usage spikes from 180 mb to around 2.2 GB

[lunny]

This is upstream related. This may happen when give the size as -1 when upload with minio sdk. Could you send an issue to https://github.com/minio/minio-go/issues/new

[jochenhz]

fyi there is some update on the minio-go issue:
minio/minio-go#1473 (comment)

[lunny]

Add PartSize to PutObjectOptions could be another PR.