Skip to content

Conversation

@zeripath
Copy link
Contributor

@zeripath zeripath commented Jan 11, 2019

When uploading files using the GUI, Gitea currently does:

a) A full clone - Including the whole object DB
b) A full branch checkout - every file
c) Copies the uploaded file to this cloned repository
d) Add/stages the changes
e) Commits
f) Pushes to the real repository
g) Cleans up the cloned repository - deleting the whole object db and checkout.

This is extremely inefficient.

This PR takes a different approach and, instead of using the git porcelain commands (git add and the like), uses the plumbing commands to (hopefully) significantly speed up this process. See my comments in #601 (#601 (comment)).

The basics of this code was in #5621, however, there has been significant refactoring with code moved out of models.Repository into a module called uploader. LFS support has been added, with transparent recognition of whether a file should be added to the LFS through checking the git attributes.

Issues

  • Currently my approach is to use the git-cli commands which can/could be migrated to go-git commands later. Some of this code could and probably should be moved into the gitea git subproject.
  • There are inefficiencies in the LFS code which could be improved - for example, when files are uploaded to the server, their sha1dcsum and sha256sum should be created at that point rather than read the whole file again - and the LFS ContentStore will currently regenerate the sha256sum and copy the file again.
  • I don't set ORIGINAL_SSH_COMMAND so the push hooks may not actually run. This is to stay in keeping with the current implementation - however, could be very easily set.
  • It's not possible to update LFS files, although it is possible to create new text files that will become LFS files.

Will fix #5600. Will fix #4778. Will fix #3557. Will fix #5727.

@codecov-io
Copy link

codecov-io commented Jan 11, 2019

Codecov Report

❗ No coverage uploaded for pull request base (master@baffea1). Click here to learn what that means.
The diff coverage is 32.51%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #5702   +/-   ##
=========================================
  Coverage          ?   38.82%           
=========================================
  Files             ?      344           
  Lines             ?    49355           
  Branches          ?        0           
=========================================
  Hits              ?    19163           
  Misses            ?    27428           
  Partials          ?     2764
Impacted Files Coverage Δ
modules/uploader/diff.go 0% <0%> (ø)
modules/uploader/delete.go 0% <0%> (ø)
models/lfs.go 32.72% <0%> (ø)
models/upload.go 0% <0%> (ø)
modules/uploader/upload.go 0% <0%> (ø)
routers/repo/editor.go 28.67% <26.31%> (ø)
modules/uploader/update.go 35.71% <35.71%> (ø)
models/repo_branch.go 58.27% <56%> (ø)
modules/uploader/repo.go 67.04% <67.04%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update baffea1...9c3603b. Read the comment docs.

@bkcsoft bkcsoft added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jan 11, 2019
@zeripath
Copy link
Contributor Author

zeripath commented Jan 11, 2019

As previously described in #5621 this provides around a 10x speed up on the current implementation even for very small repositories.


Speed ups for large repositories should be considerably better.

@lunny lunny added the type/refactoring Existing code has been cleaned up. There should be no new functionality. label Jan 12, 2019
@lunny lunny added this to the 1.8.0 milestone Jan 12, 2019
@zeripath zeripath mentioned this pull request Jan 14, 2019
7 tasks
    This is a change from the previous code but is more in keeping
    with the default behaviour of git.

Signed-off-by: Andrew Thornton <[email protected]>
Signed-off-by: Andrew Thornton <[email protected]>
Signed-off-by: Andrew Thornton <[email protected]>
Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution.
@zeripath zeripath force-pushed the make-upload-not-clone branch from 8686557 to d0d28c9 Compare January 20, 2019 12:36
@zeripath zeripath force-pushed the make-upload-not-clone branch from 800d6c3 to d6ceb0c Compare February 6, 2019 21:03
@lafriks lafriks added type/enhancement An improvement of existing functionality type/changelog Adds the changelog for a new Gitea version labels Feb 7, 2019
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Feb 7, 2019
@zeripath
Copy link
Contributor Author

zeripath commented Feb 8, 2019

@lunny needs your review.

@zeripath
Copy link
Contributor Author

Ping @lunny

@zeripath
Copy link
Contributor Author

Make lg TM work

@lafriks lafriks merged commit 296814e into go-gitea:master Feb 12, 2019
@zeripath zeripath deleted the make-upload-not-clone branch February 12, 2019 19:38
@richmahn
Copy link
Contributor

Any way to retain commit history when a file is renamed in the GUI? Now it deletes the old file, makes a new file. That isn't the behavior of GitHub when you change the name or path of an existing file.

@lafriks
Copy link
Member

lafriks commented Feb 16, 2019

@richmahn please submit issue for this

@go-gitea go-gitea locked and limited conversation to collaborators Nov 24, 2020
@delvh delvh removed the type/changelog Adds the changelog for a new Gitea version label Oct 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/bug type/enhancement An improvement of existing functionality type/refactoring Existing code has been cleaned up. There should be no new functionality.

Projects

None yet

9 participants