Skip to content

basic checkout parallelism #346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 37 commits into from
Mar 9, 2022
Merged

basic checkout parallelism #346

merged 37 commits into from
Mar 9, 2022

Conversation

Byron
Copy link
Member

@Byron Byron commented Mar 6, 2022 •
edited
Loading

  • propagate real error when accessing objects in checkout, maybe with a box if possible.
  • overwrite-existing support
  • non'static thread support
  • don't forget to hookup parallel tests in makefile
  • 'small' support without threading primitives
  • reason about race conditions on the file system related to collisions and symlinks trying to exploit.

@Byron Byron mentioned this pull request Mar 6, 2022
Open
44 tasks
@Byron Byron force-pushed the checkout-parallelism branch 7 times, most recently from 7624d3d to 7c9bb48 Compare March 7, 2022 13:13
Byron added 20 commits March 8, 2022 08:13
@Byron
switch worktree to thiserror (#301)
bacc654
@Byron
return proper errors during checkout object lookup (#301)
f9beac0
@Byron
mior refactor and notes towards parallelization (#301)
99de1ef
@Byron
a stab at making file writes safer… (#301)
805c0da 
…and maybe there are better ways using NO_FOLLOW.

it's definitely something that wants more testing.
@Byron
try to fix tests on linux (#301)
9f9d36d
@Byron
Allow symlinks to dirs to be returned, too (#301)
d3d7a7c
@Byron
fix case-insensitive tests (#301)
ccd25cb 
It helps to have a local case-insensitive file system to test against
@Byron
prepare for first overwrite test… (#301)
cd6e086 
…and have to figure out how that should work also in conjunction with
something like 'check-path'. For now it seems we can put all the work
to unlink destinations into our write-entry equivalent, allowing
check-path to not mutate anything.
@Byron
delayed symlink creation for windows, but… (#301)
77b053d 
…tests need some work, it doesn't yet run as it should.
@Byron
delayed symlink creation for everyone, but…(#301)
ab5cd3d 
…the path cache now shows failing tests due to fix.
Needs its state handling fixed.
@Byron
Fix dir-cache to properly handle its valiity which fixes test (#301)
52c0058
@Byron
overwrite-existing support with tests (#301)
49d1d34 
We do our best to test all the cases, and tests discovered quite a few
special cases.

Directories need to be removed recursively, but when that is done
it should be quite equivalent to the capabiliites of git itself.
@Byron
better symlink checking on ubuntu (#301)
facad25
@Byron
delete directories recursively on overwrite-existing (#301)
ea561e6
@Byron
See if we can remove symlinks this way on windows (#301)
0bc9489
@Byron
debug mode for windows (#301)
8f3bc5a
@Byron
some more debugging on windows (#301)
0c18443
@Byron
refactor (#301)
9ea1e44
@Byron
try to fix windows once again (#301)
ff95265
@Byron
fix windows test expecations for good (#301)
81bcb8d 
On windows, there are no symlinks, and we change our expectations
accordingly.
@Byron Byron force-pushed the checkout-parallelism branch from 7c9bb48 to 81bcb8d Compare March 8, 2022 00:16
Byron added 16 commits March 8, 2022 11:12
@Byron
refactor (#301)
542f49b
@Byron
fix interrupt::Iter (#301)
0f0d390
@Byron
feat!: interrupt::Iter, rename interrupt::Iter -> `interrupt::Ite…
8945d95 
…rWithError` (#301)
@Byron
add thread-count and chunk-size computation; interrupt capability (#301)
8cbe85d
@Byron
proper handling of interruptions during checkout (#301)
7575a58
@Byron
switch index checkout to chunk-based operation (#301)
e5f6943 
Right now we have a single-threaded optimization that we might as well
keep, and what follows is a multi-threaded one using our standard
parallelization approach which pays a little extra for chunking.
@Byron
parallel and non-parallel tests (#301)
1cd7eb3 
That should validate that the parallel version is implemented
correctly, order shouldn't change nor the outcome on disk.
@Byron
decouple amount of bytes written from progress (#301)
9ecdade
@Byron
basic parallelization, without proper reducer, just so it compiles (#301
5f29c0f 
)
@Byron
conversions from Rc to arc for Handle (#301)
c19331e
@Byron
call chunk processing in threaded processor (#301)
6bfd865
@Byron
a reducer which produces progress reporting each time it feeds (#301)
e83079d
@Byron
run multi-threaded tests as well for worktree (#301)
232832f
@Byron
thanks clippy
07a4094
@Byron
pass thread-limit along to checkout (#301)
07e9081
@Byron
stabilize assertions in parallel mode (#301)
21d6f88
@Byron Byron merged commit 21d6f88 into main Mar 9, 2022
@Byron Byron deleted the checkout-parallelism branch March 12, 2022 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Byron