sparse index support

[SidneyDouw]

Quick recap: What is a sparse index?

Instead of containing one entry for every file in the worktree ("regular" index structure), a sparse index only contains a subset of these. Additionaly, it contains entries to directories that are marked with the SKIP_WORKTREE flag. All files within these entries can be skipped by functions that read / update the index and thereby increase performance.

If the index file contains the the "Sparse Directory Entries" extension marked by the signature sdir, it is classified as a sparse index.

Motivation

The goal of this issue is to keep track of the requirements necessary to eventually fully integrate sparse index support for gitoxide.

This issue does not yet contain all the tasks and considerations by any means, but the goal is to add new knowledge and keep everything up to date as I go along and things become more clear.

Tasks

• reading
  • "regular" index with files containing the SKIP_WORKTREE flag
  • sparse index with directories containing the SKIP_WORKTREE flag, in cone mode
  • write specific tests to verify those behaviours
• write sparse indexes (sdir extension)
  • sparse index can be written by providing the corresponding extension flag in the write::Options
  • index decides automatically if it needs to be sparse or not based on what type of entries are contained
• implement functionality similar to ensure_full_index()
  • scan index for sparse directory entries (trees) and expand them into a full list of filepaths (regular index structure) for use in subsequent functions that don't support working with sparse indexes yet
  • find out where and how it make sense to use that function
• matching logic of .git/info/sparse-checkout for cone mode
• matching logic of .git/info/sparse-checkout for non-cone mode / inverted .gitignore matching logic
  does this need to be supported with non-cone mode being deprecated?

Notes

• the git sparse-checkout set / add commands modify the list of files contained in .git/info/sparse-checkout, which uses the same syntax as a .gitignore file. Cone mode and non-cone mode decide how this file gets interpreted. Cone mode will match only directories while non-cone mode will use the same matching logic used for .gitignore files.
• non-cone mode is now deprecated
• non-cone mode and sparse index are incompatible with eachother

References

• docs: git-sparse-index
• docs: git-sparse-checkout
• docs: git-index | sparse entries

[SidneyDouw]

moved the issue to byron#562