Skip to content

Commit-graph: Write incremental files #184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 18 commits into from
Closed

Commit-graph: Write incremental files #184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
7c3c0df
commit-graph: document commit-graph chains
derrickstolee May 17, 2019
a06428d
commit-graph: prepare for commit-graph chains
derrickstolee May 1, 2019
62af96e
commit-graph: rename commit_compare to oid_compare
derrickstolee May 21, 2019
0b236c2
commit-graph: load commit-graph chains
derrickstolee May 21, 2019
3c1bfcf
commit-graph: add base graphs chunk
derrickstolee May 20, 2019
6595617
commit-graph: rearrange chunk count logic
derrickstolee May 22, 2019
ee8c09f
commit-graph: write commit-graph chains
derrickstolee May 21, 2019
dfca87d
commit-graph: add --split option to builtin
derrickstolee May 21, 2019
cae2eb9
commit-graph: merge commit-graph chains
derrickstolee May 21, 2019
65b1cc6
commit-graph: allow cross-alternate chains
derrickstolee May 21, 2019
2d2977b
commit-graph: expire commit-graph files
derrickstolee May 22, 2019
71f3e9f
commit-graph: create options for split files
derrickstolee May 31, 2019
5e56632
commit-graph: verify chains with --shallow mode
derrickstolee Jun 3, 2019
a4aa473
commit-graph: clean up chains after flattened write
derrickstolee Jun 3, 2019
4877166
commit-graph: test octopus merges with --split
derrickstolee Jun 7, 2019
be4b83f
commit-graph: test --split across alternate without --split
derrickstolee Jun 7, 2019
fba62e7
commit-graph: normalize commit-graph filenames
derrickstolee Jun 17, 2019
bad9f13
commit-graph: test verify across alternates
derrickstolee Jun 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 24 additions & 2 deletions Documentation/git-commit-graph.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ SYNOPSIS
--------
[verse]
'git commit-graph read' [--object-dir <dir>]
'git commit-graph verify' [--object-dir <dir>]
'git commit-graph verify' [--object-dir <dir>] [--shallow]
'git commit-graph write' <options> [--object-dir <dir>]


Expand All @@ -26,7 +26,7 @@ OPTIONS
Use given directory for the location of packfiles and commit-graph
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Ramsay Jones wrote (reply to this):



On 06/06/2019 15:15, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <[email protected]>
> 
> The split commit-graph feature is now fully implemented, but needs
> some more run-time configurability. Allow direct callers to 'git
> commit-graph write --split' to specify the values used in the
> merge strategy and the expire time.
> 
> Update the documentation to specify these values.
> 
> Signed-off-by: Derrick Stolee <[email protected]>
> ---
>  Documentation/git-commit-graph.txt       | 21 +++++++++++++-
>  Documentation/technical/commit-graph.txt |  7 +++--
>  builtin/commit-graph.c                   | 20 +++++++++++---
>  builtin/commit.c                         |  2 +-
>  builtin/gc.c                             |  3 +-
>  commit-graph.c                           | 35 ++++++++++++++++--------
>  commit-graph.h                           | 12 ++++++--
>  t/t5323-split-commit-graph.sh            | 35 ++++++++++++++++++++++++
>  8 files changed, 112 insertions(+), 23 deletions(-)
> 
> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt
> index 624470e198..365e145e82 100644
> --- a/Documentation/git-commit-graph.txt
> +++ b/Documentation/git-commit-graph.txt
> @@ -26,7 +26,7 @@ OPTIONS
>  	Use given directory for the location of packfiles and commit-graph
>  	file. This parameter exists to specify the location of an alternate
>  	that only has the objects directory, not a full `.git` directory. The
> -	commit-graph file is expected to be at `<dir>/info/commit-graph` and
> +	commit-graph file is expected to be in the `<dir>/info` directory and
>  	the packfiles are expected to be in `<dir>/pack`.
>  
>  
> @@ -51,6 +51,25 @@ or `--stdin-packs`.)
>  +
>  With the `--append` option, include all commits that are present in the
>  existing commit-graph file.
> ++
> +With the `--split` option, write the commit-graph as a chain of multiple
> +commit-graph files stored in `<dir>/info/commit-graphs`. The new commits
> +not already in the commit-graph are added in a new "tip" file. This file
> +is merged with the existing file if the following merge conditions are
> +met:
> ++
> +* If `--size-multiple=<X>` is not specified, let `X` equal 2. If the new
> +tip file would have `N` commits and the previous tip has `M` commits and
> +`X` times `N` is greater than  `M`, instead merge the two files into a
> +single file.
> ++
> +* If `--max-commits=<M>` is specified with `M` a positive integer, and the
> +new tip file would have more than `M` commits, then instead merge the new
> +tip with the previous tip.
> ++
> +Finally, if `--expire-time=<datetime>` is not specified, let `datetime`
> +be the current time. After writing the split commit-graph, delete all
> +unused commit-graph whose modified times are older than `datetime`.
>  
>  'read'::
>  
> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt
> index aed4350a59..729fbcb32f 100644
> --- a/Documentation/technical/commit-graph.txt
> +++ b/Documentation/technical/commit-graph.txt
> @@ -248,10 +248,11 @@ When writing a set of commits that do not exist in the commit-graph stack of
>  height N, we default to creating a new file at level N + 1. We then decide to
>  merge with the Nth level if one of two conditions hold:
>  
> -  1. The expected file size for level N + 1 is at least half the file size for
> -     level N.
> +  1. `--size-multiple=<X>` is specified or X = 2, and the number of commits in
> +     level N is less than X times the number of commits in level N + 1.
>  
> -  2. Level N + 1 contains more than 64,0000 commits.
> +  2. `--max-commits=<C>` is specified with non-zero C and the number of commits
> +     in level N + 1 is more than C commits.
>  
>  This decision cascades down the levels: when we merge a level we create a new
>  set of commits that then compares to the next level.
> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c
> index c2c07d3917..18e3b61fb6 100644
> --- a/builtin/commit-graph.c
> +++ b/builtin/commit-graph.c
> @@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {
>  	N_("git commit-graph [--object-dir <objdir>]"),
>  	N_("git commit-graph read [--object-dir <objdir>]"),
>  	N_("git commit-graph verify [--object-dir <objdir>]"),
> -	N_("git commit-graph write [--object-dir <objdir>] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]"),
> +	N_("git commit-graph write [--object-dir <objdir>] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] <split options>"),
>  	NULL
>  };
>  
> @@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {
>  };
>  
>  static const char * const builtin_commit_graph_write_usage[] = {
> -	N_("git commit-graph write [--object-dir <objdir>] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]"),
> +	N_("git commit-graph write [--object-dir <objdir>] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] <split options>"),
>  	NULL
>  };
>  
> @@ -135,6 +135,7 @@ static int graph_read(int argc, const char **argv)
>  }
>  
>  extern int read_replace_refs;
> +struct split_commit_graph_opts split_opts;

This 'split_opts' variable needs to be marked 'static'.

Thanks.

ATB,
Ramsay Jones

file. This parameter exists to specify the location of an alternate
that only has the objects directory, not a full `.git` directory. The
commit-graph file is expected to be at `<dir>/info/commit-graph` and
commit-graph file is expected to be in the `<dir>/info` directory and
the packfiles are expected to be in `<dir>/pack`.


Expand All @@ -51,6 +51,25 @@ or `--stdin-packs`.)
+
With the `--append` option, include all commits that are present in the
existing commit-graph file.
+
With the `--split` option, write the commit-graph as a chain of multiple
commit-graph files stored in `<dir>/info/commit-graphs`. The new commits
not already in the commit-graph are added in a new "tip" file. This file
is merged with the existing file if the following merge conditions are
met:
+
* If `--size-multiple=<X>` is not specified, let `X` equal 2. If the new
tip file would have `N` commits and the previous tip has `M` commits and
`X` times `N` is greater than `M`, instead merge the two files into a
single file.
+
* If `--max-commits=<M>` is specified with `M` a positive integer, and the
new tip file would have more than `M` commits, then instead merge the new
tip with the previous tip.
+
Finally, if `--expire-time=<datetime>` is not specified, let `datetime`
be the current time. After writing the split commit-graph, delete all
unused commit-graph whose modified times are older than `datetime`.

'read'::

Expand All @@ -61,6 +80,9 @@ Used for debugging purposes.

Read the commit-graph file and verify its contents against the object
database. Used to check for corrupted data.
+
With the `--shallow` option, only check the tip commit-graph file in
a chain of split commit-graphs.


EXAMPLES
Expand Down
11 changes: 9 additions & 2 deletions Documentation/technical/commit-graph-format.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,9 @@ HEADER:

1-byte number (C) of "chunks"

1-byte (reserved for later use)
Current clients should ignore this value.
1-byte number (B) of base commit-graphs
We infer the length (H*B) of the Base Graphs chunk
from this value.

CHUNK LOOKUP:

Expand Down Expand Up @@ -92,6 +93,12 @@ CHUNK DATA:
positions for the parents until reaching a value with the most-significant
bit on. The other bits correspond to the position of the last parent.

Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]
This list of H-byte hashes describe a set of B commit-graph files that
form a commit-graph chain. The graph position for the ith commit in this
file's OID Lookup chunk is equal to i plus the number of commits in all
base graphs. If B is non-zero, this chunk must exist.

TRAILER:

H-byte HASH-checksum of all of the above.
195 changes: 195 additions & 0 deletions Documentation/technical/commit-graph.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,197 @@ Design Details
helpful for these clones, anyway. The commit-graph will not be read or
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Philip Oakley wrote (reply to this):

Hi Derrick ,

On 03/06/2019 17:03, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <[email protected]>
>
> Add a basic description of commit-graph chains.
Not really your problem, but I did notice that we don't actually explain 
what we mean here by a commit graph (before we start chaining them), and 
the distinction between the generic concept and the specific implementation.

If I understand it correctly, the regular DAG (directed acyclic graph) 
already inherently contains the commit graph, showing the parent(s) of 
each commit. Hence, why do we need another? (which then needs explaining 
the what/why/how)

So, in one sense, another commit chain is potentially duplicated 
redundant data. What hasn't been surfaced (for the reader coming later) 
is probably that accessing the DAG commit graph can be (a) slow, (b) one 
way (no child relationships), and (c) accesses large amounts of other 
data that isn't relevant to the task at hand.

So the commit graph (implementation) is [I think] a fast, compact, 
sorted(?), list of commit oids that provides two way linkage through the 
commit graph (?) to allow fast queries within the Git codebase.

The commit graph is normally considered immutable, however the DAG 
commit graph can be extended by new commits, trimmed by branch deletion, 
rebasing, forced push, etc, or even reorganised via 'replace' or grafts 
commits, which must then be reflected in the commit graph (implementation).

It just felt that there is a gap between the high level DAG, explained 
in the glossary, and the commit-graph That perhaps the 
technical/commit-graph.txt ought to summarise.

--
Philip
>   More details about the
> feature will be added as we add functionality. This introduction gives a
> high-level overview to the goals of the feature and the basic layout of
> commit-graph chains.
>
> Signed-off-by: Derrick Stolee <[email protected]>
> ---
>   Documentation/technical/commit-graph.txt | 59 ++++++++++++++++++++++++
>   1 file changed, 59 insertions(+)
>
> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt
> index fb53341d5e..1dca3bd8fe 100644
> --- a/Documentation/technical/commit-graph.txt
> +++ b/Documentation/technical/commit-graph.txt
> @@ -127,6 +127,65 @@ Design Details
>     helpful for these clones, anyway. The commit-graph will not be read or
>     written when shallow commits are present.
>   
> +Commit Graphs Chains
> +--------------------
> +
> +Typically, repos grow with near-constant velocity (commits per day). Over time,
> +the number of commits added by a fetch operation is much smaller than the
> +number of commits in the full history. By creating a "chain" of commit-graphs,
> +we enable fast writes of new commit data without rewriting the entire commit
> +history -- at least, most of the time.
> +
> +## File Layout
> +
> +A commit-graph chain uses multiple files, and we use a fixed naming convention
> +to organize these files. Each commit-graph file has a name
> +`$OBJDIR/info/commit-graphs/graph-{hash}.graph` where `{hash}` is the hex-
> +valued hash stored in the footer of that file (which is a hash of the file's
> +contents before that hash). For a chain of commit-graph files, a plain-text
> +file at `$OBJDIR/info/commit-graphs/commit-graph-chain` contains the
> +hashes for the files in order from "lowest" to "highest".
> +
> +For example, if the `commit-graph-chain` file contains the lines
> +
> +```
> +	{hash0}
> +	{hash1}
> +	{hash2}
> +```
> +
> +then the commit-graph chain looks like the following diagram:
> +
> + +-----------------------+
> + |  graph-{hash2}.graph  |
> + +-----------------------+
> +	  |
> + +-----------------------+
> + |                       |
> + |  graph-{hash1}.graph  |
> + |                       |
> + +-----------------------+
> +	  |
> + +-----------------------+
> + |                       |
> + |                       |
> + |                       |
> + |  graph-{hash0}.graph  |
> + |                       |
> + |                       |
> + |                       |
> + +-----------------------+
> +
> +Let X0 be the number of commits in `graph-{hash0}.graph`, X1 be the number of
> +commits in `graph-{hash1}.graph`, and X2 be the number of commits in
> +`graph-{hash2}.graph`. If a commit appears in position i in `graph-{hash2}.graph`,
> +then we interpret this as being the commit in position (X0 + X1 + i), and that
> +will be used as its "graph position". The commits in `graph-{hash2}.graph` use these
> +positions to refer to their parents, which may be in `graph-{hash1}.graph` or
> +`graph-{hash0}.graph`. We can navigate to an arbitrary commit in position j by checking
> +its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +
> +X2).
> +
>   Related Links
>   -------------
>   [0] https://bugs.chromium.org/p/git/issues/detail?id=8

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 6/6/2019 8:10 AM, Philip Oakley wrote:
> Hi Derrick ,
> 
> On 03/06/2019 17:03, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <[email protected]>
>>
>> Add a basic description of commit-graph chains.
> Not really your problem, but I did notice that we don't actually explain what we mean here by a commit graph (before we start chaining them), and the distinction between the generic concept and the specific implementation.
> 
> If I understand it correctly, the regular DAG (directed acyclic graph) already inherently contains the commit graph, showing the parent(s) of each commit. Hence, why do we need another? (which then needs explaining the what/why/how)
> 
> So, in one sense, another commit chain is potentially duplicated redundant data. What hasn't been surfaced (for the reader coming later) is probably that accessing the DAG commit graph can be (a) slow, (b) one way (no child relationships), and (c) accesses large amounts of other data that isn't relevant to the task at hand.
> 
> So the commit graph (implementation) is [I think] a fast, compact, sorted(?), list of commit oids that provides two way linkage through the commit graph (?) to allow fast queries within the Git codebase.
> 
> The commit graph is normally considered immutable,

_Commits_ are immutable. The graph grows as commits are added.

This may be the crux of your confusion, since the commit-graph
file can become stale as commits are added by 'git commit' or
'git fetch'. The point of the incremental file format is to
update the commit-graph data without rewriting the entire thing
every time.

Does this help clarify what's going on?

> however the DAG commit graph can be extended by new commits, trimmed by branch deletion, rebasing, forced push, etc, or even reorganised via 'replace' or grafts commits, which must then be reflected in the commit graph (implementation).

These things create new commit objects, which would not be in
the commit-graph file until it is rewritten.

> It just felt that there is a gap between the high level DAG, explained in the glossary, and the commit-graph That perhaps the technical/commit-graph.txt ought to summarise.

I do think that technical/commit-graph.txt does summarize a lot
about the commit-graph _file_ and how that accelerates walks on
the high-level DAG. The added content in this patch does assume
a full understanding of the previous contents of that file.

Thanks,
-Stolee

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Philip Oakley wrote (reply to this):

Hi Stolee,

We may be talking at cross-purposes.
On 06/06/2019 18:09, Derrick Stolee wrote:
> On 6/6/2019 8:10 AM, Philip Oakley wrote:
>> Hi Derrick ,
>>
>> On 03/06/2019 17:03, Derrick Stolee via GitGitGadget wrote:
>>> From: Derrick Stolee <[email protected]>
>>>
>>> Add a basic description of commit-graph chains.
>> Not really your problem, but I did notice that we don't actually explain what we mean here by a commit graph (before we start chaining them), and the distinction between the generic concept and the specific implementation.
The purpose of my comment is here. We have/had not explained why we need 
(another) commit graph, either within the man page, or the technical 
docs. It's an understanding gap.
>>
>> If I understand it correctly, the regular DAG (directed acyclic graph) already inherently contains the commit graph, showing the parent(s) of each commit. Hence, why do we need another? (which then needs explaining the what/why/how)
>>
>> So, in one sense, another commit chain is potentially duplicated redundant data. What hasn't been surfaced (for the reader coming later) is probably that accessing the DAG commit graph can be (a) slow, (b) one way (no child relationships), and (c) accesses large amounts of other data that isn't relevant to the task at hand.
>>
>> So the commit graph (implementation) is [I think] a fast, compact, sorted(?), list of commit oids that provides two way linkage through the commit graph (?) to allow fast queries within the Git codebase.
>>
>> The commit graph is normally considered immutable,
> _Commits_ are immutable. The graph grows as commits are added.
I was aware that individual commits are immutable. However the tips, 
grafts and replacements can change the topology of the graph (especially 
the grafts and replacements, hence the desire to have something that 
acts as a guide as to what, generally, is trying to be achieved).
>
> This may be the crux of your confusion, since the commit-graph
> file can become stale as commits are added by 'git commit' or
> 'git fetch'. The point of the incremental file format is to
> update the commit-graph data without rewriting the entire thing
> every time.
>
> Does this help clarify what's going on?
Only slightly, see below.
>
>> however the DAG commit graph can be extended by new commits, trimmed by branch deletion, rebasing, forced push, etc, or even reorganised via 'replace' or grafts commits, which must then be reflected in the commit graph (implementation).
> These things create new commit objects, which would not be in
> the commit-graph file until it is rewritten.
>
>> It just felt that there is a gap between the high level DAG, explained in the glossary, and the commit-graph That perhaps the technical/commit-graph.txt ought to summarise.
> I do think that technical/commit-graph.txt does summarize a lot
> about the commit-graph _file_ and how that accelerates walks on
> the high-level DAG. The added content in this patch does assume
> a full understanding of the previous contents of that file.
The current (prior) documentation is a bit Catch 22 with regard to that 
assumed full understanding, hence my comment, including the "Not really 
your problem," bit.
>
> Thanks,
> -Stolee
>
Philip

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Philip Oakley wrote (reply to this):

Is this a spelling nit?
On 06/06/2019 15:15, Derrick Stolee via GitGitGadget wrote:
> 3. When writing a new commit-graph chain based on a commit-graph file
>     in another object directory, do not allow success if the base file
>     has of the name "commit-graph" instead of
>     "commit-graphs/graoh-{hash}.graph".
s/graoh-/graph-/    ?
--
Philip

written when shallow commits are present.

Commit Graphs Chains
--------------------

Typically, repos grow with near-constant velocity (commits per day). Over time,
the number of commits added by a fetch operation is much smaller than the
number of commits in the full history. By creating a "chain" of commit-graphs,
we enable fast writes of new commit data without rewriting the entire commit
history -- at least, most of the time.

## File Layout

A commit-graph chain uses multiple files, and we use a fixed naming convention
to organize these files. Each commit-graph file has a name
`$OBJDIR/info/commit-graphs/graph-{hash}.graph` where `{hash}` is the hex-
valued hash stored in the footer of that file (which is a hash of the file's
contents before that hash). For a chain of commit-graph files, a plain-text
file at `$OBJDIR/info/commit-graphs/commit-graph-chain` contains the
hashes for the files in order from "lowest" to "highest".

For example, if the `commit-graph-chain` file contains the lines

```
{hash0}
{hash1}
{hash2}
```

then the commit-graph chain looks like the following diagram:

+-----------------------+
| graph-{hash2}.graph |
+-----------------------+
|
+-----------------------+
| |
| graph-{hash1}.graph |
| |
+-----------------------+
|
+-----------------------+
| |
| |
| |
| graph-{hash0}.graph |
| |
| |
| |
+-----------------------+

Let X0 be the number of commits in `graph-{hash0}.graph`, X1 be the number of
commits in `graph-{hash1}.graph`, and X2 be the number of commits in
`graph-{hash2}.graph`. If a commit appears in position i in `graph-{hash2}.graph`,
then we interpret this as being the commit in position (X0 + X1 + i), and that
will be used as its "graph position". The commits in `graph-{hash2}.graph` use these
positions to refer to their parents, which may be in `graph-{hash1}.graph` or
`graph-{hash0}.graph`. We can navigate to an arbitrary commit in position j by checking
its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +
X2).

Each commit-graph file (except the base, `graph-{hash0}.graph`) contains data
specifying the hashes of all files in the lower layers. In the above example,
`graph-{hash1}.graph` contains `{hash0}` while `graph-{hash2}.graph` contains
`{hash0}` and `{hash1}`.

## Merging commit-graph files

If we only added a new commit-graph file on every write, we would run into a
linear search problem through many commit-graph files. Instead, we use a merge
strategy to decide when the stack should collapse some number of levels.

The diagram below shows such a collapse. As a set of new commits are added, it
is determined by the merge strategy that the files should collapse to
`graph-{hash1}`. Thus, the new commits, the commits in `graph-{hash2}` and
the commits in `graph-{hash1}` should be combined into a new `graph-{hash3}`
file.

+---------------------+
| |
| (new commits) |
| |
+---------------------+
| |
+-----------------------+ +---------------------+
| graph-{hash2} |->| |
+-----------------------+ +---------------------+
| | |
+-----------------------+ +---------------------+
| | | |
| graph-{hash1} |->| |
| | | |
+-----------------------+ +---------------------+
| tmp_graphXXX
+-----------------------+
| |
| |
| |
| graph-{hash0} |
| |
| |
| |
+-----------------------+

During this process, the commits to write are combined, sorted and we write the
contents to a temporary file, all while holding a `commit-graph-chain.lock`
lock-file. When the file is flushed, we rename it to `graph-{hash3}`
according to the computed `{hash3}`. Finally, we write the new chain data to
`commit-graph-chain.lock`:

```
{hash3}
{hash0}
```

We then close the lock-file.

## Merge Strategy

When writing a set of commits that do not exist in the commit-graph stack of
height N, we default to creating a new file at level N + 1. We then decide to
merge with the Nth level if one of two conditions hold:

1. `--size-multiple=<X>` is specified or X = 2, and the number of commits in
level N is less than X times the number of commits in level N + 1.

2. `--max-commits=<C>` is specified with non-zero C and the number of commits
in level N + 1 is more than C commits.

This decision cascades down the levels: when we merge a level we create a new
set of commits that then compares to the next level.

The first condition bounds the number of levels to be logarithmic in the total
number of commits. The second condition bounds the total number of commits in
a `graph-{hashN}` file and not in the `commit-graph` file, preventing
significant performance issues when the stack merges and another process only
partially reads the previous stack.

The merge strategy values (2 for the size multiple, 64,000 for the maximum
number of commits) could be extracted into config settings for full
flexibility.

## Deleting graph-{hash} files

After a new tip file is written, some `graph-{hash}` files may no longer
be part of a chain. It is important to remove these files from disk, eventually.
The main reason to delay removal is that another process could read the
`commit-graph-chain` file before it is rewritten, but then look for the
`graph-{hash}` files after they are deleted.

To allow holding old split commit-graphs for a while after they are unreferenced,
we update the modified times of the files when they become unreferenced. Then,
we scan the `$OBJDIR/info/commit-graphs/` directory for `graph-{hash}`
files whose modified times are older than a given expiry window. This window
defaults to zero, but can be changed using command-line arguments or a config
setting.

## Chains across multiple object directories

In a repo with alternates, we look for the `commit-graph-chain` file starting
in the local object directory and then in each alternate. The first file that
exists defines our chain. As we look for the `graph-{hash}` files for
each `{hash}` in the chain file, we follow the same pattern for the host
directories.

This allows commit-graphs to be split across multiple forks in a fork network.
The typical case is a large "base" repo with many smaller forks.

As the base repo advances, it will likely update and merge its commit-graph
chain more frequently than the forks. If a fork updates their commit-graph after
the base repo, then it should "reparent" the commit-graph chain onto the new
chain in the base repo. When reading each `graph-{hash}` file, we track
the object directory containing it. During a write of a new commit-graph file,
we check for any changes in the source object directory and read the
`commit-graph-chain` file for that source and create a new file based on those
files. During this "reparent" operation, we necessarily need to collapse all
levels in the fork, as all of the files are invalid against the new base file.

It is crucial to be careful when cleaning up "unreferenced" `graph-{hash}.graph`
files in this scenario. It falls to the user to define the proper settings for
their custom environment:

1. When merging levels in the base repo, the unreferenced files may still be
referenced by chains from fork repos.

2. The expiry time should be set to a length of time such that every fork has
time to recompute their commit-graph chain to "reparent" onto the new base
file(s).

3. If the commit-graph chain is updated in the base, the fork will not have
access to the new chain until its chain is updated to reference those files.
(This may change in the future [5].)

Related Links
-------------
[0] https://bugs.chromium.org/p/git/issues/detail?id=8
Expand All @@ -153,3 +344,7 @@ Related Links

[4] https://public-inbox.org/git/[email protected]/T/#u
A patch to remove the ahead-behind calculation from 'status'.

[5] https://public-inbox.org/git/[email protected]/
A discussion of a "two-dimensional graph position" that can allow reading
multiple commit-graph chains at the same time.
Loading