Feature/introduce in memory git metadata #1243

JakeGinnivan · 2017-06-24T14:56:58Z

My goal here is to move GitVersion into more distinct stages.

Load all the information it needs from git, this would be branches, tags, merge bases, parent branches etc.
- Ideally this would source from both local and remote branches so we don't need to normalise
- Afaik we introduced normalisation in to simplify the codebase and remove the concept of remote and local branches, it just makes everything local.
Calculate version, this would now be entirely in memory and not touch libgit2
We could then switch out the provider which actually loads the information (for instance, libgit2 or github api or git command line)

After this happens the slow part should be step 1, which should be way easier to speed up because all git operations will be in one place and we can make better decisions how to load it all.

Keen for feedback @gep13 @asbjornu

This hopefully will end up with us getting rid of normalisation and making GitVersion much faster

This reduces scanning the branch commits by one pass

…ersionStrategy to be fixed after changes to TaggedCommitVersionStrategy

JakeGinnivan · 2017-06-24T14:59:38Z

In terms of performance, this should have reduced the number of times the current branch commits are scanned by at least one pass.

asbjornu

This is awesome, @JakeGinnivan! I really like where this is going and expected the task to be much more massive than this appears to be.

Not sure how far we should take this, but I would personally love to have a GitVersion.Core project completely independent of LibGit2 with its own home-grown, well-nurtured and fine-tuned domain model.

I think this is a huge step in the right direction, I'm just not sure how many steps in the same direction we should take yet and if this suffices for now.

I just noticed #1244 as well; I'll hopefully have time to look at that tomorrow.

asbjornu · 2017-06-25T21:35:57Z

src/GitVersionCore/GitRepoInformation/GitRepoMetadata.cs

@@ -0,0 +1,76 @@
+using System.Collections.Generic;
+
+namespace GitVersion.GitRepoInformation


If this is going to be GitVersion's "Core", i.e. Domain Model (in DDD terms), wouldn't a more authoritative sounding namespace like GitVersion.Repositories, GitVersion.Graph be good?

I like GitVersion.Graph! Will keep that in mind as a progress

asbjornu · 2017-06-25T21:36:16Z

src/GitVersionCore/GitRepoInformation/GitRepoMetadata.cs

+        public List<MTag> Tags { get; private set; }
+    }
+
+    public class MTag


What's the M prefix?

Rather than aliasing in a bunch of files which reference both this namespace and libgit2 the metadata related ones are prefixed with M. I expect this to go at some point

asbjornu · 2017-06-25T21:39:31Z

src/GitVersionCore/GitRepoInformation/GitRepoMetadata.cs

+        public MBranch MasterBranch { get; }
+        public List<MBranch> ReleaseBranches { get; }
+        public MBranch CurrentBranch { get; }
+        public List<MTag> Tags { get; private set; }


Why the private setter?

asbjornu · 2017-06-25T21:39:43Z

src/GitVersionCore/GitRepoInformation/GitRepoMetadata.cs

+        }
+
+        // Wonder if this can be 'mainline'
+        public MBranch MasterBranch { get; }


I like the immutability here. 👍

asbjornu · 2017-06-25T21:42:06Z

src/GitVersionCore/GitRepoInformation/GitRepoMetadata.cs

+        public string Name { get; }
+        public string TipSha { get; }
+        public MParent Parent { get; }
+        public List<MergeMessage> MergeMessages { get; }


Not sure it matters much, I've just got a habit of using IList<T> instead of List<T> in public declarations.

asbjornu · 2017-06-25T21:50:25Z

src/GitVersionCore/GitRepoInformation/GitRepoMetadataProvider.cs


-            public Commit MergeBase { get; private set; }
+            public string MergeBase { get; private set; }


Why the change from Commit to string here?

Just trying to reduce the number of libgit2 references

Right. Good idea.

asbjornu · 2017-06-25T21:51:57Z

src/GitVersionCore/GitRepoInformation/Libgit2RepoMetadataProvider.cs

+
+namespace GitVersion.GitRepoInformation
+{
+    public class Libgit2RepoMetadataProvider


If LibGit2 is going to become one of many repository metadata providers for GitVersion, perhaps this should be moved to its own GitVersion.LibGit2 project?

Good idea, will do once I get the structure correct

asbjornu · 2017-06-25T21:52:43Z

src/GitVersionCore/GitRepoInformation/Libgit2RepoMetadataProvider.cs

+{
+    public class Libgit2RepoMetadataProvider
+    {
+        public static GitRepoMetadata ReadMetadata(GitVersionContext context)


I would expect each metadata provider implementation to adhere to some sort of centrally defined interface, not a static class?

Yeah I think so, just wasn't sure the shape at the time, this allowed me to just hook it in.

I am trying to make seams at the moment which will simplify code elsewhere, then can put the right abstraction in place

asbjornu · 2017-06-25T21:59:28Z

src/GitVersionCore/GitRepoInformation/GitRepoMetadata.cs

+
+namespace GitVersion.GitRepoInformation
+{
+    public class GitRepoMetadata


Is this the repository metadata or data? Since GitVersion's job is to walk the graph, count commits, etc., I'd say that anything required to calculate the version is data, not metadata. I think this class can be called just Repository, no? Any collisions with LibGit2 should be resolved within the GitVersion.LibGit2 project and not exposed elsewhere, so it shouldn't be much of a problem, I think.

I am unsure at the moment to be honest. I don't want to re-implement git data structures, this really is information about the commit graph with a bunch of pre-calculated stuff. Lets see where it goes then fix naming next

asbjornu · 2017-06-25T22:10:23Z

src/GitVersionCore/GitVersionContext.cs

+        /// We should move to use this, that way libgit2 is used when bootstrapping
+        /// this will be far easier to optimise then all the logic for gitversion will be entirely in memory
+        /// </summary>
+        public GitRepoMetadata RepositoryMetadata { get; private set; }


If we agree that GitRepoMetadata should be called just Repository, the same goes for this property.

erikbra · 2019-10-16T06:01:50Z

@JakeGinnivan sorry to post on a closed PR, but I'm really interested in speeding up this stuff as well. Would you care to share some thoughts on why this PR was closed? And similar with #1244, Was it due to lack of time, or do you think they are a step in the wrong direction? I've only just started looking at GitVersion, so I haven't looked at this in detail. I just wanted to know if this was a good place to start if wanting to work on the perf wrt the integration with LibGit2.

JakeGinnivan · 2019-10-16T06:06:24Z

Hey, all good.

Was a few things, I kinda lost context around where I was and how to finish it off mainly though from memory.

Generally I think the approach should be:

Create a single interface which all interactions with libgit2 is put behind. The methods on this should be pretty specific like getPossibleParentBranches() or something (and could be memoized like you suggested in the other thread).
Once this has been refactored, try to implement a in memory version of this interface which the in memory structure is populated in the most efficient way possible ahead of time.

Doing this would also allow you to implement say a github api implementation, then gitversion as a service could be created :P

JakeGinnivan · 2019-10-16T06:07:12Z

It's been a while, but if there are other questions I'll try to answer best I can @erikbra

erikbra · 2019-10-16T06:13:47Z

Thanks a lot for your replies, @JakeGinnivan. I'm only just starting to look at this, so I don't have a good understanding of the structures required yet. I'll read more of the original source code, and fork off this PR to get an understanding of the thoughts behind it.

I really appreciate if you would be able to answer some questions when the time comes. I'll get back to you on that one.

I might be jumping ahead here, but C# has gotten a lot of perf-focused new features lately, so we might be able to do some of the reading in pure C# with Memory spans, etc. in the future. But let's work on the separation first :)

Really like the idea of GitVersion-as-a-service ;)

JakeGinnivan added 3 commits June 24, 2017 11:42

Start decoupling the runtime of gitversion from libgit2

271a403

This hopefully will end up with us getting rid of normalisation and making GitVersion much faster

Added tags to git metadata

e44aa1b

This reduces scanning the branch commits by one pass

Fixed ReleaseBranchScenarios tests which needed TrackReleaseBranchesV…

1677a83

…ersionStrategy to be fixed after changes to TaggedCommitVersionStrategy

Handle unrelated branches

20f7a14

This was referenced Jun 24, 2017

More caching of merge bases #1112

Closed

introduce in memory git metadata 2 #1244

Closed

asbjornu reviewed Jun 25, 2017

View reviewed changes

asbjornu mentioned this pull request Jun 28, 2017

Feature request: Determine version from Git service API #1122

Closed

asbjornu mentioned this pull request Aug 2, 2017

Normalization takes a very long time #1266

Closed

asbjornu mentioned this pull request Mar 9, 2018

More inheriting issues #1347

Closed

This was referenced Mar 21, 2018

GitVersion newer than v3.4.1 has severe performance problems #1066

Closed

2 remote(s) have been detected error. #1390

Closed

danijam mentioned this pull request Jul 31, 2018

GitVersionTarget build is very slow, Normalisation step with many branches #1448

Closed

JakeGinnivan closed this Aug 21, 2018

asbjornu mentioned this pull request Dec 7, 2018

Unable to use gitversion as part of AppVeyor build #1477

Closed

asbjornu mentioned this pull request Jul 1, 2019

Allow repo normalization to be disabled #1701

Closed

asbjornu mentioned this pull request Oct 15, 2019

Slow version calculation times on freshly cloned pull request #1850

Closed

asbjornu mentioned this pull request Feb 26, 2020

GitVersion.Core library should allow bypassing shellout interop #2138

Open

arturcic deleted the feature/IntroduceInMemoryGitMetadata branch March 13, 2024 07:39

		@@ -0,0 +1,76 @@
		using System.Collections.Generic;

		namespace GitVersion.GitRepoInformation


		public Commit MergeBase { get; private set; }
		public string MergeBase { get; private set; }

Feature/introduce in memory git metadata #1243

Feature/introduce in memory git metadata #1243

Uh oh!

Conversation

JakeGinnivan commented Jun 24, 2017

Uh oh!

JakeGinnivan commented Jun 24, 2017

Uh oh!

asbjornu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erikbra commented Oct 16, 2019

Uh oh!

JakeGinnivan commented Oct 16, 2019

Uh oh!

JakeGinnivan commented Oct 16, 2019

Uh oh!

erikbra commented Oct 16, 2019

Uh oh!

Uh oh!