Extremely slow cloning when using large SemVer range #1769
Description
Some info to set the stage:
- We use a monorepo style structure
GitRepositoriesdo not support parsing and extracting tags likeImagePolicies- you can parse+extract SemVer in OCI tags like
my-app-v1.2.3=>1.2.3 - you must use exclusively plain SemVer tags for
GitRepositories
- you can parse+extract SemVer in OCI tags like
- To get around this deficiency, we've allocated chunks of major SemVer git tags to projects in the repo.
- project-1 gets 0.0.0 -> 999999.x.x
- project-2 gets 1000000.0.0 -> 1999999.x.x
- ...
This means when a team wants to deploy a dev version of kustomizations that control their app, it ends up looking something like this:
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: project-1-dev
namespace: flux-system
spec:
interval: 1m0s
ref:
semver: '>=1000000 <2000000'
secretRef:
name: example-ssh-key
url: ssh://[email protected]/owner/monorepo.gitThis has been working very well for a long time, but there is a huge performance problem when the SemVar range is that big. I want to be clear, projects do not actually use that many tags (as writing, the most I see is ~60 majors).
Here is a graph of the enormous performance discrepancy:
In the graph you can see that dev cloning is 50-75x slower and this project in particular only has 49 tags. On top of this it also causes relatively high CPU usage from the source-controller pod.
Potential solutions (in order of preference)
- Allow SemVer pattern extractions like
ImagePolicies. This way we don't have to (ab)use git tags like this. - Figure out why it takes so long to compare SemVer tags. I looked around in fluxcd/pkg/git but nothing in the cloning code stood out to me.