Azure DevOps: Source controller getting stuck

[mfamador]

Hello.

We have 3 AKS clusters, all running the exact same versions of flux (0.16.1) in two different Azure regions (North Europe and East US).

The source-controller version is 0.15.3.

❯ k describe deploy source-controller -n flux-system --context aks-stag-eun | grep -i image
    Image:       ghcr.io/fluxcd/source-controller:v0.15.3
❯ k describe deploy source-controller -n flux-system --context aks-stag-ue | grep -i image
    Image:       ghcr.io/fluxcd/source-controller:v0.15.3

Both clusters are synching with the same Azure DevOps git repositories (gitImplementation: libgit2).

Everything is working great on East US clusters but in North Europe source-controller gets stuck multiple times a day and only killing it seems to make the sources to reconcile again (we've created a cronjob to restart source-controllerevery half a hour).

Even restarting every half a hour we're still getting a lot of gaps where there's no source reconciliation.

In this state, any manual reconciliation also gets stuck and never finishes:

>  flux reconcile source git core -n core --context aks-stag-eun

► annotating GitRepository core in core namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation

There's no logs on source-controller when it's in this lock state ...

I'm pretty sure it's a connectivity problem to Azure DevOps or something not directly related to source-controller, but maybe it should recover or timeout from whatever it's trying to do (?)

I've also increased concurrent from the default 2 to 6 but it seems to not be doing anything differently.

Thanks!

[okautz]

We are facing the same behavior with our AKS clusters running in Azure west europe and china east 2. The problem only occurs in the china region.

[mfamador]

Hey @o-otte, are you also using Azure DevOps git source (libgit2) or any other git like Github?

[okautz]

Hi @mfamador, yes we also use Azure DevOps

[gmaiztegi]

Hello,

We just had the same issue in one of our clusters (AWS EKS in us-east-2), which is synchronising from Github. We updated to the last version of Flux last Friday and we have only seen this behaviour this time.

[natarajmb]

Hello,
Seeing the same issue with AzureDevOps hosted git repo with libgit2. It has worked flawlessly for the last 6 months until now. The same git repo called from another cluster works fine. Any workarounds?

Git source-controller same as above v0.15.3

[kingdonb]

The latest source-controller is v0.21.2 from Flux CLI v0.27.2. There have been substantial changes in all parts of Flux since June, when the source-controller v0.15.3 was released. Some of the most recent changes are updates that targeted issues like these, and those updates might have already resolved this issue for the original poster.

Are you able to reproduce it consistently @natarajmb, or does it go away when you restart? (Are you able to try an upgrade?)

If you can reproduce it with a current release, then we can dedicate some resources to trying to reproduce it again. I have a feeling this issue is either solved now, or it will be solved soon; but it seems tricky to reproduce. It will not be possible to investigate effectively based on reports for an older version, if we do not have a report stating for sure if this issue remains with the current version of Flux.

[natarajmb]

@kingdonb I now restarted source-controller and it started to pull the latest version of the code. FYI, we run flux from single repo based on path separation for multiple environments/clusters. I think this has occurred as we had a git revert on the path specific to this cluster after which it never pulled. Restarting source-controller fixed it. Thanks for the heads up will upgrade to the latest Flux.

[pjbgf]

Re-opening until we get confirmation that the issue has been fixed.

@mfamador @natarajmb do you mind trying the fixes included in the new experimental Managed Transport and let us know whether that fixes your issue please?

[mfamador]

@pjbgf, will do that, thanks. Removed the cronjobs that are restarting the source-controller every 30 mins and will monitor and let you know if the issue is gone.