[usage] Ensure controller ticks are not concurrent

[easyCZ]

Description

Ensures that a single instance of the usage component does not invoke multiple usage controller ticks concurrently. This is done by consuming from a channel which only ever executes linearly. Additionally, we skip buffering runs when we've exceeded 1 item queued already - prevents build up.

Related Issue(s)

Fixes #

How to test

Unit tests

Release Notes

NONE

Documentation

NONE

Werft options:

• /werft with-preview

[jankeromnes]

Many thanks! Taking a look 👀

[jankeromnes]

Code looks good to me, and the unit test behaves as expected! (I also tried editing the values to make it fail -- works well).

However, I also ran a build with-preview, in order to observe skips on the real usage component.

• I set the schedule to 1ms
• But I observed no skips at all 🤔 I guess the loop is pretty fast when there are few workspaces, but still, I've seen it run only once per second (not every millisecond, as specified) -- do we somehow cap the min schedule to one second? Or is 1ms not the right format?

Still, shouldn't be a blocker, as we can safely test this in production (worst case production behavior is unchanged -- best case, parallel runs are prevented).

Another open question -- I remember @andrew-farries suggesting we use a Kubernetes cron job, in order not to re-implement a cron and a "don't run the next loop if the previous hasn't finished" mechanism. Do you have a strong preference for re-implementing this ourselves in Go as opposed to using an available Kubernetes feature?

/hold

[easyCZ]

@jankeromnes The cron library we use has the lowest resolution set to 1s, so anything below gets truncated to 1s. Indeed I wouldn't expect this to manifest in preview or even staging envs, but in prod it could. One way we could orchestrate this is to create many WSI records in preview manually which would increase the runtime above 1sec. My hope is that the test does demonstrate it would behave as expected, despite the preview/staging envs not having enough data to trigger this scenario.

[easyCZ]

Another open question -- I remember @andrew-farries suggesting we use a Kubernetes cron job, in order not to re-implement a cron and a "don't run the next loop if the previous hasn't finished" mechanism. Do you have a strong preference for re-implementing this ourselves in Go as opposed to using an available Kubernetes feature?

I would like this also, but it's a lot bigger change. We'd need to give the service a k8s service account make it interact with kubernetes. Gero wasn't convinced this is something we should do now (and I tend to agree) but yes, native k8s cron would make this a bit easier but it would increase complexity of the system (more moving parts).

[easyCZ]

I'm going to unhold this to get it into staging so we can do a test deploy this afternoon. Happy to follow-up on any of these in more details, or fix-up in other PRs.

/unhold