Fix alerting on Stripe webhook failures

[jldec]

We are not getting alerts for Stripe webhook failures.
E.g. See slack thread (internal)
and another one (internal)

[easyCZ]

The alert which should be firing for this is defined here.

The query used in the alert, displayed in grafana is here. From this, the alert should've fired.

[easyCZ]

But from the alerts log, it hasn't.

[jldec]

@easyCZ how can we test alerting on Stripe webhook failures?
Is there a way to inject failures in dev-mode and confirm that the alerts are triggered>

[easyCZ]

The webhook is sent against our API. So we can mimic the webhook by sending such a payload. We will likely need to craft the payload carefully as there is signature validation but should be doable, in theory.

[easyCZ]

Reviewing the alert, and the metrics, the alert looks to be setup correctly. If we evaluate the alert in grafana, we can see that it would indeed fire.

https://grafana.gitpod.io/explore?orgId=1&left=%7B%22datasource%22:%22P4169E866C3094E38%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22P4169E866C3094E38%22%7D,%22editorMode%22:%22code%22,%22expr%22:%22sum%28increase%28gitpod_http_request_duration_seconds_count%7Bhandler%3D%5C%22%2Fstripe%2Finvoices%2Fwebhook%5C%22,%20code%3D~%5C%225.%2A%5C%22%7D%5B30m%5D%29%29%20%3E%200%22,%22legendFormat%22:%22__auto%22,%22range%22:true,%22instant%22:true%7D%5D,%22range%22:%7B%22from%22:%22now-30d%22,%22to%22:%22now%22%7D%7D

I'll reduce the duration in the alert, for it to fire earlier as well.