Description
Is your feature request related to a problem? Please describe.
We use Cortex as a remote_write backend. Newer versions of our remote_write client have introduced a delay in sending data to Cortex, such that recording rules that rely on this data are undercounting. It turns out Prometheus has a configuration for exactly this scenario, documented here, but it's not yet supported by Cortex.
Describe the solution you'd like
Ideally Cortex would support both the global rule_query_offset ruler configuration option, as well as the per-rulegroup override. The global configuration option could work similarly to the existing evaluation_interval option, which can also be overridden on individual rule groups.
Describe alternatives you've considered
None
Additional context
The new rule query offset configuration was added in prometheus/prometheus#14061, whch is scheduled for the upcoming prometheus 2.53.0 release. The prometheus dependency on master in this repo has already been bumped to a sha that includes that change.