Reuse retry topic for the maxInterval delay of exponential backoff

[jgslima]

Related to the discussion below:

#2484

Expected Behavior
In order to allow the configuration of an exponential backoff policy that keeps trying for a long time (say some hours), to have some way to use a single retry topic for the attempts corresponding to the maxInterval delay. This makes it more viable to configure the policy to keep trying for a long time because a lot of topics are not needed.

Current Behavior
When using exponential backoff, the number of needed retry topics is always the configured maxAttempts minus 1.

Context
For instance, when configuring the exponential backoff with initialInterval=1000, multiplier=2, and maxInterval=16000, in order to keep trying for one hour, one would need to configure maxAttempts as 229, and by default the needed retry topics would be:

• -retry-1000
• -retry-2000
• -retry-4000
• -retry-8000
• -retry-16000-0
• -retry-16000-1
• -retry-16000-2
• ...
• -retry-16000-224

When using the strategy that reuses the retry topic for the same intervals, in the same configuration above the needed retry topics would be only 5:

• -retry-1000
• -retry-2000
• -retry-4000
• -retry-8000
• -retry-16000

[ctoomey]

@jgslima what's the reason you don't want to just use a single retry topic for all intervals (like the single topic option for fixed delays)? That's what we'd like to do.

@garyrussell, anyone: why is the single retry topic option only supported for fixed backoff policies and not exponential or other sleeping backoff policies? Looking at the code I see it tracks the next sleep interval in a kafka header so doesn't require that it be part of the topic name.

[jgslima]

@jgslima what's the reason you don't want to just use a single retry topic for all intervals (like the single topic option for fixed delays)? That's what we'd like to do.

@garyrussell, anyone: why is the single retry topic option only supported for fixed backoff policies and not exponential or other sleeping backoff policies? Looking at the code I see it tracks the next sleep interval in a kafka header so doesn't require that it be part of the topic name.

It is not possible to use the single retry option for backoffs that may have distinct sleep times for distinct records. The general algorithm only works with the assumption that, in a partition (actually in the topic as a whole), records with higher offsets should not be processed before records whith lower offsets. Because the Consumer is paused based on the sleep time of the next record.

The motivation to have this new kind of policy is to allow a configuration where you can have the first retries done quickly (with a low interval), keep retrying for a large amount of time (say 4 or 6 hours), but without having to have an excessive amount of retry topics. Currently, when using exponential backoff, you need a large amount of retries topics for this. See how I explained in the issue description.

[ctoomey]

It is not possible to use the single retry option for backoffs that may have distinct sleep times for distinct records. The general algorithm only works with the assumption that, in a partition (actually in the topic as a whole), records with higher offsets should not be processed before records whith lower offsets. Because the Consumer is paused based on the sleep time of the next record.

Ah right, I'd forgotten about the problem w/ mixing records w/ different sleep times, thanks.