RetryableTopic - Add Option To Only Retain Last Exception Headers

[garyrussell]

See https://stackoverflow.com/questions/69830692/spring-kafka-retryabletopic-causes-recordtoolargeexception

Since RecordHeaders can have multiple instances of the same Header, this can cause a linear growth in the record size, with many retries.

[garyrussell]

Added a work around: https://stackoverflow.com/questions/69830692/spring-kafka-retryabletopic-causes-recordtoolargeexception/69831100#69831100

cc/ @tomazfernandes FYI

[tomazfernandes]

Hi @garyrussell, we have logic in the DeadLetterPublishingRecoverer to handle the original headers, I think we'd probably have to add logic there to remove pre-existing exception headers based on this option, right? Or did you think of something else?

I guess it's too late to make this the default for retryable topics?

If we don't want to add more options to the annotation and configuration builder, we could use something like this:

  @Bean(RetryTopicInternalBeanNames.DEAD_LETTER_PUBLISHING_RECOVERER_PROVIDER_NAME)
  public DeadLetterPublishingRecovererFactory dlprf(DestinationTopicResolver resolver) {
	  return new DeadLetterPublishingRecovererFactory(resolver).onlyRetainLastExceptionHeaders();
  }

In this approach it would be a global configuration rather than per-annotation / configuration instance, which I guess might be a good thing for most cases. And we can always add it to the annotation later if needed.

[tomazfernandes]

Also, maybe we should add a new constant RetryTopicInternalBeanNames.DEAD_LETTER_PUBLISHING_RECOVERER_FACTORY_NAME for naming consistency, having the same value as the one with PROVIDER on it

[garyrussell]

@tomazfernandes Thanks for the comments; I was thinking of adding a new property stripPreviousExceptionHeaders, or similar, to the DLPR.

I think we can change the default to true in the upcoming 2.8 release (false in 2.7).

I am not sure we need a change to the factory since it already has as recovererCustomizer property if the user wants to change the default.

Agreed about the constant.

[tomazfernandes]

Sounds great @garyrussell. I’m glad we can change the default, makes a lot more sense.

I agree that we don’t need to change the factory and can rely on the customizer for this. Might be worth it if we couldn’t change the default, which is not the case.

Thanks a lot.