Add an option to interpret request headers as Latin1 encoded

[analogrelay]

We have customers interested in treating incoming request headers as Latin1-encoded but our default logic only supports UTF-8.

In a 3.1 patch we should provide:

• A config setting to change the default encoding to Latin1
• When enabled, Kestrel will widen each incoming byte to a UTF-16 character and build a string from that set of characters, instead of interpreting bytes as UTF-8.
• We will continue to reject control characters from the ASCII set (0x00-0x1F and 0x7F), but not reject control characters from the widened Latin1 set (because they would collide with other interpretations of this data, like UTF-8).

This would allow a consumer to reinterpret the data in this string as a UTF-8 sequence if desired.

In 5.0 we'll look at broader work to improve this experience. The 3.1 goals are scoped down to specific requests we've received.

[lodejard]

Sounds great, thank you! If I understand what you've written this it means:

• When 3.1 config flag enabled for latin1 request headers:
  • visible USASCII header-value octets 0x20-0x7E are mapped to char codes U+0020-U+007E
  • opaque header-value octets 0x80-0xFF are mapped to char codes U+0080-U+00FF
  • header-value octets 0x00-0x1F and 0x7F are unprintable USASCII and not expected to map to unicode char in any request header value strings
    • some unprintable ASCII is already consumed as http protocol delimiters
    • other unprintable ASCII will cause the request to be rejected as malformed

[analogrelay]

Yes, that is correct.

[lodejard]

Will there be a corresponding response header flag, to round-trip opaque data? Like

• When 3.1 config flag enabled for latin1 response headers:
  • char codes U+0020-U+007E are mapped to visible USASCII header-value octets 0x20-0x7E
  • char codes U+0080-U+00FF are mapped to opaque header-value octets 0x80-0xFF
  • all other char codes (U+0000-U+001F, U+007F, U+0100 and higher) are not expected to map to header-value octets
    • if present may cause the response to be rejected server-side

[halter73]

Will there be a corresponding response header flag, to round-trip opaque data?

We are considering this for 5.0, but so far it seems like this doesn't meet the bar for a 3.1 patch.

[analogrelay]

@Tratcher @halter73 between the two of you, could you take a look at getting this prepped on release/3.1? I'd like to try and take this in Feb, which means we need a ready-to-merge PR next week.

[analogrelay]

De-milestoned to make this come up in triage again for awareness.

[Tratcher]

@halter73 when could you have bandwidth to start this? I'm busy for a few days.

[halter73]

I can start looking into this tomorrow between my build rotation responsibilities.