[Java] DTO's computeEncodedLength() will return invalid length for certain encodings

[the-thing]

It is impossible to use String encodings with DTOs that encode string using different byte length and Unicode units - UTF-16, UTF-32 etc. This is problematic when using tryClaim or anything else that pre allocates the exact amount of bytes.

To recreate. Add/change custom UTF-16 field result in different estimated length and encoded length. Tests such as uk.co.real_logic.sbe.generation.java.DtoTest will fail.

<composite name="varUTF16Encoding">
    <type name="length" primitiveType="uint32" maxValue="1073741824"/>
    <type name="varData" primitiveType="uint8" length="0" characterEncoding="UTF-16"/>
</composite>

<sbe:message name="Car" id="1" description="Description of a basic Car">
    [...]
    <data name="activationCode" id="20" type="varUTF16Encoding"/>
</sbe:message>

Unfortunately this means that the string has to be converted to byte[] array using SBE encoding in the xml file for that field (e.g. extension.CarDecoder#activationCodeCharacterEncoding). It could be optimized by estimating string lengths for common charsets defined in java.nio.charset.StandardCharsets.

// encodedLength += this.activationCode.length();
encodedLength += this.activationCode.getBytes(CarDecoder.activationCodeCharacterEncoding()).length;

[vyazelenko]

cc @ZachBray

[ZachBray]

Thank you for the report, @the-thing. We've recently run into this ourselves. @marc-adaptive has a fix and will look to get it merged in the coming week or so.

[marc-adaptive]

Hi @the-thing. Pushed change to address this with 4e89da2. Data without character encoding will be represented as byte[] - which we can use array length. Data with character encoding will be represented as strings. ASCII encoded strings can use .length() but other types of Charsets will be converted to bytes. We would have to potentially use average or max estimate - which could give us an invalid or large overestimate. For simplicity and correctness went with this approach.

[the-thing]

Thanks for fixing.

You could implement accurate estimation without byte conversion for standard charsets e.g. https://guava.dev/releases/19.0/api/docs/com/google/common/base/Utf8.html#encodedLength(java.lang.CharSequence) (this one is particularly efficient). It is more of a problem where to put common code as SBE is quite "raw" and probably would have to go to Agrona.

Anyway. Looks good.