[libc] Make char parsing code encoding independent

[michaelrj-google]

What: Some of our string parsing code assumes ASCII, for example the string to integer code: https://github.com/llvm/llvm-project/blob/main/libc/src/__support/str_to_integer.h#L37 We should move this to be encoding independant, likely using switch statements.

Why change it: Easier support for non-ASCII character encodings (e.g. wide character, EBCDIC).

Will be bad for performance: No, it might actually be better. Clang is very good at optimizing this sort of switch statement: https://godbolt.org/z/qvrebqvvr

[llvmbot]

@llvm/issue-subscribers-libc

Author: Michael Jones (michaelrj-google)

What: Some of our string parsing code assumes ASCII, for example the string to integer code: https://github.com/llvm/llvm-project/blob/main/libc/src/__support/str_to_integer.h#L37 We should move this to be encoding independant, likely using switch statements.

Why change it: Easier support for non-ASCII character encodings (e.g. wide character, EBCDIC).

Will be bad for performance: No, it might actually be better. Clang is very good at optimizing this sort of switch statement: https://godbolt.org/z/qvrebqvvr

[AlyElashram]

@michaelrj-google I can take a look at this, but I have a couple of questions.

• Are we only concerned with chars a-z and A-Z and numbers 0-9 only as per the godbolt you attached?
• What is the use case of these functions in LibC for testing purposes.

[michaelrj-google]

@AlyElashram I've already gotten started on this issue, see #110574, but if you are interested I could use some help making sure that these functions are used everywhere strings are being parsed (e.g. in printf).

[AlyElashram]

@michaelrj-google do we have any documentation pages that list all the functions that I can check out ? because basic searching in the IDE would not suffice.

[michaelrj-google]

I don't think there's any documentation specifically listing all the string parsing functions, but I do have some suggestions for functions to look at: Anything in ctype, the string to number functions (str_to_integer and str_to_float), printf and scanf, also I'd recommend searching the codebase for | 32 which is an idiom I used to lowercase letters in a couple places. You should also replace any uses of b36_char_to_int, which is currently defined in str_to_integer.h, with the new version in ctype_utils.h.

I don't know if that's a complete list, but it should get most of the string parsing functions.

[michaelrj-google]

Fixed by #110574