Improved handling of strings and unicode

[ddevault]

Treating []u8 as strings is incorrect. []u8 is an array of octets, not an array of characters. Zig should support Unicode more explicitly and enforce the distinction between []u8 and str in the language and standard library.

I propose adding a rune type, which holds one unicode codepoint. The underlying storage mechanism isn't relevant to the programmer, who can only assume it's an int capable of holding a unicode codepoint. On platforms whose pointers are sufficiently sized, it should probably be a usize under the covers. I also propose adding a str type, which is opaque but offers length and indexing of runes. The underlying string encoding is also not important to the programmer, but some possible strategies include always using UTF-8 or UCS-32, or upgrading the encoding as necessary to fit the runes the user attempts to place in it.

Also provided should be standard library functions for manipulating strings separately from []u8, and helpful functions to convert str to []u8 and back again in arbitrary encodings.

[andrewrk]

It seems to me that what this issue is calling for is standard library code to handle strings.

Can you explain what the use case is for the rune and str types you proposed? Under what circumstances would you use them?

For example, it seems to me that the hello world application should use []u8 and not str for the command line arguments to main, as well as the bytes being printed to stdout, because that's what is happening - command line args are arrays of octets, and what goes to stdout is arrays of octets. Text editors typically encode characters as UTF-8, and zig allows UTF-8 (any array of octets, really) in string literals. So if, for example, the hello world application used these types at all, it would introduce an unnecessary runtime conversion between str encoded data and []u8.

Point being, when actually do we want to use these types being proposed? I can certainly think of some use cases, such as implementing a text box in a GUI application. But at that point, why does it need to be part of the language? Would not a standard library module suffice?

[ddevault]

Can you explain what the use case is for the rune and str types you proposed? Under what circumstances would you use them?

During any string manipulation. Iterating over a str with for would give you a bunch of runes. This is more useful than iterating over a bunch of u8s, which are not characters but could be partial runes depending on the encoding of the []u8.

zig allows UTF-8 (any array of octets, really) in string literals [...] it would introduce an unnecessary runtime conversion between str encoded data and []u8.

It is necessary. Not decoding the strings will produce buggy code. The compiler can optimize the conversion away by compiling string literals into str structs instead of []u8, which it should do.

hello world application should use []u8 and not str for the command line arguments to main, as well as the bytes being printed to stdout

Args could be []u8, sure, because that's what they are. You would have to decode them before using them as strings. But there's no reason you couldn't just write []u8 to stdout if you prefer. Could also have a special syntax ala Python for string literals that are encoded as UTF-8 and become []u8 rather than str. Would also be nice to detect the signature of main and do sane argument decoding in the crt0 if the user requests args as []str.

Point being, when actually do we want to use these types being proposed? I can certainly think of some use cases, such as implementing a text box in a GUI application. But at that point, why does it need to be part of the language? Would not a standard library module suffice?

String handling is intrinsic to any programming language. Well over half of programs will do string manipulation, I expect. []u8 are not strings, and if you try to do string manipulation with them, your code will be broken. I strongly encourage you to add a language-level distinction between []u8 and str to prevent users from running into bugs. This discussion happened for Python 3 and they made the correct choice, by the way.

[andrewrk]

It is necessary. Not decoding the strings will produce buggy code.

const io = @import("std").io;

pub fn main(args: [][]u8) -> %void {
    %%io.stdout.printf("Hello, 世界\n");
}

$ ./test
Hello, 世界

Where's the bug?

What does an additional type in this example accomplish?

It sounds like you're saying, this is not an example where it is necessary, but users will have other use cases where they want to be doing string manipulation rather than array of octet manipulation, such as, say, taking stdin, uppercasing it, and printing it to stdout. Would that be a fair use case?

[ddevault]

The bug doesn't present itself in the simple case. Here are a number of examples that would break or be unsupported:

• Splitting the string by non-ascii characters
• Index of a character
• Get the last character
• Get the first n characters of the string
• Reverse the string
• Insert characters at index n

Your example only works because you're really just copying []u8 around. You're not actually doing string operations.

[ddevault]

Why do you think any major language designed in the past 10 years, no matter how low level, has had proper unicode string support?

[thejoshwolfe]

So proper unicode string support is doable in userland without any modification to the Zig language or runtime. String literals can be converted to anything at compile time with userland functions, and all those string manipulation operations mentioned above can be done in userland functions as well, even also at compile time.

Am I right in saying that this is a proposal for a standard library module, and not a proposal for a language change?

[ddevault]

Not as far as I can tell. I presume that making some_str[1] do the right thing involves language changes, and making string literals emit the string type instead of []u8 is also a language change. I'm not sure if zig already supports compile-time reflection to determine the parameters of main - to support the crt0 change I proposed that may require language changes.

[thejoshwolfe]

I presume that making some_str[1] do the right thing involves language changes

With a userland string solution, you would probably not be able to do some_str[1], unless some_str was a slice. If your string solution is a struct that contains a slice, then perhaps some_str.chars[1] would work without language changes.

But if your string solution is utf8-encoded or a rope data structure or something else that's not simply a slice of characters, then character access wouldn't be as simple as the [] makes it look. Zig does not have operator overloading, and that's to avoid hidden runtime costs. If character access requires a O(log(n)) tree traversal, then make that operation a function and call it like a function.

String literals can be converted to any encoding you want at compile time like this:

const motd = decodeUtf8("こんにちは");

I'm not sure if zig already supports compile-time reflection to determine the parameters of main - to support the crt0 change I proposed that may require language changes.

Is there a reason other than convenience why you wouldn't want to do the conversion at the top of your main implementation by calling a userland function? Is there a reason to do it in the crt0 instead?

[ddevault]

But if your string solution is utf8-encoded or a rope data structure or something else that's not simply a slice of characters, then character access wouldn't be as simple as the [] makes it look. Zig does not have operator overloading, and that's to avoid hidden runtime costs. If character access requires a O(log(n)) tree traversal, then make that operation a function and call it like a function.

I'm not suggesting operator overloading - I'm just suggesting strings behave this way, which is why a language change is required.

String literals can be converted to any encoding you want at compile time like this:

What does that even do? The behavior of your example is not predictable by people who understand string encodings without reading the docs and probably the code because when you write the docs you will likely fail to understand what's confusing about it.

Is there a reason other than convenience why you wouldn't want to do the conversion at the top of your main implementation by calling a userland function? Is there a reason to do it in the crt0 instead?

No, just convenience.

[thejoshwolfe]

There's an elephant in the room in this discussion, which is that Zig wants to take memory allocation seriously. How exactly to handle memory allocation is a big discussion, and one that should probably happen in a different issue. Some high level points that are relevant here are:

• Functions that must allocate memory might want to declare an allocator as a parameter. fn splitString(s: &const String, sep: &const String, allocator: &Allocator)->[]String.
• Or perhaps there's a global allocator that can be set. fn setGlobalAllocator(allocator: &Allocator).
• Or perhaps all Zig functions have a secret context parameter, like in Jai.
• Or perhaps there's some kind of thread local storage that manages allocators.

There's a lot there to discuss, and again, that should probably be in another issue. The point I'm trying to make in this issue is that we can't supply functions like string splitting without thinking about memory allocation. Languages that don't ask you to think about memory allocation are, from Zig's perspective, sub-optimal languages. Python 3, JavaScript, and Java all have garbage collection, which makes string manipulation look very nice at a high level, but fails Zig's goal of optimality.

So far, Zig provides a List(T) class that does memory management by asking explicitly for an allocator when you construct the list. If we run with this idea for now, then you could make a string builder class that can decode from utf8, encode to a utf8 output buffer, stores unicode data opaquely, even offers functions like splitting and random character access. Does it make sense for one of these string builds to exist at compile time? Maybe, but it would be complicated, since allocators probably need to work differently at runtime vs compile time. Does Zig want to supply such a string builder class in its standard library? Maybe.

@SirCmpwn Can you give examples of low level languages with proper unicode string support? I'd like to check out how they do memory management.

[ddevault]

Rust and Go come to mind as being fairly low level and having sane Unicode support. Also, you can do a lot of things with this design without bringing allocation into the discussion.

[thejoshwolfe]

After a few minutes of research, it looks like Rust supports 1 global allocator per build artifact determined at compile time. This lacks the features of the Jai solution, which allow multiple distinct concurrent allocators running in the same application at the same time.

I believe Go has a memory management strategy with hidden allocations and a garbage collector. I've heard some people call Go a low level language, and I'm aware of an intense debate on the internet about Go's memory management strategy, even getting so intense as to call Go's marketers liars. But Go drama aside, Go's memory management strategy is not acceptable for Zig, so Go's unicode strategy is not very helpful in designing a unicode strategy for Zig.

[ddevault]

I wouldn't mind a global allocator. Perhaps you could use an allocator keyword to set a new allocator for a given scope? Again, though, many many sane Unicode string handling functions don't need allocators. And for that matter non-sane string splitting probably needs allocation as well. I don't really this it's relevant to this issue.

[thejoshwolfe]

I wouldn't mind a global allocator. Perhaps you could use an allocator keyword to set a new allocator for a given scope?

That sounds like Rust and Jai respectively.

The allocation discussion is a bit off topic, but it is relevant to keep in mind. Let's get back to string support and discuss some string functions/methods we might want to have.

I've gone through the list of Java 7's String methods and pasted in some highlights to discuss. The code examples are Java's API, not a proposed signature for Zig, although I'd like to discuss what Zig's version of each feature would be.

• String(byte[] bytes, Charset charset) and byte[] getBytes(Charset charset): What charsets should be supported? Just UTF-8? Maybe also ISO-8859-1? Maybe also Windows-1252? Maybe "all of them"? Maybe that's configurable at compile time? Or maybe there could be a dynamic library that provides these? Memory allocation is relevant here.
• int compareTo(String anotherString) and boolean equals(Object anObject): Easy to implement. And I believe these will also work with naive UTF-8 []u8 lexicographical comparison.
• int compareToIgnoreCase(String str), boolean equalsIgnoreCase(String anotherString), String toLowerCase(), and String toUpperCase(): This requires a table of unicode points with data about each character. This would be a significant feature to provide, and we may want to provide a standard solution to this. Memory allocation is relevant to the last two methods here.
• String toLowerCase(Locale locale) and String toUpperCase(Locale locale): I didn't know that uppercasing and lowercasing were sensitive to locale. Should Zig worry about this?
• int indexOf(int ch) and int indexOf(String str): Easy.
• boolean startsWith(String prefix) and boolean endsWith(String suffix): Easy.
• int hashCode(): Implementations would be easy, but deciding on an implementation might be hard.
• String replace(char oldChar, char newChar) and String replace(CharSequence target, CharSequence replacement): Algorithmically easy. Memory allocation is relevant here.
• String[] split(String regex): Probably wouldn't drag regex into this, but otherwise easy to implement. Memory allocation is relevant here.
• String substring(int beginIndex, int endIndex) and String trim(): Easy to implement. Memory allocation might be relevant here depending on the underlying string implementation.

Additionally, both Rust and Java 7 seem to have methods related to interpreting UTF-8 or UTF-16 bytes as sequences of variable-length codepoints, but that might not be necessary depending on the string implementation. It does raise a question though, which is how should unicode strings really be implemented?

I propose adding a rune type, which holds one unicode codepoint. The underlying storage mechanism isn't relevant to the programmer, who can only assume it's an int capable of holding a unicode codepoint.

To me, this just means it's a u32. The range of possible unicode codepoints isn't a mystery. It's 0-0x10FFFF, which is bigger than a u16 and small enough for a u32.

On platforms whose pointers are sufficiently sized, it should probably be a usize under the covers.

Wouldn't this be way too big on 64-bit platforms?

The underlying string encoding is also not important to the programmer, but some possible strategies include always using UTF-8 or UCS-32, or upgrading the encoding as necessary to fit the runes the user attempts to place in it.

I've seen these strategies done before, and they've all got their strengths and weaknesses. I've come to the conclusion that there is no such thing as a single best implementation of a unicode string, but rather countless subtle optimizations you can make to suit your different usecases. (This makes string implementations very similar to memory allocators in that regard.)

Zig can provide a general-purpose string implementation, but I don't like the idea of the standard implementation getting any special treatment that a homemade implementation can't get. An optimal string implementation is part of Zig's quest for optimality, and that's not possible with a standard string implementation. This means that userland string solutions need to be first-class citizens.