Restrict indexing into strings to a special `ByteIndex` or `StringIndex` type

[nalimilan]

This is a proposal that's been done in Rust and apparently is still under discussion (see rust-lang/rust#10044 (comment)), but I thought it could be interesting for Julia.

The idea is that since indexing strings with a number like s[3] only makes sense when 3 actually corresponds to the boundary of a unicode code point, it represents a trap for developers who only test it on ASCII, making bugs appear only in production when used with non-ASCII text. Typical cases are the naive:

julia> s = "noël";
julia> s[4] # Thinking that this provides the fourth "character"
ERROR: invalid UTF-8 character index
 in next at ./utf8.jl:68
 in getindex at string.jl:57

the tempting:

julia> s[end - 1] # Thinking you skip the last character, works well... until it breaks
ERROR: invalid UTF-8 character index
 in next at ./utf8.jl:68
 in getindex at string.jl:57

(Julia equivalent of http://www.reddit.com/r/rust/comments/1zlq21/should_rust_be_more_careful_with_unicode/cfush88)

or the slightly more involved:

julia> s[match(r"l", s).offset - 1]
ERROR: invalid UTF-8 character index
 in next at ./utf8.jl:68
 in getindex at string.jl:57

Instead of letting people do incorrect-but-easy things like this, it could be useful to restrict string indexing to a special type, say ByteIndex, instead of a plain integer. match would provide offset as that type too. It would prevent both naive indexing using integers as well as doing incorrect arithmetic on indexes you get from functions, encouraging people to always use dedicated functions.

It might also make sense to allow arithmetic operations on this type, so that idx + 1 means "the code point after the one at position idx", which would be O(n) but starting from the index -- and usually you don't take very large offsets. I'm not saying this is necessarily a good idea, though, because ByteIndex implies a reasoning in bytes, and then arithmetic operations would switch to a reasoning in code points. It could be named StringIndex instead, and made opaque so that people never see the integer index which is in bytes.

Finally, it might be possible to perform some optimizations by removing checks that the index corresponds to the start of a code point, if the index held a reference to the string it was build from, so that it can be checked that it matches the indexed string. Not sure it would be significant, though.

[elextr]

Remove indexing totally, only allow accessing strings via iterators? Access is O(1), and it knows the string it applies to.

[nalimilan]

@elextr Not remove: as the title says, restrict it to ByteIndex/StringIndex.

[elextr]

@nalimilan I was continuing on from your last comment about having a reference to the string in the index. That makes it possible to use it as an iterator, so there is no chance of applying it to the wrong string, and it can be passed to functions as a single parameter. And moving away from the indexing syntax removes the implication that it is guaranteed to be O(1).

[nalimilan]

Ah, OK. I'm not sure this is very practical since indexing is quite a common pattern, and anyway you will need a function to extract a character or a substring at a given position. Though that would allow for more explicit names like slice_bytes or slice_graphemes, as was proposed in the Rust issue.

[elextr]

Yes, byte(ByteIterator), char(CodepointIterator) and grapheme(GraphemeIterator) would definitely make it clear that different things are being found, especially as the last may be more than one codepoint which may be more than one byte whereas string[xxxIndex] doesn't really make it clear it returns an xxx. Looks more interesting the more I think about it, but as you say it may be just too radical for a language steeped in the [] index paradigm.

[StefanKarpinski]

I've considered that kind of StringIndex approach but felt that having the index reference the string object might cause performance issues and that it seemed like a kind of clunky, annoying API. If you want to play around with it and see if you can mock something up that has good performance and isn't annoying to use, that would be good and we could consider using it.

[nalimilan]

@StefanKarpinski In my mind the reference to the string object was only meant as an optimization, so if it hurts it could be avoided. The minimal version of StringIndex is just a wrapper around an integer; other points are just suggestions. What parts of my proposal do you consider worth investigating?

[johnmyleswhite]

I've argued before that indexing should be removed completely in the future, so I'll argue that again.

(1) We know there's no clearly right definition of indexing. This proposal is meant to force people to remove ambiguity about what they hope indexing will mean. So it's a huge step forward. But there's still point (2).

(2) I believe a lot of the people who will find indexing into strings intuitive will be distraught to learn that you can't mutate strings via setindex!. This is why I think indexing is really an operation best left for the mutable bytes level that shadows strings.

[nalimilan]

Actually I'd be rather in favor of removing string indexing. Making strings look like array sounds too confusing to me, because of the immutability, but also because of the absence of equivalence between bytes and codepoints.

But we would need to identify common use cases and how they would work with a new API. For example, the incorrect s[match(r"l", s).offset - 1] could become grapheme(s, match(r"l", s).offset - 1) or codepoint(s, match(r"l", s).offset - 1), depending on the application. This could work if match().offset returned a ByteIndex/StringIndex. Would an efficient indexing method based on bytes still be needed? If so, in which cases? Could it be completely avoided, with enough functions returning a ByteIndex/StringIndex for more safety?

[nalimilan]

FWIW, I've just bumped into the Rust issue where string indexing was removed: rust-lang/rust#12710 See also http://doc.rust-lang.org/guide-strings.html But they still allow taking byte slices via an explicit call, raising a run-time error on things like "aé".slice(1, 2). Issue rust-lang/rust#10044 was precisely about renaming slice and friends to something more explicit, which sounds like a very sane approach.

Rust also provides iterators for bytes, codepoints and graphemes, which must be chosen explicitly. This also sounds like a good idea to me, removing all issues from naive uses of Unicode strings. And it's possible thanks to @stevengj's PR #9261.

Go doesn't sound like a good model to follow IMHO, not least because they allow arbitrary bytes (in my words, "random garbage") in their str type, and therefore also allow indexing in the middle of codepoints... Though they still special-case UTF-8 in iteration. :-/ http://blog.golang.org/strings

[StefanKarpinski]

I don't think disallowing indexing into strings is helpful or sensible. You need a way of talking about positions in strings. Otherwise how do you talk about things like "give me the next character after this position in this string" or "give me the substring between this position and this position"? You can't use characters to talk about positions because characters can occur multiple times and even if they didn't this would imply doing an O(n) search to find a position, which is clearly a non-starter.

What might be sensible is disallowing arithmetic with indices into strings. Whenever someone writes str[i+1] they are writing Unicode-unsafe code and the Unicode-correct version should be str[nextind(str,i)]. The idea behind keeping a reference to the string as part of the StringIndex type is to do something like this:

immutable StringIndex{S<:String}
    string::S
    index::Int
end

Base.getindex(s::String, si::StringIndex) =
    s === si.string ? s[si.index] : s[chr2ind(s, ind2chr(si.string, si.index))]

function +(si::StringIndex, j::Integer)
    j < 0 && return si - (-j)
    i = si.index
    while j > 0
        i = nextind(si.string, i)
        j -= 1
    end
    return StringIndex(si.string, i)
end

function -(si::StringIndex, j::Integer)
    j < 0 && return si + (-j)
    i = si.index
    while j > 0
        i = prevind(si.string, i)
        j -= 1
    end
    return StringIndex(si.string, i)
end

That's pretty complicated code given that by far the most common use case is str[i + 1] which you want to just boil down to str[nextind(str,i)], which basically necessitates very aggressive inlining – more aggressive that we're currently able to do with the above code.

[StefanKarpinski]

I've tried a bunch of tricks to try to make this StringIndex approach generate efficient code and I can't manage it. It may be possible, but I'm wary of an approach that makes this so difficult to optimize. Still, it's plausible to have a StringIndex type that is a wrapper around Int disallowing index arithmetic.

[nalimilan]

I think there are two issues:

1. Whether indexing using [] should be allowed: this is a question of syntax/API. Anyway there needs to be a way to extract a codepoint, grapheme or substring.
2. What kind of indexing should be supported, either by getindex or dedicated functions. And whether arithmetic should be supported on that index.

Regarding 1), I don't have a strong opinion though I'm not a fan of indexing strings. (Actually I'm more concerned about iteration having different legitimate meanings.)

Regarding 2), I agree with @StefanKarpinski that inefficient arithmetic would be a problem, as it would let people use str[i + 1] when the most efficient solution is str[nextind(str,i)] (this could be fixed if/when function specialization depending on argument values was possible). But StringIndex could start as a wrapper to an int, without arithmetic, and it would already be a protection against incorrect uses. Then improvements could always come later. Though as I noted at the end of the description of the issue, storing a reference to the string could allow for some optimizations, even without allowing arithmetic, since it would save some checks in getindex about whether the index is in the middle of a code point or not.

[johnmyleswhite]

@StefanKarpinski: I don't believe the concept of "position" has any coherent meaning for strings. After all, our current indexing rules refer to indexing into bytes, whereas Python 3's indexing rules refer to indexing into codepoints. In light of multiple ways to break strings apart into pieces, [] is a fundamentally ambiguous operation -- and I personally think our choice of defaulting to byte indices doesn't offer a good performance/safety tradeoff. By the time you're indexing into bytes for performance, you should be talking about a mutable array of bytes that derives from a specific string encoding that you are fully aware of. For safety, though, you want to talk about codepoints or graphemes without reference to bytes or even encodings.