'𠮷'.length should return 1 instead of 2

[Cat-sushi]

I understand that this is a kind of massive breaking change, and I'm looking for a migration path, but haven't found one yet.

As you can see, current String.length is confusing and harmful especially for non-english speakers.
I believe there are almost no correct usage of current String.length, because we can't manipulate String with it correctly.
The root cause of this is that String is sequence of ints representing code units but not code points even at surface of the API.
So, this proposal include introducing Rune (not Runes) as int representing a code point and changing String as a sequence of Runes at least from view point of API.
This proposal might also include deprecating current operator [] of String and introducing runeAt() to String.
Don't misunderstand me, I don't assert that the internal representation of String should be sequence of code points, by this proposal.

[Cat-sushi]

I hope a character code constant of #886 is a Rune as a int extended by #42 or #1426.

[rakudrama]

Look at https://pub.dev/packages/characters
You should be able to get the right result with '𠮷'.characters.length

[rakudrama]

@lrhn @kwalrath Lasse - do you think the String class documentation should mention the Characters package / class in the 'See Also` section?

[Cat-sushi]

@rakudrama
The extension characters introduces Characters which is a iterable of grapheme clusters, but not a code point dedicated type.
My point is that, Rune as int representing code point should be a first citizen of the language defined by dart:core and seemingly String should be a sequence of Rune.

[Cat-sushi]

Sorry, Rune is not mandatory, which can be alternated by int.
But '𠮷'.length is confusing even after introduction of characters extension.

[rakudrama]

@Cat-sushi
It is also a reasonable position that using individual code points / runes is incorrect for many of the same reasons that using UTF-16 code units is incorrect.

For example,
'🇬🇵'.length --> 4
'🇬🇵'.runes.length --> 2
'🇬🇵'.characters.length --> 1

How does your string.runeAt(i) differ from string.runes.elementAt(i)?

You say you are looking for a migration path. What is it that you are migrating, and what do you want to migrate to?
Would analyzer warnings about using String.length, String.[] and String.codeUnitAt be helpful in finding or avoiding incorrect uses?

[Cat-sushi]

The characters extension is a big step for non-english speaking country, and I appreciated the introduction very much.
Now, I'm looking at a next step.

The code point is the primitive of natural languages, but not necessarily of computers, only which can represent every single character all over the world.
I'm talking about natural languages, so forget emoji, for now.
As you may know, code unit of UTF-16 is the primitive of major programing languages including dart for a long time because of performance reasons.
But, I think it is a kind of million dollar mistake, because length and operator [] of String can't even manipulate every single character correctly, in the era of Internet.
I think, dart as a modern, sophisticated, high level and client side language is the language deserves introduction of code point as a feature of the language core.

As I said, I don't have a migration path, now.
But, I also said that almost all existing pieces of code are not correct.

Would analyzer warnings about using String.length, String.[] and String.codeUnitAt be helpful in finding or avoiding incorrect uses?

Possibly, yes.

[Cat-sushi]

For your information, character constant in the proposal #886 is being code point.
To keep consistent, String should be a sequence of code points, for this reason, too.

[Cat-sushi]

I hope #42 or #1426 provides a sophisticated solution, including a migration path.

[Cat-sushi]

Cf.
Dart string manipulation done right 👉 | by Tao Dong | Dart | Medium
It's true not only for emoji but also Han-ji.

[lrhn]

It's not actually true that "Code point is the primitive of natural language". That primitive is the grapheme cluster, "user-perceived characters" as the Unicode standard describes them.

Even writing é might can one or two code points (because of the redundant pre-composed character, but also the e + combining accent), and I've seen some platforms (Mac IIRC) produce the two code-point version just writing normal text. Caring about code point for text like that would be a mistake, just as grave as splitting a two code-unit surrogate pair.

So, as @rakudrama says, focusing on code points is simply not enough to solve the real problems that people see with non-trivial grapheme clusters. It's one step above code units, but not a significant step, because you still have all the same kinds of problems, just with slightly fewer cases each. The only real solution is grapheme clusters, which means characters.

Changing the length of String would be extremely breaking, won't match the other functions like indexOf or lastIndexOf (unless they too change and that's even more breaking), inefficient (you need to iterate the entire string to figure out its length), and with no way to make it efficient when compiled to JavaScript. So, not going to just do that.

The approach I'd go for is to use #1426 to provide a Characters-like API on top of String. Then you can opt-in to using Characters instead String by simply writing the type.

A String would still be a sequence of UTF-16 code units (because that's what it is in JavaScript, and we can't change that).

[Cat-sushi]

Yes, I knew.
But, e U+0065 and ́ U+0301 are logically separable, in addition there is a combined code point é U+00E9.
On the other hand, 𠮷 U+20BB7 is not separable, which is much more important be represented by single code point supported by the language core.
I mean code pint is not perfect, but much better and good enough to represent characters in most case.
It is not a one/ zero discussion, I prefer better world with dart.
And, I must repeat that almost all pieces of code with length, operetaro [] or codeUnitAt() are already incorrect, regardless of migration.

By the way, what is the problem with Javascript?
I think Google always proposees correct things to Javascript wold, so I believe the obstacle, if exists, will be removed by Google.

The approach I'd go for is to use #1426 to provide a Characters-like API on top of String.

Sounds good.

Then you can opt-in to using Characters instead String by simply writing the type.

Again, It should be a part of language core, because it is the primitive of natural languages.

[Cat-sushi]

@lrhn

Changing the length of String would be extremely breaking, won't match the other functions like indexOf or lastIndexOf (unless they too change and that's even more breaking), inefficient (you need to iterate the entire string to figure out its length),

Do you think length, indexOf or lastIndexOf can manipulate natural languages correctly?
Don't ignore natural languages other than english any more, it is not politically correct (sorry, I'm joking).
But, please Imagine the world where english can't be manipulate with functionality in language core.
We are oppressed for a long long time.

Java introducing GC in practical world regardless of its inefficiency is revolutional, and Dart introducing sound null safety in practical world regardless of its difficulty is also revolutional.
I think, it is the time to introduce code point in the language core to make the language more revolutional.
It is the era of the Internet and Flutter, it is drastically changed from 2011!

[lrhn]

I agree that doing something to make natural language processing easier and have fewer pitfalls is a good idea.
I disagree completely that code points are relevant to that. You need to work with grapheme clusters. Anything below that on the abstraction scale will not suffice, and will just have the same problems as using code units.

So, if we change anything, it will be to provide a grapheme cluster based abstraction, not a code point based one.
And most likely it will be on top of, or in addition to, the current String because otherwise we break a lot of existing code.
And I do mean a lot.

Some of that existing code is correct (parsing JSON works fine by looking at code units because there is no parsing-relevant character which is not a single code unit, only string contents, same for parsing URIs, dates, integers, etc.). Some is just "good enough" and breaking it won't make people happier.

I don't believe that the String API is something we can change by a single breaking change, it's something that's going to take a very long adjustment and migration period, possibly through several layers of abstraction being added, and at the end, maybe we can deprecate the plain String. I'm not sure that will ever be possible. (This is something we have thought about a lot, it's not just me being a contrarian. I do want it to happen, but I also don't want to break something so fundamental without very good reason and an easy way to fix it).