UTF surrogate pairs broken

[gramster]

These seemed to be working a few days ago, and still work when using tools/build.py/test.py, but no longer work in the editor. For example,

'\uD800\uDC00'

In the editor I now get:

Multiple markers at this line
    - Unexpected token 'ILLEGAL'
    - Expected ',' or ')', but got
     'ILLEGAL'

This is with editor build 7552.

[devoncarew]

You can reproduce the problem w/ this snippet:

void main() {
  var bar = '\uD800\uDC00';
}

while this code doesn't exhibit the problem:

void main() {
  var bar = '\u0041\u0041';
}

I think it's an issue in the scanner.

---

Set owner to @scheglov.
Added Area-Editor, Triaged labels.

[scheglov]

1. I don't believe that this worked "a few days ago". I see code in DartScanner which fires this error, and it was changed last time in February.
2. I don't see in spec anything about surrogate pairs. But I see \u{HEX DIGIT SEQUENCE}. So, I think that you should specify valid Unicode scalar value, not UTF16 surrogates.

This code works:
var foo = '\u{10000}';
var bar = '\u{10FFFF}';

---

Added AsDesigned label.

[gramster]

1. I am working on uriEncode and uriDecode library functions. I have unit tests for these that use the '\uD800\uDC00' string. That unit test compiled and passed last week in the editor. Believe what you will; that is what I saw. That test now will not compile in the editor, but it will compile and run fine from the command line.
2. Given 1, we now have inconsistent behavior. I can use '\uD800\uDC00' when running tests from tools/build.py, tools/test.py, but not in the editor. We should be consistent.

[gramster]

BTW I just checked and I still have the project I created in the editor last week that worked. Here is the exact line that worked last week:

    test("\uD800\uDC00", "%F0%90%80%80");

[scheglov]

1. Well, I just don't believe in miracles. If DartScanner was not changed, then its behavior was same. What you may be saw is that there was bug in Editor which masked displaying errors (not this specific - all errors and warnings) and now this bug was fixed.
2. I think only indisputable authority is spec, not just some implementation. Spec has \u{hex_digits}, but not surrogate pairs. I've asked Gilad to clarify this issue.

[scheglov]

Added NeedsInfo label.

[gbracha]

As the spec stands, the the hex sequence must be a valid Unicode scalar value. Apparently D800 is not such a value. So the code is illegal.

If this works in one implementation and not another, then we may have bugs - but fixing them will simply ensure that this code is uniformly rejected.

The real question is does it make sense to change the spec (and to what?). I will consult with our local Unicode gurus tomorrow.

[scheglov]

VM was fixed to reject such code.
http://codereview.chromium.org/10388179/

[scheglov]

Added AsDesigned label.