Skip to content

Support separators in floating point literals #1385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 13, 2020
Merged

Support separators in floating point literals #1385

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
82d64d8
Support separators in floating point literals
invalid-email-address Jul 10, 2020
bc05fbf
Add missing return type and use replaceAll
invalid-email-address Jul 10, 2020
d889fd1
@ts-ignore a replaceAll call
invalid-email-address Jul 10, 2020
0b91fe9
Update ts target to esnext
invalid-email-address Jul 10, 2020
6c2440e
Disallow leading 0_
invalid-email-address Jul 10, 2020
8117c06
Add ESNext libs to tsconfig
invalid-email-address Jul 10, 2020
bd6b5eb
Optimize tokenizing floats with separators
invalid-email-address Jul 11, 2020
3e709c3
Move esnext libs to portable compiler options
invalid-email-address Jul 11, 2020
ce27f4e
Merge branch 'master' into float-separators
invalid-email-address Jul 11, 2020
21b1bb1
Add integer to count separators
invalid-email-address Jul 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions NOTICE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ under the licensing terms detailed in LICENSE:
* Julien Letellier <[email protected]>
* Guido Zuidhof <[email protected]>
* ncave <[email protected]>
* Andrew Davis <[email protected]>

Portions of this software are derived from third-party works licensed under
the following terms:
Expand Down
81 changes: 61 additions & 20 deletions src/tokenizer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1494,37 +1494,78 @@ export class Tokenizer extends DiagnosticEmitter {
}

readDecimalFloat(): f64 {
// TODO: numeric separators (parseFloat can't handle these)
var text = this.source.text;
var pos = this.pos;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This local variable pos was added in #1376 , but made using a helper function awkward. I could either:

  • leave things how I have them now
  • use a local pos and do this.pos = pos before each call to the helper and pos = this.pos after each call to the helper
  • eliminate the helper function and just inline it all in this function.
  • pass a start argument into the helper (but that still leaves the problem of returning the end position, since I am already using the return value for indicating whether a separator is present).

Thoughts?

Copy link
Member

@MaxGraey MaxGraey Jul 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's caching this.pos is made some thing little complicated but it's speedup self-bootstrap version of AS. I suggest leave all as is because that cacheing most significant only for unsafeNext which changes really rarely. Other methods can leave as is for simplicity

var end = this.end;
var start = pos;
while (pos < end && isDecimalDigit(text.charCodeAt(pos))) {
++pos;
}
if (pos < end && text.charCodeAt(pos) == CharCode.DOT) {
++pos;
while (pos < end && isDecimalDigit(text.charCodeAt(pos))) {
++pos;
}
var start = this.pos;
var sepCount = 0;

sepCount += this.readDecimalFloatPartial(false);
if (this.pos < end && text.charCodeAt(this.pos) == CharCode.DOT) {
++this.pos;
sepCount += this.readDecimalFloatPartial();
}
if (pos < end) {
let c = text.charCodeAt(pos);
if (this.pos < end) {
let c = text.charCodeAt(this.pos);
if ((c | 32) == CharCode.e) {
if (
++pos < end &&
(c = text.charCodeAt(pos)) == CharCode.MINUS || c == CharCode.PLUS &&
isDecimalDigit(text.charCodeAt(pos + 1))
++this.pos < end &&
(c = text.charCodeAt(this.pos)) == CharCode.MINUS || c == CharCode.PLUS &&
isDecimalDigit(text.charCodeAt(this.pos + 1))
) {
++pos;
++this.pos;
}
while (pos < end && isDecimalDigit(text.charCodeAt(pos))) {
++pos;
sepCount += this.readDecimalFloatPartial();
}
}
let result = text.substring(start, this.pos);
if (sepCount > 0) result = result.replaceAll("_", "");
return parseFloat(result);
}

/** Reads past one section of a decimal float literal. Returns the number of separators encountered. */
private readDecimalFloatPartial(allowLeadingZeroSep: bool = true): u32 {
var text = this.source.text;
var pos = this.pos;
var start = pos;
var end = this.end;
var sepEnd = start;
var sepCount = 0;

while (pos < end) {
let c = text.charCodeAt(pos);

if (c == CharCode._) {
if (sepEnd == pos) {
this.error(
sepEnd == start
? DiagnosticCode.Numeric_separators_are_not_allowed_here
: DiagnosticCode.Multiple_consecutive_numeric_separators_are_not_permitted,
this.range(pos)
);
} else if (!allowLeadingZeroSep && pos - 1 == start && text.charCodeAt(pos - 1) == CharCode._0) {
this.error(
DiagnosticCode.Numeric_separators_are_not_allowed_here,
this.range(pos)
);
}
sepEnd = pos + 1;
++sepCount;
} else if (!isDecimalDigit(c)) {
break;
}

++pos;
}

if (pos != start && sepEnd == pos) {
this.error(
DiagnosticCode.Numeric_separators_are_not_allowed_here,
this.range(sepEnd - 1)
);
}

this.pos = pos;
return parseFloat(text.substring(start, pos));
return sepCount;
}

readHexFloat(): f64 {
Expand Down
2 changes: 1 addition & 1 deletion src/tsconfig.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"outDir": "../out",
"allowJs": false,
"sourceMap": true,
"target": "ES2016",
"target": "esnext",
"strict": true
},
"include": [
Expand Down
3 changes: 2 additions & 1 deletion std/portable.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
"downlevelIteration": true,
"preserveConstEnums": true,
"typeRoots": [ "types" ],
"types": [ "portable" ]
"types": [ "portable" ],
"lib": ["esnext", "esnext.string"]
}
}
22 changes: 22 additions & 0 deletions tests/parser/numeric-separators.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@
0b01_01_01;
0o12_12_12;
0x23_23_23;
1_000_000.1234_1234;
1_0e1_0;
1_000_000e-1_0;
0.0_0;
1_0e0_0;
1_0e0_1;

// error cases that should still continue parsing:

Expand All @@ -16,3 +22,19 @@

0x23_23_23_; // 6188
0x23__23_23; // 6189

1000_.1234; // 6188
1000._1234; // 6188
1000.1234_; // 6188

10__00.1234; // 6189
1000.12__34; // 6189

1_e2; // 6188
1e_2; // 6188
1e2_; // 6188
1e-1__0; // 6189

0_0.0; // 6188
0_0.0_0; // 6188
0_0e0_0; // 6188
42 changes: 36 additions & 6 deletions tests/parser/numeric-separators.ts.fixture.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@
21;
41610;
2302755;
1000000.12341234;
100000000000;
0.0001;
0;
10;
100;
111111;
111111;
21;
Expand All @@ -10,11 +16,35 @@
41610;
2302755;
2302755;
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(8,9+0)
// ERROR 6189: "Multiple consecutive numeric separators are not permitted." in numeric-separators.ts(9,4+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(11,11+0)
// ERROR 6189: "Multiple consecutive numeric separators are not permitted." in numeric-separators.ts(12,6+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(14,11+0)
// ERROR 6189: "Multiple consecutive numeric separators are not permitted." in numeric-separators.ts(15,6+0)
1000.1234;
1000.1234;
1000.1234;
1000.1234;
1000.1234;
100;
100;
100;
1e-10;
0;
0;
0;
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(14,9+0)
// ERROR 6189: "Multiple consecutive numeric separators are not permitted." in numeric-separators.ts(15,4+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(17,11+0)
// ERROR 6189: "Multiple consecutive numeric separators are not permitted." in numeric-separators.ts(18,6+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(20,11+0)
// ERROR 6189: "Multiple consecutive numeric separators are not permitted." in numeric-separators.ts(21,6+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(23,11+0)
// ERROR 6189: "Multiple consecutive numeric separators are not permitted." in numeric-separators.ts(24,6+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(26,5+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(27,6+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(28,10+0)
// ERROR 6189: "Multiple consecutive numeric separators are not permitted." in numeric-separators.ts(30,4+0)
// ERROR 6189: "Multiple consecutive numeric separators are not permitted." in numeric-separators.ts(31,9+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(33,2+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(34,3+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(35,4+0)
// ERROR 6189: "Multiple consecutive numeric separators are not permitted." in numeric-separators.ts(36,6+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(38,2+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(39,2+0)
// ERROR 6188: "Numeric separators are not allowed here." in numeric-separators.ts(40,2+0)