Skip to content

Perform structural fallbacks for reliable variance results less liberally #54754

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Perform structural fallbacks for reliable variance results less liberally #54754

wants to merge 1 commit into from
+21 −25

Conversation

weswigham
Copy link
Member

Only performing structural fallbacks of variance-based comparisons if the comparison that actually failed is the unreliable/unmeasurable one allows us to utilize variance results more often, which should be good for performance.

This does not, ultimately, fix #54542 in the case of lodash, unfortunately (I had to narrow the change a bit to fix an issue some tests caught, so I'm still back to looking at other ways to attack the underlying issue of "way too many instantiations required to check an insanely complicated type hierarchy"), but may fix (well, hide, really) other reports.

@weswigham
Copy link
Member Author

@typescript-bot test this
@typescript-bot run dt
@typescript-bot test top100
@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jun 23, 2023 •
edited
Loading

Heya @weswigham, I've started to run the diff-based top-repos suite on this PR at 33729e6. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jun 23, 2023 •
edited
Loading

Heya @weswigham, I've started to run the parallelized Definitely Typed test suite on this PR at 33729e6. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jun 23, 2023 •
edited
Loading

Heya @weswigham, I've started to run the extended test suite on this PR at 33729e6. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jun 23, 2023 •
edited
Loading

Heya @weswigham, I've started to run the perf test suite on this PR at 33729e6. You can monitor the build here.

Update: The results are in!

@typescript-bot typescript-bot added the For Milestone Bug PRs that fix a bug with a specific milestone label Jun 23, 2023
@typescript-bot
Copy link
Collaborator

@weswigham
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - main..54754
Metric main 54754 Delta Best Worst p-value
Angular - node (v18.10.0, x64)
Memory used 366,611k (± 0.00%) 366,620k (± 0.01%) ~ 366,570k 366,669k p=0.872 n=6
Parse Time 3.41s (± 0.54%) 3.43s (± 0.35%) ~ 3.41s 3.44s p=0.101 n=6
Bind Time 1.12s (± 0.00%) 1.12s (± 0.36%) ~ 1.11s 1.12s p=0.405 n=6
Check Time 8.86s (± 0.41%) 8.84s (± 0.45%) ~ 8.80s 8.90s p=0.420 n=6
Emit Time 7.48s (± 0.60%) 7.50s (± 0.75%) ~ 7.44s 7.58s p=0.747 n=6
Total Time 20.87s (± 0.27%) 20.89s (± 0.42%) ~ 20.81s 21.05s p=0.872 n=6
Compiler-Unions - node (v18.10.0, x64)
Memory used 190,959k (± 0.01%) 191,907k (± 1.20%) ~ 190,941k 196,601k p=0.748 n=6
Parse Time 1.49s (± 0.55%) 1.50s (± 0.81%) ~ 1.48s 1.51s p=0.677 n=6
Bind Time 0.77s (± 0.98%) 0.77s (± 0.82%) ~ 0.76s 0.78s p=0.718 n=6
Check Time 9.42s (± 0.62%) 9.46s (± 0.41%) ~ 9.40s 9.51s p=0.258 n=6
Emit Time 2.73s (± 1.06%) 2.76s (± 1.03%) ~ 2.73s 2.81s p=0.072 n=6
Total Time 14.41s (± 0.35%) 14.48s (± 0.31%) +0.07s (+ 0.50%) 14.41s 14.55s p=0.045 n=6
Monaco - node (v18.10.0, x64)
Memory used 346,796k (± 0.01%) 346,788k (± 0.00%) ~ 346,764k 346,809k p=0.298 n=6
Parse Time 2.58s (± 0.80%) 2.59s (± 0.82%) ~ 2.56s 2.61s p=0.572 n=6
Bind Time 1.00s (± 0.54%) 1.00s (± 0.83%) ~ 1.00s 1.02s p=0.855 n=6
Check Time 7.19s (± 0.48%) 7.16s (± 0.63%) ~ 7.11s 7.23s p=0.127 n=6
Emit Time 4.26s (± 0.83%) 4.25s (± 0.87%) ~ 4.22s 4.32s p=0.462 n=6
Total Time 15.05s (± 0.40%) 15.01s (± 0.48%) ~ 14.96s 15.15s p=0.172 n=6
TFS - node (v18.10.0, x64)
Memory used 300,866k (± 0.00%) 300,853k (± 0.01%) ~ 300,828k 300,865k p=0.375 n=6
Parse Time 2.07s (± 1.53%) 2.08s (± 0.72%) ~ 2.06s 2.09s p=1.000 n=6
Bind Time 1.14s (± 1.03%) 1.14s (± 1.47%) ~ 1.12s 1.17s p=0.933 n=6
Check Time 6.64s (± 0.48%) 6.65s (± 0.52%) ~ 6.59s 6.68s p=0.570 n=6
Emit Time 3.86s (± 1.42%) 3.86s (± 0.28%) ~ 3.85s 3.88s p=0.936 n=6
Total Time 13.71s (± 0.60%) 13.73s (± 0.30%) ~ 13.66s 13.78s p=0.471 n=6
material-ui - node (v18.10.0, x64)
Memory used 482,313k (± 0.01%) 482,368k (± 0.02%) ~ 482,243k 482,448k p=0.297 n=6
Parse Time 3.11s (± 0.59%) 3.13s (± 0.62%) ~ 3.11s 3.16s p=0.254 n=6
Bind Time 0.92s (± 0.89%) 0.92s (± 1.07%) ~ 0.90s 0.93s p=0.604 n=6
Check Time 16.99s (± 0.75%) 17.04s (± 1.07%) ~ 16.87s 17.26s p=0.748 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 21.02s (± 0.63%) 21.09s (± 0.96%) ~ 20.89s 21.34s p=0.810 n=6
xstate - node (v18.10.0, x64)
Memory used 564,156k (± 0.02%) 561,492k (± 0.02%) -2,664k (- 0.47%) 561,344k 561,687k p=0.005 n=6
Parse Time 3.83s (± 0.49%) 3.84s (± 0.39%) ~ 3.83s 3.86s p=0.210 n=6
Bind Time 1.63s (± 1.31%) 1.64s (± 0.74%) ~ 1.62s 1.65s p=0.566 n=6
Check Time 2.80s (± 0.42%) 2.70s (± 0.69%) 🟩-0.09s (- 3.34%) 2.68s 2.73s p=0.005 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 0.00%) ~ 0.08s 0.08s p=1.000 n=6
Total Time 8.34s (± 0.40%) 8.27s (± 0.36%) -0.07s (- 0.84%) 8.23s 8.31s p=0.013 n=6
Angular - node (v16.17.1, x64)
Memory used 366,001k (± 0.01%) 365,989k (± 0.01%) ~ 365,960k 366,038k p=0.689 n=6
Parse Time 3.59s (± 0.49%) 3.59s (± 0.70%) ~ 3.54s 3.61s p=0.935 n=6
Bind Time 1.18s (± 0.35%) 1.19s (± 0.63%) +0.01s (+ 0.85%) 1.18s 1.20s p=0.024 n=6
Check Time 9.63s (± 0.60%) 9.67s (± 0.49%) ~ 9.59s 9.73s p=0.332 n=6
Emit Time 8.00s (± 0.67%) 8.06s (± 0.71%) ~ 7.97s 8.13s p=0.125 n=6
Total Time 22.41s (± 0.30%) 22.50s (± 0.46%) ~ 22.36s 22.59s p=0.128 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 192,851k (± 0.01%) 192,829k (± 0.04%) ~ 192,687k 192,870k p=0.936 n=6
Parse Time 1.59s (± 1.63%) 1.58s (± 2.06%) ~ 1.54s 1.62s p=1.000 n=6
Bind Time 0.82s (± 0.77%) 0.82s (± 0.63%) ~ 0.81s 0.82s p=0.386 n=6
Check Time 10.10s (± 0.80%) 10.12s (± 0.73%) ~ 9.99s 10.19s p=0.575 n=6
Emit Time 3.01s (± 0.62%) 3.02s (± 0.91%) ~ 2.98s 3.06s p=0.624 n=6
Total Time 15.51s (± 0.71%) 15.53s (± 0.56%) ~ 15.37s 15.61s p=1.000 n=6
Monaco - node (v16.17.1, x64)
Memory used 346,061k (± 0.01%) 346,067k (± 0.01%) ~ 346,029k 346,113k p=0.936 n=6
Parse Time 2.74s (± 0.75%) 2.75s (± 0.81%) ~ 2.72s 2.78s p=0.406 n=6
Bind Time 1.09s (± 0.69%) 1.09s (± 1.61%) ~ 1.07s 1.12s p=0.397 n=6
Check Time 7.88s (± 0.41%) 7.90s (± 0.21%) ~ 7.88s 7.92s p=0.075 n=6
Emit Time 4.45s (± 0.46%) 4.51s (± 1.15%) +0.05s (+ 1.24%) 4.45s 4.59s p=0.044 n=6
Total Time 16.16s (± 0.32%) 16.25s (± 0.50%) ~ 16.13s 16.36s p=0.066 n=6
TFS - node (v16.17.1, x64)
Memory used 300,205k (± 0.01%) 300,189k (± 0.01%) ~ 300,159k 300,212k p=0.128 n=6
Parse Time 2.16s (± 0.25%) 2.19s (± 1.20%) +0.03s (+ 1.55%) 2.17s 2.24s p=0.004 n=6
Bind Time 1.24s (± 0.79%) 1.24s (± 0.72%) ~ 1.23s 1.25s p=0.798 n=6
Check Time 7.33s (± 0.46%) 7.31s (± 0.37%) ~ 7.28s 7.36s p=0.568 n=6
Emit Time 4.33s (± 0.78%) 4.34s (± 0.75%) ~ 4.29s 4.39s p=0.414 n=6
Total Time 15.06s (± 0.37%) 15.08s (± 0.45%) ~ 15.00s 15.19s p=0.873 n=6
material-ui - node (v16.17.1, x64)
Memory used 481,602k (± 0.01%) 481,548k (± 0.01%) ~ 481,490k 481,612k p=0.093 n=6
Parse Time 3.25s (± 0.23%) 3.25s (± 0.19%) ~ 3.24s 3.26s p=0.718 n=6
Bind Time 0.95s (± 0.54%) 0.95s (± 0.79%) ~ 0.94s 0.96s p=0.241 n=6
Check Time 18.02s (± 0.35%) 18.00s (± 0.63%) ~ 17.84s 18.16s p=0.872 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.22s (± 0.28%) 22.21s (± 0.52%) ~ 22.04s 22.37s p=0.873 n=6
xstate - node (v16.17.1, x64)
Memory used 561,807k (± 0.02%) 559,013k (± 0.02%) -2,795k (- 0.50%) 558,901k 559,130k p=0.005 n=6
Parse Time 4.00s (± 0.21%) 4.02s (± 0.49%) ~ 3.99s 4.05s p=0.145 n=6
Bind Time 1.73s (± 0.77%) 1.73s (± 0.37%) ~ 1.72s 1.74s p=0.720 n=6
Check Time 3.06s (± 0.56%) 2.93s (± 1.03%) 🟩-0.12s (- 4.09%) 2.89s 2.97s p=0.005 n=6
Emit Time 0.09s (± 4.45%) 0.09s (± 0.00%) ~ 0.09s 0.09s p=0.405 n=6
Total Time 8.88s (± 0.32%) 8.77s (± 0.32%) -0.11s (- 1.29%) 8.74s 8.82s p=0.005 n=6
Angular - node (v14.21.3, x64)
Memory used 360,039k (± 0.01%) 360,025k (± 0.01%) ~ 359,968k 360,101k p=0.575 n=6
Parse Time 3.67s (± 0.58%) 3.69s (± 0.53%) ~ 3.67s 3.72s p=0.145 n=6
Bind Time 1.22s (± 0.00%) 1.22s (± 0.33%) ~ 1.21s 1.22s p=0.405 n=6
Check Time 10.11s (± 0.42%) 10.05s (± 0.29%) -0.06s (- 0.61%) 10.02s 10.10s p=0.030 n=6
Emit Time 8.38s (± 0.61%) 8.41s (± 0.93%) ~ 8.29s 8.51s p=0.470 n=6
Total Time 23.38s (± 0.25%) 23.37s (± 0.33%) ~ 23.28s 23.47s p=0.688 n=6
Compiler-Unions - node (v14.21.3, x64)
Memory used 188,143k (± 0.01%) 188,141k (± 0.01%) ~ 188,111k 188,173k p=1.000 n=6
Parse Time 1.62s (± 0.61%) 1.62s (± 0.39%) ~ 1.61s 1.63s p=0.733 n=6
Bind Time 0.85s (± 0.89%) 0.84s (± 0.61%) ~ 0.84s 0.85s p=0.247 n=6
Check Time 10.28s (± 0.55%) 10.28s (± 0.71%) ~ 10.16s 10.37s p=1.000 n=6
Emit Time 3.16s (± 0.97%) 3.17s (± 1.42%) ~ 3.10s 3.21s p=0.573 n=6
Total Time 15.91s (± 0.53%) 15.91s (± 0.59%) ~ 15.81s 16.06s p=1.000 n=6
Monaco - node (v14.21.3, x64)
Memory used 341,198k (± 0.00%) 341,182k (± 0.01%) ~ 341,147k 341,226k p=0.229 n=6
Parse Time 2.80s (± 1.09%) 2.83s (± 0.37%) ~ 2.81s 2.84s p=0.145 n=6
Bind Time 1.12s (± 0.49%) 1.11s (± 1.05%) ~ 1.10s 1.13s p=0.498 n=6
Check Time 8.22s (± 0.63%) 8.20s (± 0.63%) ~ 8.15s 8.29s p=0.747 n=6
Emit Time 4.71s (± 0.61%) 4.69s (± 0.86%) ~ 4.65s 4.75s p=0.374 n=6
Total Time 16.85s (± 0.24%) 16.83s (± 0.38%) ~ 16.76s 16.93s p=0.332 n=6
TFS - node (v14.21.3, x64)
Memory used 295,299k (± 0.00%) 295,305k (± 0.00%) ~ 295,302k 295,309k p=0.076 n=6
Parse Time 2.40s (± 1.10%) 2.40s (± 1.33%) ~ 2.37s 2.45s p=0.935 n=6
Bind Time 1.07s (± 0.48%) 1.07s (± 0.92%) ~ 1.06s 1.08s p=0.931 n=6
Check Time 7.66s (± 0.22%) 7.66s (± 0.53%) ~ 7.61s 7.70s p=1.000 n=6
Emit Time 4.30s (± 0.31%) 4.33s (± 1.07%) ~ 4.29s 4.40s p=0.513 n=6
Total Time 15.43s (± 0.23%) 15.46s (± 0.63%) ~ 15.33s 15.56s p=0.470 n=6
material-ui - node (v14.21.3, x64)
Memory used 477,105k (± 0.00%) 477,082k (± 0.01%) ~ 477,027k 477,117k p=0.378 n=6
Parse Time 3.33s (± 0.83%) 3.34s (± 0.36%) ~ 3.32s 3.35s p=0.627 n=6
Bind Time 1.00s (± 0.54%) 1.01s (± 0.74%) ~ 1.00s 1.02s p=0.137 n=6
Check Time 18.85s (± 0.50%) 19.15s (± 0.56%) +0.30s (+ 1.61%) 19.03s 19.30s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 23.18s (± 0.47%) 23.50s (± 0.48%) +0.31s (+ 1.34%) 23.39s 23.67s p=0.005 n=6
xstate - node (v14.21.3, x64)
Memory used 550,717k (± 0.00%) 548,046k (± 0.00%) -2,670k (- 0.48%) 548,031k 548,069k p=0.005 n=6
Parse Time 4.24s (± 0.55%) 4.25s (± 0.51%) ~ 4.23s 4.28s p=0.518 n=6
Bind Time 1.68s (± 2.02%) 1.66s (± 2.23%) ~ 1.61s 1.69s p=0.461 n=6
Check Time 3.14s (± 0.62%) 3.00s (± 0.57%) 🟩-0.14s (- 4.51%) 2.98s 3.03s p=0.005 n=6
Emit Time 0.10s (± 5.34%) 0.09s (± 0.00%) 🟩-0.01s (- 6.90%) 0.09s 0.09s p=0.025 n=6
Total Time 9.15s (± 0.39%) 9.01s (± 0.53%) -0.14s (- 1.57%) 8.95s 9.08s p=0.005 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.21.3, x64)
Scenarios
  • Angular - node (v18.10.0, x64)
  • Angular - node (v16.17.1, x64)
  • Angular - node (v14.21.3, x64)
  • Compiler-Unions - node (v18.10.0, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Compiler-Unions - node (v14.21.3, x64)
  • Monaco - node (v18.10.0, x64)
  • Monaco - node (v16.17.1, x64)
  • Monaco - node (v14.21.3, x64)
  • TFS - node (v18.10.0, x64)
  • TFS - node (v16.17.1, x64)
  • TFS - node (v14.21.3, x64)
  • material-ui - node (v18.10.0, x64)
  • material-ui - node (v16.17.1, x64)
  • material-ui - node (v14.21.3, x64)
  • xstate - node (v18.10.0, x64)
  • xstate - node (v16.17.1, x64)
  • xstate - node (v14.21.3, x64)
Benchmark Name Iterations
Current 54754 6
Baseline main 6

TSServer

Comparison Report - main..54754
Metric main 54754 Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,558ms (± 0.70%) 2,552ms (± 0.51%) ~ 2,535ms 2,570ms p=0.689 n=6
Req 2 - geterr 5,344ms (± 0.93%) 5,338ms (± 0.69%) ~ 5,296ms 5,390ms p=0.936 n=6
Req 3 - references 338ms (± 0.63%) 341ms (± 0.68%) +3ms (+ 0.99%) 338ms 344ms p=0.043 n=6
Req 4 - navto 288ms (± 0.18%) 287ms (± 0.34%) ~ 286ms 288ms p=0.417 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 81ms (± 3.27%) 83ms (± 3.19%) ~ 80ms 86ms p=0.122 n=6
CompilerTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,648ms (± 1.26%) 2,639ms (± 0.92%) ~ 2,607ms 2,674ms p=0.810 n=6
Req 2 - geterr 4,094ms (± 0.41%) 4,096ms (± 0.34%) ~ 4,081ms 4,113ms p=0.630 n=6
Req 3 - references 348ms (± 0.41%) 348ms (± 0.67%) ~ 345ms 352ms p=0.871 n=6
Req 4 - navto 286ms (± 0.36%) 288ms (± 0.22%) +2ms (+ 0.58%) 287ms 289ms p=0.019 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 64ms (± 3.09%) 64ms (± 3.19%) ~ 63ms 68ms p=0.655 n=6
xstateTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 3,102ms (± 0.67%) 3,100ms (± 0.45%) ~ 3,081ms 3,123ms p=1.000 n=6
Req 2 - geterr 1,567ms (± 1.64%) 1,505ms (± 0.63%) 🟩-62ms (- 3.96%) 1,489ms 1,516ms p=0.005 n=6
Req 3 - references 113ms (± 0.67%) 115ms (± 1.19%) +2ms (+ 1.62%) 113ms 116ms p=0.037 n=6
Req 4 - navto 371ms (± 1.28%) 380ms (± 0.37%) +10ms (+ 2.56%) 378ms 382ms p=0.019 n=6
Req 5 - completionInfo count 2,870 (± 0.00%) 2,870 (± 0.00%) ~ 2,870 2,870 p=1.000 n=6
Req 5 - completionInfo 372ms (± 2.29%) 370ms (± 1.68%) ~ 359ms 378ms p=0.936 n=6
Compiler-UnionsTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,650ms (± 0.47%) 2,652ms (± 0.90%) ~ 2,614ms 2,678ms p=0.572 n=6
Req 2 - geterr 6,017ms (± 0.65%) 6,015ms (± 0.52%) ~ 5,977ms 6,067ms p=0.936 n=6
Req 3 - references 353ms (± 0.90%) 352ms (± 0.83%) ~ 347ms 355ms p=0.936 n=6
Req 4 - navto 285ms (± 1.22%) 288ms (± 2.11%) ~ 281ms 296ms p=0.466 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 91ms (± 1.96%) 91ms (± 4.86%) ~ 82ms 93ms p=0.210 n=6
CompilerTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,836ms (± 0.24%) 2,826ms (± 0.26%) ~ 2,817ms 2,834ms p=0.063 n=6
Req 2 - geterr 4,666ms (± 0.28%) 4,663ms (± 0.40%) ~ 4,630ms 4,681ms p=0.936 n=6
Req 3 - references 364ms (± 0.87%) 364ms (± 0.64%) ~ 361ms 367ms p=1.000 n=6
Req 4 - navto 283ms (± 0.47%) 283ms (± 0.88%) ~ 278ms 285ms p=0.869 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 67ms (± 0.77%) 68ms (± 1.24%) ~ 66ms 68ms p=0.533 n=6
xstateTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 3,238ms (± 0.58%) 3,235ms (± 0.18%) ~ 3,230ms 3,246ms p=1.000 n=6
Req 2 - geterr 1,724ms (± 0.62%) 1,667ms (± 0.86%) 🟩-57ms (- 3.31%) 1,639ms 1,680ms p=0.005 n=6
Req 3 - references 124ms (± 1.10%) 130ms (± 8.62%) ~ 120ms 148ms p=1.000 n=6
Req 4 - navto 352ms (± 0.76%) 367ms (± 1.67%) +16ms (+ 4.45%) 358ms 374ms p=0.005 n=6
Req 5 - completionInfo count 2,870 (± 0.00%) 2,870 (± 0.00%) ~ 2,870 2,870 p=1.000 n=6
Req 5 - completionInfo 412ms (± 2.49%) 410ms (± 1.68%) ~ 398ms 417ms p=0.470 n=6
Compiler-UnionsTSServer - node (v14.21.3, x64)
Req 1 - updateOpen 2,789ms (± 0.35%) 2,800ms (± 0.34%) ~ 2,785ms 2,811ms p=0.077 n=6
Req 2 - geterr 6,186ms (± 0.88%) 6,175ms (± 0.48%) ~ 6,132ms 6,216ms p=0.575 n=6
Req 3 - references 363ms (± 1.04%) 362ms (± 0.56%) ~ 359ms 365ms p=0.745 n=6
Req 4 - navto 291ms (± 0.35%) 292ms (± 0.78%) ~ 290ms 296ms p=0.366 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 104ms (± 0.72%) 102ms (± 3.87%) ~ 94ms 104ms p=0.383 n=6
CompilerTSServer - node (v14.21.3, x64)
Req 1 - updateOpen 2,981ms (± 0.32%) 2,987ms (± 0.70%) ~ 2,963ms 3,015ms p=1.000 n=6
Req 2 - geterr 4,600ms (± 1.90%) 4,563ms (± 1.80%) ~ 4,488ms 4,717ms p=0.298 n=6
Req 3 - references 376ms (± 0.85%) 377ms (± 0.57%) ~ 374ms 379ms p=0.278 n=6
Req 4 - navto 298ms (± 0.47%) 300ms (± 0.46%) ~ 297ms 301ms p=0.101 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 79ms (± 9.11%) 76ms (± 1.36%) ~ 74ms 77ms p=0.357 n=6
xstateTSServer - node (v14.21.3, x64)
Req 1 - updateOpen 3,484ms (± 0.60%) 3,476ms (± 1.02%) ~ 3,435ms 3,526ms p=0.810 n=6
Req 2 - geterr 1,824ms (± 0.21%) 1,752ms (± 0.96%) 🟩-72ms (- 3.95%) 1,733ms 1,780ms p=0.005 n=6
Req 3 - references 141ms (± 7.82%) 150ms (± 9.63%) ~ 132ms 164ms p=0.378 n=6
Req 4 - navto 399ms (± 0.14%) 392ms (± 1.12%) -7ms (- 1.67%) 386ms 397ms p=0.004 n=6
Req 5 - completionInfo count 2,870 (± 0.00%) 2,870 (± 0.00%) ~ 2,870 2,870 p=1.000 n=6
Req 5 - completionInfo 435ms (± 2.15%) 425ms (± 1.95%) ~ 420ms 441ms p=0.167 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.21.3, x64)
Scenarios
  • Compiler-UnionsTSServer - node (v18.10.0, x64)
  • Compiler-UnionsTSServer - node (v16.17.1, x64)
  • Compiler-UnionsTSServer - node (v14.21.3, x64)
  • CompilerTSServer - node (v18.10.0, x64)
  • CompilerTSServer - node (v16.17.1, x64)
  • CompilerTSServer - node (v14.21.3, x64)
  • xstateTSServer - node (v18.10.0, x64)
  • xstateTSServer - node (v16.17.1, x64)
  • xstateTSServer - node (v14.21.3, x64)
Benchmark Name Iterations
Current 54754 6
Baseline main 6

Startup

Comparison Report - main..54754
Metric main 54754 Delta Best Worst p-value
tsc-startup - node (v16.17.1, x64)
Execution time 142.32ms (± 0.19%) 142.75ms (± 0.22%) +0.43ms (+ 0.31%) 142.04ms 148.73ms p=0.000 n=600
tsserver-startup - node (v16.17.1, x64)
Execution time 221.44ms (± 0.18%) 222.45ms (± 0.19%) +1.02ms (+ 0.46%) 221.34ms 228.36ms p=0.000 n=600
tsserverlibrary-startup - node (v16.17.1, x64)
Execution time 223.19ms (± 0.18%) 223.91ms (± 0.21%) +0.72ms (+ 0.32%) 222.75ms 231.81ms p=0.000 n=600
typescript-startup - node (v16.17.1, x64)
Execution time 204.86ms (± 0.15%) 205.81ms (± 0.21%) +0.95ms (+ 0.46%) 204.61ms 213.66ms p=0.000 n=600
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • tsc-startup - node (v16.17.1, x64)
  • tsserver-startup - node (v16.17.1, x64)
  • tsserverlibrary-startup - node (v16.17.1, x64)
  • typescript-startup - node (v16.17.1, x64)
Benchmark Name Iterations
Current 54754 6
Baseline main 6

Developer Information:

Download Benchmark

@typescript-bot
Copy link
Collaborator

@weswigham Here are the results of running the top-repos suite comparing main and refs/pull/54754/merge:

Something interesting changed - please have a look.

Details

microsoft/vscode

5 of 53 projects failed to build with the old tsc and were ignored

src/tsconfig.tsec.json

  • error TS2352: Conversion of type 'ITestItemChildren<TestItem>' to type 'ITestChildrenLike<TestItemImpl>' may be a mistake because neither type sufficiently overlaps with the other. If this was intentional, convert the expression to 'unknown' first.

react-hook-form/react-hook-form

2 of 3 projects failed to build with the old tsc and were ignored

tsconfig.json

  • error TS2322: Type '{ formState: FormState<TFieldValues>; watch: UseFormWatch<TFieldValues>; getValues: UseFormGetValues<TFieldValues>; ... 11 more ...; setFocus: UseFormSetFocus<...>; }' is not assignable to type 'UseFormReturn<TFieldValues, TContext, TTransformedValues>'.
  • error TS18048: '_formControl.current' is possibly 'undefined'.
  • error TS2322: Type 'UseFormReturn<TFieldValues, TContext, TTransformedValues> | undefined' is not assignable to type 'UseFormReturn<TFieldValues, TContext, TTransformedValues>'.

statelyai/xstate

12 of 29 projects failed to build with the old tsc and were ignored

packages/xstate-inspect/tsconfig.json

packages/xstate-test/tsconfig.json

@typescript-bot
Copy link
Collaborator

Hey @weswigham, the results of running the DT tests are ready.
There were interesting changes:

Branch only errors:

Package: backbone-relational
Error:

Error: Errors in typescript@local for external dependencies:
../backbone/index.d.ts(226,32): error TS2344: Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: wordpress__admin
Error:

Error: Errors in typescript@local for external dependencies:
../backbone/index.d.ts(226,32): error TS2344: Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: alloy
Error:

Error: Errors in typescript@local for external dependencies:
../backbone/index.d.ts(226,32): error TS2344: Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: backbone
Error:

Error: /home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/DefinitelyTyped/types/backbone/index.d.ts:226:32
ERROR: 226:32  expect  TypeScript@local compile error: 
Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: backbone.layoutmanager
Error:

Error: Errors in typescript@local for external dependencies:
../backbone/index.d.ts(226,32): error TS2344: Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: forms
Error:

Error: /home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/DefinitelyTyped/types/forms/forms-tests.ts:86:7
ERROR: 86:7  expect  TypeScript@local compile error: 
Type 'FormBound<{ name: Field<string>; email: Field<string>; }, Partial<FormData<{ name: Field<string>; email: Field<string>; }>>>' is not assignable to type 'FormBound<{ name: Field<string>; email: Field<string>; }, FormData<{ name: Field<string>; email: Field<string>; }>>'.
  Type 'Partial<FormData<{ name: Field<string>; email: Field<string>; }>>' is not assignable to type 'FormData<{ name: Field<string>; email: Field<string>; }>'.
    Types of property 'name' are incompatible.
      Type 'string | undefined' is not assignable to type 'string'.
        Type 'undefined' is not assignable to type 'string'.
ERROR: 89:5  expect  TypeScript@local compile error: 
Type '(form: FormBound<{ name: Field<string>; email: Field<string>; }, FormData<{ name: Field<string>; email: Field<string>; }>>) => void' is not assignable to type 'FormHandleCallback<{ name: Field<string>; email: Field<string>; }, { name: string; }>'.
  Types of parameters 'form' and 'form' are incompatible.
    Type 'FormBound<{ name: Field<string>; email: Field<string>; }, { name: string; }>' is not assignable to type 'FormBound<{ name: Field<string>; email: Field<string>; }, FormData<{ name: Field<string>; email: Field<string>; }>>'.
      Property 'email' is missing in type '{ name: string; }' but required in type 'FormData<{ name: Field<string>; email: Field<string>; }>'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: knockback
Error:

Error: Errors in typescript@local for external dependencies:
../backbone/index.d.ts(226,32): error TS2344: Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: backbone-associations
Error:

Error: Errors in typescript@local for external dependencies:
../backbone/index.d.ts(226,32): error TS2344: Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: backbone.localstorage
Error:

Error: Errors in typescript@local for external dependencies:
../backbone/index.d.ts(226,32): error TS2344: Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: backgrid
Error:

Error: Errors in typescript@local for external dependencies:
../backbone/index.d.ts(226,32): error TS2344: Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: backbone-fetch-cache
Error:

Error: Errors in typescript@local for external dependencies:
../backbone/index.d.ts(226,32): error TS2344: Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: backbone.marionette
Error:

Error: Errors in typescript@local for external dependencies:
../backbone/index.d.ts(226,32): error TS2344: Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: giraffe
Error:

Error: Errors in typescript@local for external dependencies:
../backbone/index.d.ts(226,32): error TS2344: Type 'this' does not satisfy the constraint 'Model<any, ModelSetOptions, any>'.
  Type 'Model<T, S, E>' is not assignable to type 'Model<any, ModelSetOptions, any>'.
    Type 'S' is not assignable to type 'ModelSetOptions'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

Package: mongoose-promise
Error:

Error: Errors in typescript@local for external dependencies:
node_modules/@types/mongodb/index.d.ts(1944,54): error TS2536: Type 'Key' cannot be used to index type 'WithId<Required<TSchema>>'.
node_modules/@types/mongodb/index.d.ts(1949,54): error TS2536: Type 'Key' cannot be used to index type 'WithId<Required<TSchema>>'.
node_modules/@types/mongodb/index.d.ts(1955,37): error TS2536: Type 'Key' cannot be used to index type 'WithId<Required<TSchema>>'.
node_modules/@types/mongodb/index.d.ts(1960,54): error TS2536: Type 'Key' cannot be used to index type 'WithId<Required<TSchema>>'.

    at testTypesVersion (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:194:15)
    at async runTests (/home/vsts/work/1/s/DefinitelyTyped-tools/packages/dtslint-runner/node_modules/@definitelytyped/dtslint/dist/index.js:151:9)

You can check the log here.

@DanielRosenwasser DanielRosenwasser mentioned this pull request Jun 26, 2023
Merged
@RyanCavanaugh RyanCavanaugh mentioned this pull request Oct 11, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewbranch andrewbranch Awaiting requested review from andrewbranch

@RyanCavanaugh RyanCavanaugh Awaiting requested review from RyanCavanaugh

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

@weswigham weswigham

Labels
Author: Team For Milestone Bug PRs that fix a bug with a specific milestone
Projects
None yet
Milestone
No milestone
2 participants
@weswigham @typescript-bot