Skip to content

Cache expression type when checking assertion #54224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
May 13, 2023
Merged

Conversation

gabritto
Copy link
Member

@gabritto gabritto commented May 12, 2023

Experiment to see if we can revert the perf impact of #53261.

@gabritto
Copy link
Member Author

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 12, 2023

Heya @gabritto, I've started to run the abridged perf test suite on this PR at 1175676. You can monitor the build here.

@gabritto
Copy link
Member Author

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 12, 2023

Heya @gabritto, I've started to run the abridged perf test suite on this PR at 1175676. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@gabritto
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..54224

Metric main 54224 Delta Best Worst p-value
Angular - node (v16.17.1, x64)
Memory used 364,884k (± 0.02%) 365,015k (± 0.02%) +131k (+ 0.04%) 364,873k 365,101k p=0.020 n=6
Parse Time 3.56s (± 0.86%) 3.55s (± 0.43%) ~ 3.53s 3.57s p=0.871 n=6
Bind Time 1.19s (± 0.69%) 1.18s (± 0.46%) ~ 1.18s 1.19s p=0.859 n=6
Check Time 9.60s (± 0.52%) 9.64s (± 0.30%) ~ 9.59s 9.67s p=0.172 n=6
Emit Time 7.96s (± 0.78%) 7.95s (± 0.64%) ~ 7.90s 8.01s p=0.688 n=6
Total Time 22.31s (± 0.42%) 22.33s (± 0.39%) ~ 22.22s 22.44s p=0.810 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,278k (± 0.72%) 192,906k (± 0.04%) ~ 192,805k 193,007k p=0.093 n=6
Parse Time 1.59s (± 1.52%) 1.60s (± 1.35%) ~ 1.56s 1.61s p=1.000 n=6
Bind Time 0.83s (± 0.99%) 0.83s (± 1.08%) ~ 0.82s 0.84s p=0.550 n=6
Check Time 10.34s (± 0.54%) 10.18s (± 1.13%) -0.17s (- 1.61%) 10.04s 10.31s p=0.013 n=6
Emit Time 3.03s (± 1.38%) 3.02s (± 1.07%) ~ 2.99s 3.07s p=0.935 n=6
Total Time 15.79s (± 0.37%) 15.62s (± 0.86%) -0.17s (- 1.07%) 15.43s 15.78s p=0.020 n=6
Monaco - node (v16.17.1, x64)
Memory used 345,854k (± 0.01%) 345,852k (± 0.01%) ~ 345,822k 345,881k p=1.000 n=6
Parse Time 2.73s (± 0.76%) 2.74s (± 0.23%) ~ 2.73s 2.75s p=1.000 n=6
Bind Time 1.09s (± 0.37%) 1.09s (± 0.69%) ~ 1.08s 1.10s p=0.389 n=6
Check Time 7.88s (± 0.25%) 7.84s (± 0.59%) ~ 7.79s 7.90s p=0.196 n=6
Emit Time 4.48s (± 0.99%) 4.46s (± 0.81%) ~ 4.41s 4.51s p=0.376 n=6
Total Time 16.18s (± 0.40%) 16.12s (± 0.48%) ~ 16.01s 16.21s p=0.336 n=6
TFS - node (v16.17.1, x64)
Memory used 300,095k (± 0.01%) 299,968k (± 0.01%) -127k (- 0.04%) 299,936k 299,995k p=0.005 n=6
Parse Time 2.16s (± 0.68%) 2.17s (± 1.09%) ~ 2.14s 2.21s p=0.684 n=6
Bind Time 1.24s (± 0.42%) 1.24s (± 0.94%) ~ 1.22s 1.25s p=0.666 n=6
Check Time 7.32s (± 0.55%) 7.28s (± 0.47%) ~ 7.25s 7.34s p=0.157 n=6
Emit Time 4.35s (± 1.10%) 4.32s (± 0.45%) ~ 4.30s 4.34s p=0.167 n=6
Total Time 15.08s (± 0.42%) 15.01s (± 0.46%) ~ 14.92s 15.10s p=0.147 n=6
material-ui - node (v16.17.1, x64)
Memory used 480,931k (± 0.00%) 480,931k (± 0.01%) ~ 480,905k 480,963k p=1.000 n=6
Parse Time 3.26s (± 0.50%) 3.25s (± 0.58%) ~ 3.23s 3.28s p=0.254 n=6
Bind Time 0.94s (± 0.80%) 0.94s (± 0.43%) ~ 0.94s 0.95s p=1.000 n=6
Check Time 18.11s (± 1.06%) 18.05s (± 1.41%) ~ 17.80s 18.33s p=0.471 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.32s (± 0.86%) 22.25s (± 1.15%) ~ 21.98s 22.51s p=0.575 n=6
xstate - node (v16.17.1, x64)
Memory used 560,454k (± 0.01%) 560,439k (± 0.02%) ~ 560,334k 560,671k p=0.630 n=6
Parse Time 4.02s (± 0.34%) 4.02s (± 0.19%) ~ 4.01s 4.03s p=0.406 n=6
Bind Time 1.76s (± 0.31%) 1.76s (± 0.46%) ~ 1.75s 1.77s p=0.859 n=6
Check Time 3.06s (± 0.57%) 3.06s (± 0.84%) ~ 3.03s 3.09s p=0.936 n=6
Emit Time 0.09s (± 0.00%) 0.09s (± 0.00%) ~ 0.09s 0.09s p=1.000 n=6
Total Time 8.95s (± 0.32%) 8.93s (± 0.30%) ~ 8.90s 8.97s p=0.520 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • Angular - node (v16.17.1, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Monaco - node (v16.17.1, x64)
  • TFS - node (v16.17.1, x64)
  • material-ui - node (v16.17.1, x64)
  • xstate - node (v16.17.1, x64)
Benchmark Name Iterations
Current 54224 6
Baseline main 6

Developer Information:

Download Benchmark

@gabritto
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 12, 2023

Heya @gabritto, I've started to run the perf test suite on this PR at 1175676. You can monitor the build here.

Update: The results are in!

@gabritto gabritto changed the title Cache expression type Cache expression type when checking assertion May 12, 2023
@gabritto gabritto marked this pull request as draft May 12, 2023 20:40
@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels May 12, 2023
@typescript-bot
Copy link
Collaborator

@gabritto
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - main..54224
Metric main 54224 Delta Best Worst p-value
Angular - node (v18.10.0, x64)
Memory used 365,421k (± 0.01%) 365,591k (± 0.01%) +171k (+ 0.05%) 365,544k 365,622k p=0.005 n=6
Parse Time 3.44s (± 0.63%) 3.42s (± 0.57%) ~ 3.39s 3.44s p=0.288 n=6
Bind Time 1.12s (± 0.67%) 1.12s (± 0.46%) ~ 1.11s 1.12s p=0.241 n=6
Check Time 8.82s (± 0.21%) 8.74s (± 0.37%) -0.07s (- 0.83%) 8.71s 8.79s p=0.005 n=6
Emit Time 7.52s (± 0.44%) 7.40s (± 0.39%) -0.12s (- 1.57%) 7.37s 7.45s p=0.005 n=6
Total Time 20.89s (± 0.16%) 20.68s (± 0.25%) -0.21s (- 1.00%) 20.61s 20.76s p=0.005 n=6
Compiler-Unions - node (v18.10.0, x64)
Memory used 191,412k (± 0.51%) 192,075k (± 1.26%) ~ 191,018k 197,006k p=0.173 n=6
Parse Time 1.50s (± 1.82%) 1.50s (± 1.07%) ~ 1.47s 1.51s p=0.934 n=6
Bind Time 0.78s (± 1.05%) 0.78s (± 1.08%) ~ 0.77s 0.79s p=0.718 n=6
Check Time 9.66s (± 0.34%) 9.45s (± 0.25%) -0.21s (- 2.14%) 9.42s 9.48s p=0.005 n=6
Emit Time 2.75s (± 0.93%) 2.74s (± 1.20%) ~ 2.68s 2.77s p=0.809 n=6
Total Time 14.68s (± 0.32%) 14.47s (± 0.34%) -0.22s (- 1.50%) 14.38s 14.52s p=0.005 n=6
Monaco - node (v18.10.0, x64)
Memory used 346,594k (± 0.01%) 346,620k (± 0.01%) ~ 346,561k 346,653k p=0.128 n=6
Parse Time 2.59s (± 1.25%) 2.61s (± 1.18%) ~ 2.58s 2.66s p=0.627 n=6
Bind Time 1.01s (± 0.75%) 1.01s (± 1.16%) ~ 1.00s 1.03s p=0.796 n=6
Check Time 7.18s (± 0.51%) 7.12s (± 0.37%) -0.06s (- 0.84%) 7.08s 7.16s p=0.018 n=6
Emit Time 4.26s (± 1.10%) 4.26s (± 0.59%) ~ 4.23s 4.29s p=0.687 n=6
Total Time 15.04s (± 0.53%) 15.00s (± 0.46%) ~ 14.93s 15.12s p=0.332 n=6
TFS - node (v18.10.0, x64)
Memory used 300,761k (± 0.01%) 300,624k (± 0.01%) -137k (- 0.05%) 300,595k 300,656k p=0.005 n=6
Parse Time 2.08s (± 0.71%) 2.07s (± 0.56%) ~ 2.05s 2.08s p=0.618 n=6
Bind Time 1.14s (± 0.72%) 1.14s (± 1.08%) ~ 1.12s 1.15s p=0.730 n=6
Check Time 6.65s (± 0.40%) 6.60s (± 0.41%) -0.05s (- 0.73%) 6.57s 6.63s p=0.023 n=6
Emit Time 3.89s (± 0.53%) 3.86s (± 0.83%) ~ 3.82s 3.91s p=0.072 n=6
Total Time 13.76s (± 0.35%) 13.67s (± 0.34%) -0.09s (- 0.65%) 13.61s 13.74s p=0.016 n=6
material-ui - node (v18.10.0, x64)
Memory used 481,686k (± 0.00%) 481,659k (± 0.01%) ~ 481,582k 481,708k p=0.748 n=6
Parse Time 3.12s (± 0.60%) 3.12s (± 0.62%) ~ 3.10s 3.15s p=0.564 n=6
Bind Time 0.92s (± 1.15%) 0.91s (± 0.45%) ~ 0.91s 0.92s p=0.528 n=6
Check Time 16.91s (± 0.94%) 16.86s (± 0.49%) ~ 16.78s 17.01s p=0.936 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.94s (± 0.76%) 20.89s (± 0.37%) ~ 20.82s 21.02s p=1.000 n=6
xstate - node (v18.10.0, x64)
Memory used 562,915k (± 0.02%) 562,825k (± 0.03%) ~ 562,690k 563,161k p=0.173 n=6
Parse Time 3.86s (± 0.67%) 3.86s (± 0.46%) ~ 3.84s 3.89s p=0.935 n=6
Bind Time 1.64s (± 0.50%) 1.63s (± 0.34%) -0.01s (- 0.71%) 1.62s 1.63s p=0.025 n=6
Check Time 2.82s (± 0.55%) 2.81s (± 0.71%) ~ 2.78s 2.84s p=0.368 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 0.00%) ~ 0.08s 0.08s p=1.000 n=6
Total Time 8.40s (± 0.30%) 8.38s (± 0.29%) ~ 8.34s 8.41s p=0.257 n=6
Angular - node (v16.17.1, x64)
Memory used 364,884k (± 0.02%) 364,997k (± 0.02%) +112k (+ 0.03%) 364,894k 365,058k p=0.031 n=6
Parse Time 3.56s (± 0.86%) 3.57s (± 0.34%) ~ 3.56s 3.59s p=0.195 n=6
Bind Time 1.19s (± 0.69%) 1.18s (± 0.44%) ~ 1.18s 1.19s p=0.523 n=6
Check Time 9.60s (± 0.52%) 9.66s (± 0.41%) ~ 9.62s 9.72s p=0.146 n=6
Emit Time 7.96s (± 0.78%) 7.96s (± 0.66%) ~ 7.89s 8.02s p=1.000 n=6
Total Time 22.31s (± 0.42%) 22.38s (± 0.38%) ~ 22.28s 22.50s p=0.199 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,278k (± 0.72%) 192,894k (± 0.02%) ~ 192,854k 192,973k p=0.066 n=6
Parse Time 1.59s (± 1.52%) 1.60s (± 0.47%) ~ 1.59s 1.61s p=1.000 n=6
Bind Time 0.83s (± 0.99%) 0.83s (± 0.49%) ~ 0.83s 0.84s p=0.206 n=6
Check Time 10.34s (± 0.54%) 10.19s (± 0.71%) -0.16s (- 1.51%) 10.06s 10.26s p=0.005 n=6
Emit Time 3.03s (± 1.38%) 2.99s (± 0.62%) ~ 2.98s 3.03s p=0.222 n=6
Total Time 15.79s (± 0.37%) 15.62s (± 0.50%) -0.18s (- 1.12%) 15.48s 15.71s p=0.005 n=6
Monaco - node (v16.17.1, x64)
Memory used 345,854k (± 0.01%) 345,870k (± 0.00%) ~ 345,859k 345,885k p=0.228 n=6
Parse Time 2.73s (± 0.76%) 2.73s (± 0.45%) ~ 2.72s 2.75s p=0.935 n=6
Bind Time 1.09s (± 0.37%) 1.08s (± 0.38%) -0.01s (- 0.92%) 1.08s 1.09s p=0.008 n=6
Check Time 7.88s (± 0.25%) 7.84s (± 0.41%) ~ 7.79s 7.88s p=0.052 n=6
Emit Time 4.48s (± 0.99%) 4.48s (± 0.86%) ~ 4.43s 4.52s p=0.936 n=6
Total Time 16.18s (± 0.40%) 16.13s (± 0.35%) ~ 16.06s 16.20s p=0.336 n=6
TFS - node (v16.17.1, x64)
Memory used 300,095k (± 0.01%) 299,962k (± 0.01%) -133k (- 0.04%) 299,935k 299,995k p=0.005 n=6
Parse Time 2.16s (± 0.68%) 2.17s (± 0.68%) ~ 2.16s 2.20s p=0.452 n=6
Bind Time 1.24s (± 0.42%) 1.23s (± 0.61%) ~ 1.22s 1.24s p=0.247 n=6
Check Time 7.32s (± 0.55%) 7.26s (± 0.28%) -0.06s (- 0.82%) 7.24s 7.29s p=0.040 n=6
Emit Time 4.35s (± 1.10%) 4.34s (± 0.73%) ~ 4.28s 4.37s p=0.517 n=6
Total Time 15.08s (± 0.42%) 15.01s (± 0.40%) ~ 14.91s 15.07s p=0.126 n=6
material-ui - node (v16.17.1, x64)
Memory used 480,931k (± 0.00%) 480,937k (± 0.00%) ~ 480,896k 480,957k p=0.575 n=6
Parse Time 3.26s (± 0.50%) 3.26s (± 0.43%) ~ 3.24s 3.28s p=0.744 n=6
Bind Time 0.94s (± 0.80%) 0.94s (± 0.58%) ~ 0.94s 0.95s p=0.476 n=6
Check Time 18.11s (± 1.06%) 17.97s (± 0.95%) ~ 17.80s 18.28s p=0.230 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.32s (± 0.86%) 22.18s (± 0.77%) ~ 21.99s 22.49s p=0.261 n=6
xstate - node (v16.17.1, x64)
Memory used 560,454k (± 0.01%) 560,461k (± 0.01%) ~ 560,338k 560,559k p=0.810 n=6
Parse Time 4.02s (± 0.34%) 4.01s (± 0.55%) ~ 3.97s 4.03s p=0.287 n=6
Bind Time 1.76s (± 0.31%) 1.77s (± 0.87%) ~ 1.74s 1.78s p=0.152 n=6
Check Time 3.06s (± 0.57%) 3.06s (± 0.82%) ~ 3.03s 3.09s p=0.685 n=6
Emit Time 0.09s (± 0.00%) 0.09s (± 4.62%) ~ 0.08s 0.09s p=0.405 n=6
Total Time 8.95s (± 0.32%) 8.94s (± 0.50%) ~ 8.85s 8.97s p=0.748 n=6
Angular - node (v14.21.3, x64)
Memory used 358,871k (± 0.01%) 359,030k (± 0.01%) +159k (+ 0.04%) 358,979k 359,061k p=0.005 n=6
Parse Time 3.68s (± 0.53%) 3.69s (± 0.70%) ~ 3.65s 3.71s p=0.745 n=6
Bind Time 1.22s (± 0.52%) 1.22s (± 0.80%) ~ 1.21s 1.23s p=0.733 n=6
Check Time 10.04s (± 0.52%) 10.02s (± 0.50%) ~ 9.95s 10.07s p=0.872 n=6
Emit Time 8.29s (± 0.97%) 8.33s (± 0.49%) ~ 8.27s 8.38s p=0.148 n=6
Total Time 23.23s (± 0.49%) 23.25s (± 0.41%) ~ 23.11s 23.36s p=0.689 n=6
Compiler-Unions - node (v14.21.3, x64)
Memory used 188,062k (± 0.02%) 188,282k (± 0.02%) +220k (+ 0.12%) 188,248k 188,323k p=0.005 n=6
Parse Time 1.62s (± 0.75%) 1.62s (± 0.39%) ~ 1.61s 1.63s p=0.673 n=6
Bind Time 0.85s (± 0.96%) 0.85s (± 0.89%) ~ 0.84s 0.86s p=0.340 n=6
Check Time 10.44s (± 0.57%) 10.30s (± 0.28%) -0.13s (- 1.28%) 10.26s 10.34s p=0.005 n=6
Emit Time 3.13s (± 1.15%) 3.13s (± 0.66%) ~ 3.10s 3.16s p=1.000 n=6
Total Time 16.03s (± 0.45%) 15.90s (± 0.29%) -0.13s (- 0.80%) 15.83s 15.96s p=0.013 n=6
Monaco - node (v14.21.3, x64)
Memory used 340,979k (± 0.00%) 341,028k (± 0.00%) +50k (+ 0.01%) 341,018k 341,043k p=0.005 n=6
Parse Time 2.81s (± 0.83%) 2.81s (± 0.65%) ~ 2.78s 2.83s p=1.000 n=6
Bind Time 1.11s (± 0.89%) 1.12s (± 0.67%) ~ 1.11s 1.13s p=0.081 n=6
Check Time 8.18s (± 0.47%) 8.22s (± 0.61%) ~ 8.16s 8.29s p=0.169 n=6
Emit Time 4.70s (± 1.02%) 4.74s (± 0.51%) ~ 4.70s 4.77s p=0.089 n=6
Total Time 16.80s (± 0.61%) 16.89s (± 0.24%) ~ 16.83s 16.93s p=0.149 n=6
TFS - node (v14.21.3, x64)
Memory used 295,306k (± 0.00%) 295,125k (± 0.00%) -181k (- 0.06%) 295,117k 295,134k p=0.005 n=6
Parse Time 2.40s (± 0.85%) 2.41s (± 0.73%) ~ 2.38s 2.43s p=0.564 n=6
Bind Time 1.07s (± 0.48%) 1.07s (± 0.59%) ~ 1.06s 1.08s p=0.386 n=6
Check Time 7.62s (± 0.53%) 7.60s (± 0.32%) ~ 7.57s 7.63s p=0.147 n=6
Emit Time 4.35s (± 0.84%) 4.33s (± 0.93%) ~ 4.25s 4.37s p=0.167 n=6
Total Time 15.44s (± 0.46%) 15.40s (± 0.24%) ~ 15.33s 15.44s p=0.223 n=6
material-ui - node (v14.21.3, x64)
Memory used 476,506k (± 0.00%) 476,503k (± 0.01%) ~ 476,457k 476,542k p=1.000 n=6
Parse Time 3.33s (± 0.44%) 3.34s (± 0.66%) ~ 3.31s 3.37s p=0.625 n=6
Bind Time 1.00s (± 0.52%) 1.00s (± 0.52%) ~ 0.99s 1.00s p=1.000 n=6
Check Time 18.77s (± 0.75%) 18.76s (± 0.87%) ~ 18.55s 18.96s p=0.810 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 23.11s (± 0.67%) 23.11s (± 0.73%) ~ 22.87s 23.32s p=1.000 n=6
xstate - node (v14.21.3, x64)
Memory used 549,404k (± 0.00%) 549,427k (± 0.00%) +24k (+ 0.00%) 549,403k 549,440k p=0.037 n=6
Parse Time 4.27s (± 0.63%) 4.27s (± 0.67%) ~ 4.22s 4.30s p=0.686 n=6
Bind Time 1.61s (± 1.54%) 1.61s (± 2.17%) ~ 1.59s 1.68s p=0.870 n=6
Check Time 3.19s (± 0.70%) 3.20s (± 0.82%) ~ 3.15s 3.22s p=0.570 n=6
Emit Time 0.09s (± 0.00%) 0.09s (± 0.00%) ~ 0.09s 0.09s p=1.000 n=6
Total Time 9.17s (± 0.44%) 9.17s (± 0.41%) ~ 9.12s 9.22s p=1.000 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.21.3, x64)
Scenarios
  • Angular - node (v18.10.0, x64)
  • Angular - node (v16.17.1, x64)
  • Angular - node (v14.21.3, x64)
  • Compiler-Unions - node (v18.10.0, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Compiler-Unions - node (v14.21.3, x64)
  • Monaco - node (v18.10.0, x64)
  • Monaco - node (v16.17.1, x64)
  • Monaco - node (v14.21.3, x64)
  • TFS - node (v18.10.0, x64)
  • TFS - node (v16.17.1, x64)
  • TFS - node (v14.21.3, x64)
  • material-ui - node (v18.10.0, x64)
  • material-ui - node (v16.17.1, x64)
  • material-ui - node (v14.21.3, x64)
  • xstate - node (v18.10.0, x64)
  • xstate - node (v16.17.1, x64)
  • xstate - node (v14.21.3, x64)
Benchmark Name Iterations
Current 54224 6
Baseline main 6

TSServer

Comparison Report - main..54224
Metric main 54224 Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,548ms (± 0.50%) 2,544ms (± 0.37%) ~ 2,535ms 2,562ms p=0.521 n=6
Req 2 - geterr 5,688ms (± 0.30%) 5,565ms (± 0.47%) -123ms (- 2.16%) 5,535ms 5,601ms p=0.005 n=6
Req 3 - references 335ms (± 0.77%) 337ms (± 0.66%) ~ 334ms 340ms p=0.258 n=6
Req 4 - navto 286ms (± 0.86%) 288ms (± 1.37%) ~ 284ms 295ms p=0.371 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 85ms (± 1.15%) 85ms (± 0.88%) ~ 84ms 86ms p=0.858 n=6
CompilerTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,661ms (± 0.42%) 2,656ms (± 0.71%) ~ 2,631ms 2,674ms p=0.810 n=6
Req 2 - geterr 4,345ms (± 0.45%) 4,267ms (± 0.43%) -78ms (- 1.79%) 4,245ms 4,299ms p=0.005 n=6
Req 3 - references 348ms (± 0.42%) 349ms (± 0.50%) ~ 347ms 352ms p=0.188 n=6
Req 4 - navto 291ms (± 0.84%) 292ms (± 0.36%) ~ 290ms 293ms p=1.000 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 62ms (± 0.83%) 63ms (± 1.20%) ~ 62ms 64ms p=0.247 n=6
xstateTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 3,078ms (± 0.34%) 3,089ms (± 0.83%) ~ 3,060ms 3,127ms p=0.572 n=6
Req 2 - geterr 1,612ms (± 1.07%) 1,603ms (± 1.53%) ~ 1,568ms 1,638ms p=0.689 n=6
Req 3 - references 107ms (± 1.18%) 116ms (± 6.62%) +9ms (+ 8.72%) 112ms 132ms p=0.005 n=6
Req 4 - navto 366ms (± 0.75%) 363ms (± 0.73%) ~ 359ms 366ms p=0.259 n=6
Req 5 - completionInfo count 2,862 (± 0.00%) 2,862 (± 0.00%) ~ 2,862 2,862 p=1.000 n=6
Req 5 - completionInfo 374ms (± 1.81%) 375ms (± 1.94%) ~ 369ms 388ms p=0.936 n=6
Compiler-UnionsTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,673ms (± 0.83%) 2,687ms (± 0.61%) ~ 2,666ms 2,706ms p=0.378 n=6
Req 2 - geterr 6,165ms (± 0.55%) 6,025ms (± 0.52%) -141ms (- 2.28%) 5,983ms 6,072ms p=0.005 n=6
Req 3 - references 348ms (± 0.51%) 353ms (± 0.59%) +5ms (+ 1.29%) 350ms 355ms p=0.013 n=6
Req 4 - navto 287ms (± 0.48%) 287ms (± 0.57%) ~ 284ms 289ms p=0.591 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 83ms (± 5.71%) 91ms (± 5.92%) ~ 80ms 94ms p=0.144 n=6
CompilerTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,827ms (± 0.90%) 2,827ms (± 0.57%) ~ 2,805ms 2,843ms p=0.575 n=6
Req 2 - geterr 4,700ms (± 0.53%) 4,654ms (± 0.25%) -46ms (- 0.98%) 4,636ms 4,665ms p=0.016 n=6
Req 3 - references 364ms (± 0.41%) 363ms (± 0.62%) ~ 359ms 365ms p=0.247 n=6
Req 4 - navto 286ms (± 1.02%) 290ms (± 1.01%) ~ 284ms 292ms p=0.086 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 68ms (± 1.79%) 67ms (± 1.12%) ~ 66ms 68ms p=0.502 n=6
xstateTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 3,219ms (± 0.39%) 3,241ms (± 0.45%) +22ms (+ 0.69%) 3,215ms 3,259ms p=0.030 n=6
Req 2 - geterr 1,747ms (± 0.54%) 1,738ms (± 0.77%) ~ 1,719ms 1,754ms p=0.374 n=6
Req 3 - references 122ms (± 1.98%) 131ms (± 8.12%) ~ 123ms 145ms p=0.089 n=6
Req 4 - navto 343ms (± 0.77%) 345ms (± 0.70%) ~ 341ms 348ms p=0.373 n=6
Req 5 - completionInfo count 2,862 (± 0.00%) 2,862 (± 0.00%) ~ 2,862 2,862 p=1.000 n=6
Req 5 - completionInfo 404ms (± 1.25%) 404ms (± 1.31%) ~ 395ms 409ms p=1.000 n=6
Compiler-UnionsTSServer - node (v14.21.3, x64)
Req 1 - updateOpen 2,793ms (± 0.67%) 2,802ms (± 0.54%) ~ 2,783ms 2,821ms p=0.471 n=6
Req 2 - geterr 6,329ms (± 0.41%) 6,212ms (± 0.41%) -117ms (- 1.85%) 6,177ms 6,254ms p=0.005 n=6
Req 3 - references 362ms (± 0.60%) 363ms (± 0.38%) ~ 361ms 365ms p=0.254 n=6
Req 4 - navto 291ms (± 0.51%) 289ms (± 0.47%) ~ 288ms 291ms p=0.121 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 99ms (± 8.26%) 101ms (± 2.17%) ~ 99ms 103ms p=0.498 n=6
CompilerTSServer - node (v14.21.3, x64)
Req 1 - updateOpen 2,975ms (± 0.21%) 2,981ms (± 0.34%) ~ 2,967ms 2,994ms p=0.467 n=6
Req 2 - geterr 4,591ms (± 0.50%) 4,527ms (± 0.38%) -64ms (- 1.40%) 4,504ms 4,549ms p=0.005 n=6
Req 3 - references 380ms (± 1.16%) 378ms (± 1.12%) ~ 374ms 386ms p=0.807 n=6
Req 4 - navto 297ms (± 0.65%) 300ms (± 0.35%) +3ms (+ 0.90%) 298ms 301ms p=0.018 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 77ms (± 1.09%) 77ms (± 0.67%) ~ 76ms 77ms p=0.533 n=6
xstateTSServer - node (v14.21.3, x64)
Req 1 - updateOpen 3,494ms (± 0.49%) 3,488ms (± 1.45%) ~ 3,421ms 3,544ms p=0.936 n=6
Req 2 - geterr 1,853ms (± 0.74%) 1,845ms (± 0.44%) ~ 1,833ms 1,854ms p=0.520 n=6
Req 3 - references 149ms (± 9.09%) 148ms (± 8.50%) ~ 136ms 160ms p=0.806 n=6
Req 4 - navto 402ms (± 2.05%) 399ms (± 2.43%) ~ 387ms 412ms p=0.809 n=6
Req 5 - completionInfo count 2,862 (± 0.00%) 2,862 (± 0.00%) ~ 2,862 2,862 p=1.000 n=6
Req 5 - completionInfo 435ms (± 0.38%) 429ms (± 3.06%) ~ 402ms 436ms p=0.328 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.21.3, x64)
Scenarios
  • Compiler-UnionsTSServer - node (v18.10.0, x64)
  • Compiler-UnionsTSServer - node (v16.17.1, x64)
  • Compiler-UnionsTSServer - node (v14.21.3, x64)
  • CompilerTSServer - node (v18.10.0, x64)
  • CompilerTSServer - node (v16.17.1, x64)
  • CompilerTSServer - node (v14.21.3, x64)
  • xstateTSServer - node (v18.10.0, x64)
  • xstateTSServer - node (v16.17.1, x64)
  • xstateTSServer - node (v14.21.3, x64)
Benchmark Name Iterations
Current 54224 6
Baseline main 6

Startup

Comparison Report - main..54224
Metric main 54224 Delta Best Worst p-value
tsc-startup - node (v16.17.1, x64)
Execution time 141.49ms (± 0.21%) 141.84ms (± 0.23%) +0.35ms (+ 0.25%) 141.02ms 145.86ms p=0.000 n=600
tsserver-startup - node (v16.17.1, x64)
Execution time 220.00ms (± 0.18%) 220.83ms (± 0.20%) +0.82ms (+ 0.37%) 219.78ms 227.84ms p=0.000 n=600
tsserverlibrary-startup - node (v16.17.1, x64)
Execution time 222.10ms (± 0.32%) 222.52ms (± 0.22%) +0.42ms (+ 0.19%) 221.27ms 228.61ms p=0.000 n=600
typescript-startup - node (v16.17.1, x64)
Execution time 203.65ms (± 0.20%) 206.65ms (± 0.36%) +3.00ms (+ 1.47%) 203.29ms 213.06ms p=0.000 n=600
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • tsc-startup - node (v16.17.1, x64)
  • tsserver-startup - node (v16.17.1, x64)
  • tsserverlibrary-startup - node (v16.17.1, x64)
  • typescript-startup - node (v16.17.1, x64)
Benchmark Name Iterations
Current 54224 6
Baseline main 6

Developer Information:

Download Benchmark

@gabritto
Copy link
Member Author

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 12, 2023

Heya @gabritto, I've started to run the abridged perf test suite on this PR at c790579. You can monitor the build here.

Update: The results are in!

@DanielRosenwasser
Copy link
Member

@typescript-bot test this
@typescript-bot test top100
@typescript-bot user test this
@typescript-bot user test tsserver
@typescript-bot test tsserver top100
@typescript-bot run dt

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 12, 2023

Heya @DanielRosenwasser, I've started to run the diff-based top-repos suite (tsserver) on this PR at c790579. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 12, 2023

Heya @DanielRosenwasser, I've started to run the extended test suite on this PR at c790579. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 12, 2023

Heya @DanielRosenwasser, I've started to run the diff-based user code test suite on this PR at c790579. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 12, 2023

Heya @DanielRosenwasser, I've started to run the parallelized Definitely Typed test suite on this PR at c790579. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 12, 2023

Heya @DanielRosenwasser, I've started to run the diff-based user code test suite (tsserver) on this PR at c790579. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 12, 2023

Heya @DanielRosenwasser, I've started to run the diff-based top-repos suite on this PR at c790579. You can monitor the build here.

Update: The results are in!

@DanielRosenwasser
Copy link
Member

Looks like there's going to be a memory bump for Angular, but I think that is a worthwhile tradeoff.

@gabritto gabritto marked this pull request as ready for review May 12, 2023 22:02
@gabritto
Copy link
Member Author

I was doing the comparison of the results of this PR (#54224 (comment)) with the ones in the original PR that introduces the regression (#53261 (comment)). It's tricky to compare those directly, but it seems like this PR takes care of most of the regression on compiler-unions, but it doesn't seem to improve the perf of xstate.

@typescript-bot
Copy link
Collaborator

@gabritto
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..54224

Metric main 54224 Delta Best Worst p-value
Angular - node (v16.17.1, x64)
Memory used 364,884k (± 0.02%) 365,026k (± 0.01%) +142k (+ 0.04%) 364,955k 365,092k p=0.008 n=6
Parse Time 3.56s (± 0.86%) 3.55s (± 0.83%) ~ 3.52s 3.59s p=1.000 n=6
Bind Time 1.19s (± 0.69%) 1.19s (± 0.87%) ~ 1.17s 1.20s p=0.932 n=6
Check Time 9.60s (± 0.52%) 9.64s (± 0.34%) ~ 9.59s 9.68s p=0.172 n=6
Emit Time 7.96s (± 0.78%) 7.90s (± 0.66%) ~ 7.86s 7.98s p=0.142 n=6
Total Time 22.31s (± 0.42%) 22.28s (± 0.41%) ~ 22.18s 22.39s p=0.520 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,278k (± 0.72%) 192,918k (± 0.02%) ~ 192,862k 192,977k p=0.066 n=6
Parse Time 1.59s (± 1.52%) 1.61s (± 1.71%) ~ 1.56s 1.63s p=0.289 n=6
Bind Time 0.83s (± 0.99%) 0.83s (± 0.98%) ~ 0.82s 0.84s p=0.204 n=6
Check Time 10.34s (± 0.54%) 10.24s (± 0.53%) -0.10s (- 0.97%) 10.16s 10.31s p=0.024 n=6
Emit Time 3.03s (± 1.38%) 3.06s (± 1.94%) ~ 2.99s 3.13s p=0.421 n=6
Total Time 15.79s (± 0.37%) 15.75s (± 0.36%) ~ 15.66s 15.81s p=0.422 n=6
Monaco - node (v16.17.1, x64)
Memory used 345,854k (± 0.01%) 345,862k (± 0.01%) ~ 345,838k 345,886k p=0.575 n=6
Parse Time 2.73s (± 0.76%) 2.74s (± 0.75%) ~ 2.72s 2.78s p=0.744 n=6
Bind Time 1.09s (± 0.37%) 1.09s (± 0.37%) ~ 1.08s 1.09s p=0.218 n=6
Check Time 7.88s (± 0.25%) 7.81s (± 0.50%) -0.07s (- 0.87%) 7.75s 7.85s p=0.008 n=6
Emit Time 4.48s (± 0.99%) 4.47s (± 0.68%) ~ 4.43s 4.51s p=1.000 n=6
Total Time 16.18s (± 0.40%) 16.11s (± 0.24%) ~ 16.06s 16.16s p=0.053 n=6
TFS - node (v16.17.1, x64)
Memory used 300,095k (± 0.01%) 299,966k (± 0.01%) -129k (- 0.04%) 299,951k 299,993k p=0.005 n=6
Parse Time 2.16s (± 0.68%) 2.15s (± 0.42%) ~ 2.14s 2.16s p=0.164 n=6
Bind Time 1.24s (± 0.42%) 1.24s (± 0.72%) ~ 1.23s 1.25s p=0.541 n=6
Check Time 7.32s (± 0.55%) 7.27s (± 0.27%) -0.05s (- 0.68%) 7.24s 7.29s p=0.050 n=6
Emit Time 4.35s (± 1.10%) 4.32s (± 0.71%) ~ 4.28s 4.35s p=0.255 n=6
Total Time 15.08s (± 0.42%) 14.99s (± 0.25%) -0.09s (- 0.59%) 14.93s 15.03s p=0.037 n=6
material-ui - node (v16.17.1, x64)
Memory used 480,931k (± 0.00%) 480,927k (± 0.00%) ~ 480,908k 480,941k p=0.873 n=6
Parse Time 3.26s (± 0.50%) 3.25s (± 0.55%) ~ 3.22s 3.27s p=0.256 n=6
Bind Time 0.94s (± 0.80%) 0.94s (± 0.43%) ~ 0.94s 0.95s p=1.000 n=6
Check Time 18.11s (± 1.06%) 18.07s (± 1.23%) ~ 17.85s 18.39s p=0.873 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.32s (± 0.86%) 22.26s (± 1.00%) ~ 22.04s 22.59s p=0.575 n=6
xstate - node (v16.17.1, x64)
Memory used 560,454k (± 0.01%) 560,576k (± 0.02%) +121k (+ 0.02%) 560,459k 560,707k p=0.031 n=6
Parse Time 4.02s (± 0.34%) 4.00s (± 0.34%) -0.02s (- 0.54%) 3.98s 4.02s p=0.033 n=6
Bind Time 1.76s (± 0.31%) 1.77s (± 0.46%) ~ 1.76s 1.78s p=0.859 n=6
Check Time 3.06s (± 0.57%) 3.05s (± 0.29%) ~ 3.04s 3.06s p=0.370 n=6
Emit Time 0.09s (± 0.00%) 0.09s (± 4.45%) ~ 0.09s 0.10s p=0.405 n=6
Total Time 8.95s (± 0.32%) 8.91s (± 0.25%) -0.04s (- 0.43%) 8.88s 8.93s p=0.020 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • Angular - node (v16.17.1, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Monaco - node (v16.17.1, x64)
  • TFS - node (v16.17.1, x64)
  • material-ui - node (v16.17.1, x64)
  • xstate - node (v16.17.1, x64)
Benchmark Name Iterations
Current 54224 6
Baseline main 6

Developer Information:

Download Benchmark

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser Here are the results of running the user test suite comparing main and refs/pull/54224/merge:

Everything looks good!

@DanielRosenwasser
Copy link
Member

DanielRosenwasser commented May 12, 2023

I was doing the comparison of the results of this PR (#54224 (comment)) with the ones in the original PR that introduces the regression (#53261 (comment)). It's tricky to compare those directly, but it seems like this PR takes care of most of the regression on compiler-unions, but it doesn't seem to improve the perf of xstate.

I think that's fine - but I have some ideas we can try.

Two optimizations that would be interesting to try in follow-up PRs would be:

  1. Get rid of the intermediate helper that gets both the type and expression. It's only used in two places, and unfortunately we can't always depend on the runtime to inline the function and eliminate the allocation.
  2. Don't defer the work if either type has any of TypeFlags.{Any,Unknown,Never} since those types will always be castable to-and-from each other.

}
const links = getNodeLinks(node);
links.assertionExpressionType = checkExpression(expression, checkMode);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By moving the checkExpression for the as const case into the if, it's no longer used in the deferred-check case at all; so you shouldn't actually need to cache it like this (which is a tad unsafe, so if it's avoidable, I'd like to avoid it), since you'll either do it once early or once deferred.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what're you thinking @weswigham? Place this in an explicit else?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or I guess, keep it in the same as before, but then cache it on links unconditionally?

That makes sense, you're calling it twice in some cases anyway now, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, that's what I was doing in https://github.com/microsoft/TypeScript/pull/53261/files/45124b1447f2ff88bdf2f9e65f0e1766a80a70a3, and that's what caused xstate checking perf to increase by 10-13% (#53261 (comment)).

Copy link
Member

@DanielRosenwasser DanielRosenwasser May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gabritto but aren't you calling checkExpression twice now? whoops, duh, it's an early return.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess there's no "checking twice" but there's some code deduplication - checkExpression is going to happen anyway. I'd rather we got rid of that and go back to having exprType

That said, I don't see why avoiding deferral is a problem in that case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather we got rid of that and go back to having exprType

I wouldn't. The CheckMode-independent unconditional cache is unsafe and could lead to weird inconsistencies between signature checking states, which can lead to odd editor/command line error inconsistencies. We do it in one or two other places for perf reasons, but it's to be avoided where possible.

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser Here are the results of running the user test suite comparing main and refs/pull/54224/merge:

There were infrastructure failures potentially unrelated to your change:

  • 1 instance of "Unknown failure"
  • 1 instance of "Package install failed"

Otherwise...

Everything looks good!

@weswigham
Copy link
Member

weswigham commented May 12, 2023

Get rid of the intermediate helper that gets both the type and expression. It's only used in two places, and unfortunately we can't always depend on the runtime to inline the function and eliminate the allocation.

We can do this, but the impact should be a minor memory pressure improvement - perf-wise it should be neutral, unless it just barely puts us over the GC line in a project. I'll be surprised if this does too much. If it does.... I'd just call the original change unlucky(?), since almost any new temporary object could have triggered the increased cost.

Don't defer the work if either type has any of TypeFlags.{Any,Unknown,Never} since those types will always be castable to-and-from each other.

That would require actually checking the expression early, which we can entirely avoid in the non-as-const case which is what we needed to avoid to prevent the circularity error in the first place. For the expression type, anyway. For the assertion type, sure; but, again, just deferring a node's check is pretty free (all it means is we add an id to a set and come back to it later) - in fact, usually it's perf-positive to defer a node check because it prevents us from repeating the work when we check an expression tree multiple times during signature resolution (by only doing it once at the end once the signature types are locked in). Avoiding the deferral isn't something I think'll actually be too useful in terms of performance - mostly just complicated. Who knows, I could be wrong, but my hopes aren't high.

Back on this fix, though - If xstate still has worse perf even with us only calling checkExpression once (during either eager or deferred checking), while compiler-unions is fixed, then I'd wager it's, for some reason, impacted in an outsized way by something else that's duplicated between the two stages. The only other work duplicated is looking up the expression and type nodes - if it was JS, I'd buy it could be that (we don't cache jsdoc type node lookups, which may get convoluted), but it's not - it's a TS project. The other thing duplicated is the getTypeFromTypeNode call on the asserted type; usually type nodes which have complex types cache their type internally, but some type node kinds don't. Next thing I'd try is caching the getTypeFromTypeNode call, so we only resolve the type node into actual types once. If that is the issue, it implies we need better caching within getTypeFromTypeNode itself - it implies a broader issue that this just happened to touch, which we'd want to investigate after triaging this (aka: we found a problem that we may be able to fix more broadly for broader perf wins!).

@weswigham
Copy link
Member

It's also possible that the original fix actually just removed circularity errors in xstate. The perf suite doesn't report error metrics, but if we fixed some circularities in xstate, we'd expect the perf to be worse, since we fixed some bogus any types and started doing some real compatibility checking.

@DanielRosenwasser
Copy link
Member

That would require actually checking the expression early

If I'm not mistaken, I think the expression already is checked early, and unconditionally. The deferred work is checking if the two types are comparable in either direction.

@weswigham
Copy link
Member

If I'm not mistaken, I think the expression already is checked early, and unconditionally. The deferred work is checking if the two types are comparable in either direction.

You're right, it is. I'm just thinking we can actually defer the expression check now, too in the non-as-const case, which should be a savings.

@gabritto
Copy link
Member Author

gabritto commented May 12, 2023

It's also possible that the original fix actually just removed circularity errors in xstate. The perf suite doesn't report error metrics, but if we fixed some circularities in xstate, we'd expect the perf to be worse, since we fixed some bogus any types and started doing some real compatibility checking.

That sounds like a reasonable explanation for the weird xstate behavior, so I'm gonna look into it, since if that's the case, then we can just check the expression in the deferred check and no need to cache types.

@weswigham
Copy link
Member

weswigham commented May 12, 2023

That sounds like a reasonable explanation for the weird xstate behavior, so I'm gonna look into it, since if that's the case, then we can just check the expression in the deferred check and no need to cache types.

I just checked - the xstate perf test has 7 errors in 5.0, but only 4 in 5.1, so it's possible. Definitely need to see if the change in error output is attributable to the original fix.

@gabritto
Copy link
Member Author

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 12, 2023

Heya @gabritto, I've started to run the abridged perf test suite on this PR at 5b536d5. You can monitor the build here.

Update: The results are in!

@gabritto
Copy link
Member Author

Ok, I compiled xstate with the current main and with a version of this PR where we only call checkExpression in the deferred check, and the errors look exactly the same, so it doesn't immediately seem to be the case that calling checkExpression only in the deferred check eliminates some earlier error that cause us to do more work, so the mystery remains. 🙁

checkSourceElement(typeNode);
const type = getTypeFromTypeNode(typeNode);
// See if we need to check whether the expression and asserted types are comparable.
if (!isErrorType(type) && !((exprType.flags | type.flags) & (TypeFlags.AnyOrUnknown | TypeFlags.Never))) {
Copy link
Member

@weswigham weswigham May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, this is silly - this is quite literally what the first lines of isTypeAssignableTo (via isSimpleTypeRelatedTo) does, and all doing it early avoids is putting a node id into a set (in exchange for duplicating this comparison in the cases where that isn't avoided, which is far and away more common).

@typescript-bot
Copy link
Collaborator

@gabritto
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..54224

Metric main 54224 Delta Best Worst p-value
Angular - node (v16.17.1, x64)
Memory used 364,931k (± 0.01%) 364,903k (± 0.01%) ~ 364,846k 364,958k p=0.298 n=6
Parse Time 3.55s (± 0.34%) 3.57s (± 0.49%) +0.02s (+ 0.56%) 3.55s 3.60s p=0.046 n=6
Bind Time 1.18s (± 0.46%) 1.18s (± 0.35%) ~ 1.18s 1.19s p=0.282 n=6
Check Time 9.61s (± 0.72%) 9.61s (± 0.47%) ~ 9.57s 9.70s p=1.000 n=6
Emit Time 7.91s (± 0.63%) 7.95s (± 0.78%) ~ 7.87s 8.02s p=0.228 n=6
Total Time 22.25s (± 0.46%) 22.31s (± 0.47%) ~ 22.19s 22.49s p=0.630 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 192,688k (± 0.00%) 192,848k (± 0.02%) +160k (+ 0.08%) 192,811k 192,886k p=0.005 n=6
Parse Time 1.59s (± 1.76%) 1.59s (± 1.34%) ~ 1.55s 1.61s p=0.803 n=6
Bind Time 0.83s (± 0.62%) 0.83s (± 0.49%) ~ 0.82s 0.83s p=0.595 n=6
Check Time 10.33s (± 0.34%) 10.20s (± 0.94%) -0.12s (- 1.19%) 10.05s 10.34s p=0.024 n=6
Emit Time 3.00s (± 1.17%) 3.01s (± 0.74%) ~ 2.97s 3.03s p=0.742 n=6
Total Time 15.74s (± 0.32%) 15.64s (± 0.56%) -0.11s (- 0.69%) 15.50s 15.74s p=0.029 n=6
Monaco - node (v16.17.1, x64)
Memory used 345,863k (± 0.01%) 345,844k (± 0.00%) ~ 345,824k 345,865k p=0.298 n=6
Parse Time 2.73s (± 0.20%) 2.74s (± 0.74%) ~ 2.72s 2.78s p=0.663 n=6
Bind Time 1.09s (± 0.47%) 1.09s (± 0.50%) ~ 1.08s 1.09s p=0.640 n=6
Check Time 7.88s (± 0.31%) 7.83s (± 0.56%) ~ 7.78s 7.91s p=0.054 n=6
Emit Time 4.46s (± 0.58%) 4.50s (± 0.85%) ~ 4.43s 4.54s p=0.053 n=6
Total Time 16.16s (± 0.13%) 16.17s (± 0.52%) ~ 16.04s 16.27s p=0.872 n=6
TFS - node (v16.17.1, x64)
Memory used 300,102k (± 0.01%) 299,951k (± 0.01%) -151k (- 0.05%) 299,934k 299,984k p=0.005 n=6
Parse Time 2.16s (± 0.77%) 2.16s (± 0.77%) ~ 2.14s 2.18s p=1.000 n=6
Bind Time 1.24s (± 0.44%) 1.24s (± 0.85%) ~ 1.22s 1.25s p=1.000 n=6
Check Time 7.28s (± 0.28%) 7.24s (± 0.53%) ~ 7.20s 7.29s p=0.075 n=6
Emit Time 4.34s (± 0.66%) 4.33s (± 0.77%) ~ 4.28s 4.38s p=0.936 n=6
Total Time 15.01s (± 0.20%) 14.96s (± 0.47%) ~ 14.88s 15.05s p=0.295 n=6
material-ui - node (v16.17.1, x64)
Memory used 480,930k (± 0.00%) 480,938k (± 0.00%) ~ 480,907k 480,970k p=0.936 n=6
Parse Time 3.25s (± 0.30%) 3.24s (± 0.26%) ~ 3.24s 3.26s p=0.396 n=6
Bind Time 0.94s (± 0.43%) 0.94s (± 0.80%) ~ 0.93s 0.95s p=0.389 n=6
Check Time 17.89s (± 0.92%) 17.93s (± 0.99%) ~ 17.75s 18.23s p=0.630 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.08s (± 0.77%) 22.11s (± 0.81%) ~ 21.92s 22.40s p=0.873 n=6
xstate - node (v16.17.1, x64)
Memory used 560,441k (± 0.02%) 560,425k (± 0.02%) ~ 560,365k 560,592k p=1.000 n=6
Parse Time 4.02s (± 0.37%) 4.00s (± 0.38%) ~ 3.98s 4.02s p=0.139 n=6
Bind Time 1.76s (± 0.72%) 1.76s (± 0.66%) ~ 1.75s 1.78s p=0.801 n=6
Check Time 3.07s (± 0.73%) 3.06s (± 0.53%) ~ 3.04s 3.08s p=0.294 n=6
Emit Time 0.09s (± 4.62%) 0.09s (± 4.62%) ~ 0.08s 0.09s p=1.000 n=6
Total Time 8.95s (± 0.22%) 8.92s (± 0.36%) ~ 8.88s 8.96s p=0.195 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • Angular - node (v16.17.1, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Monaco - node (v16.17.1, x64)
  • TFS - node (v16.17.1, x64)
  • material-ui - node (v16.17.1, x64)
  • xstate - node (v16.17.1, x64)
Benchmark Name Iterations
Current 54224 6
Baseline main 6

Developer Information:

Download Benchmark

@gabritto
Copy link
Member Author

Yep, like Wesley pointed out, latest changes didn't really help. I'll revert.

@DanielRosenwasser
Copy link
Member

DanielRosenwasser commented May 12, 2023

Seems reasonable! I wanted to have that as an experiment in a separate PR anyway.

@@ -34523,6 +34523,8 @@ export function createTypeChecker(host: TypeCheckerHost): TypeChecker {
}
return getRegularTypeOfLiteralType(exprType);
}
const links = getNodeLinks(node);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const links = getNodeLinks(node);
function checkAssertionWorker(node: JSDocTypeAssertion | AssertionExpression, checkMode: CheckMode | undefined) {
if (isConstTypeReference(type)) {
if (!isValidConstAssertionArgument(expression)) {
error(expression, Diagnostics.A_const_assertions_can_only_be_applied_to_references_to_enum_members_or_string_number_boolean_array_or_object_literals);
}
return getRegularTypeOfLiteralType(checkExpression(expression, checkMode));
}
checkSourceElement(type);
checkNodeDeferred(node);
return getTypeFromTypeNode(type);
}

and no changes in checkAssertionDeferred from current main should result in the same number of checkExpression calls as the unsafe cache here, but without needing an unsafe cache, is what I'm trying to say. :P

Copy link
Member

@DanielRosenwasser DanielRosenwasser May 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So

  1. getting rid of assertionExpressionType on NodeLinks entirely
  2. just only doing checkExpression in const contexts and in the deferred check.

Makes sense.

@typescript-bot
Copy link
Collaborator

Hey @DanielRosenwasser, the results of running the DT tests are ready.
Everything looks the same!
You can check the log here.

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser Here are the results of running the top-repos suite comparing main and refs/pull/54224/merge:

Everything looks good!

@gabritto gabritto merged commit 48328ca into main May 13, 2023
@gabritto gabritto deleted the gabritto/assertion branch May 13, 2023 00:22
@typescript-bot
Copy link
Collaborator

@DanielRosenwasser Here are the results of running the top-repos suite comparing main and refs/pull/54224/merge:

Everything looks good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants