dotnet store has bad algorithmic complexity, invocations take orders of magnitude longer than necessary #1682
Comments
noahfalk
added a commit
to noahfalk/JitBench
that referenced
this issue
Oct 27, 2017
I've been encountering a number of usability and reliability issues that I wanted to fix: 1) Pinning the .Net runtime and .Net SDK to netcoreapp2.0 / version 2.0.0 by default. A previous change rolled forward all the .Net references to 2.1 prerelease builds but given this is the rel/2.0.0 branch that didn't seem like the right thing to do. Certainly future .Net release builds should promise back-compat with ASP.Net 2.0 but interim development builds may have back-compatibility bugs. I doubt any of the folks doing perf work in this repo want to take on the responsibility of tracking such bugs down. Allowing floating .Net versions also doesn't provide as stable a baseline for performance comparisons vs. updating individual binaries. 2) There is a major perf issue in the released version of the dotnet store, filed as dotnet/sdk#1682. I have included a workaround because even after it is fixed the branch is still pinned to the older 2.0.0 SDK that has the bug. 3) There is a secondary less serious perf issue with dotnet store that causes msbuild to recursively enumerate every file in the directory containing CreateStore.proj every time it invokes crossgen.exe. Effectively this meant every file in the JitBench repo + every file in the new .dotnet and .store directories was enumerated dozens of times. I mitigated the issue by putting the project in a sub-directory. Together this fix and the previous one cut dotnet store time down to about 45 seconds on my machine. 4) Due to an (unforetunate) design choice, the dotnet.exe in the newly installed .dotnet directory wasn't actually isolated. It would scan and load files from the global dotnet directory if the global directory had more recent versions. This behavior could easily contribute to non-deterministic issues when running the scenario on different machines that have different content installed in the global location. Thankfully there is an environment variable DOTNET_MULTILEVEL_LOOKUP that can be used to opt-out of that behavior. 5) For reasons I don't understand, the previous logging configuration in Program.cs of MusicStore did not actually suppress logs below the warning level. This meant the key per numbers had a few console screens of unneeded spew interspersed. The alternate overload of the logging filter works as you would expect it to.
noahfalk
added a commit
to noahfalk/JitBench
that referenced
this issue
Oct 27, 2017
I've been encountering a number of usability and reliability issues that I wanted to fix: 1) Pinning the .Net runtime and .Net SDK to netcoreapp2.0 / version 2.0.0 by default. A previous change rolled forward all the .Net references to 2.1 prerelease builds but given this is the rel/2.0.0 branch that didn't seem like the right thing to do. Certainly future .Net release builds should promise back-compat with ASP.Net 2.0 but interim development builds may have back-compatibility bugs. I doubt any of the folks doing perf work in this repo want to take on the responsibility of tracking such bugs down. Allowing floating .Net versions also doesn't provide as stable a baseline for performance comparisons vs. updating individual binaries. 2) There is a major perf issue in the released version of the dotnet store, filed as dotnet/sdk#1682. I have included a workaround because even after it is fixed the branch is still pinned to the older 2.0.0 SDK that has the bug. 3) There is a secondary less serious perf issue with dotnet store that causes msbuild to recursively enumerate every file in the directory containing CreateStore.proj every time it invokes crossgen.exe. Effectively this meant every file in the JitBench repo + every file in the new .dotnet and .store directories was enumerated dozens of times. I mitigated the issue by putting the project in a sub-directory. Together this fix and the previous one cut dotnet store time down to about 45 seconds on my machine. 4) Due to an (unforetunate) design choice, the dotnet.exe in the newly installed .dotnet directory wasn't actually isolated. It would scan and load files from the global dotnet directory if the global directory had more recent versions. This behavior could easily contribute to non-deterministic issues when running the scenario on different machines that have different content installed in the global location. Thankfully there is an environment variable DOTNET_MULTILEVEL_LOOKUP that can be used to opt-out of that behavior. 5) For reasons I don't understand, the previous logging configuration in Program.cs of MusicStore did not actually suppress logs below the warning level. This meant the key perf numbers had a few console screens of unneeded spew interspersed. The alternate overload of the logging filter works as you would expect it to.
noahfalk
added a commit
to noahfalk/JitBench
that referenced
this issue
Oct 27, 2017
I've been encountering a number of usability and reliability issues that I wanted to fix: 1) Pinning the .Net runtime and .Net SDK to netcoreapp2.0 / version 2.0.0 by default. A previous change rolled forward all the .Net references to 2.1 prerelease builds but given this is the rel/2.0.0 branch that didn't seem like the right thing to do. Certainly future .Net release builds should promise back-compat with ASP.Net 2.0 but interim development builds may have back-compatibility bugs. I doubt any of the folks doing perf work in this repo want to take on the responsibility of tracking such bugs down. Allowing floating .Net versions also doesn't provide as stable a baseline for performance comparisons vs. updating individual binaries. 2) There is a major perf issue in the released version of the dotnet store, filed as dotnet/sdk#1682. I have included a workaround because even after it is fixed the branch is still pinned to the older 2.0.0 SDK that has the bug. 3) There is a secondary less serious perf issue with dotnet store that causes msbuild to recursively enumerate every file in the directory containing CreateStore.proj every time it invokes crossgen.exe. Effectively this meant every file in the JitBench repo + every file in the new .dotnet and .store directories was enumerated dozens of times. I mitigated the issue by putting the project in a sub-directory. Together this fix and the previous one cut dotnet store time down to about 45 seconds on my machine. 4) Due to an (unforetunate) design choice, the dotnet.exe in the newly installed .dotnet directory wasn't actually isolated. It would scan and load files from the global dotnet directory if the global directory had more recent versions. This behavior could easily contribute to non-deterministic issues when running the scenario on different machines that have different content installed in the global location. Thankfully there is an environment variable DOTNET_MULTILEVEL_LOOKUP that can be used to opt-out of that behavior. 5) For reasons I don't understand, the previous logging configuration in Program.cs of MusicStore did not actually suppress logs below the warning level. This meant the key perf numbers had a few console screens of unneeded spew interspersed. The alternate overload of the logging filter works as you would expect it to.
noahfalk
added a commit
to noahfalk/JitBench
that referenced
this issue
Oct 27, 2017
I've been encountering a number of usability and reliability issues that I wanted to fix: 1) Pinning the .Net runtime and .Net SDK to netcoreapp2.0 / version 2.0.0 by default. A previous change rolled forward all the .Net references to 2.1 prerelease builds but given this is the rel/2.0.0 branch that didn't seem like the right thing to do. Certainly future .Net release builds should promise back-compat with ASP.Net 2.0 but interim development builds may have back-compatibility bugs. I doubt any of the folks doing perf work in this repo want to take on the responsibility of tracking such bugs down. Allowing floating .Net versions also doesn't provide as stable a baseline for performance comparisons vs. updating individual binaries. 2) There is a major perf issue in the released version of the dotnet store, filed as dotnet/sdk#1682. I have included a workaround because even after it is fixed the branch is still pinned to the older 2.0.0 SDK that has the bug. 3) There is a secondary less serious perf issue with dotnet store that causes msbuild to recursively enumerate every file in the directory containing CreateStore.proj every time it invokes crossgen.exe. Effectively this meant every file in the JitBench repo + every file in the new .dotnet and .store directories was enumerated dozens of times. I mitigated the issue by putting the project in a sub-directory. Together this fix and the previous one cut dotnet store time down to about 45 seconds on my machine. 4) Due to an (unforetunate) design choice, the dotnet.exe in the newly installed .dotnet directory wasn't actually isolated. It would scan and load files from the global dotnet directory if the global directory had more recent versions. This behavior could easily contribute to non-deterministic issues when running the scenario on different machines that have different content installed in the global location. Thankfully there is an environment variable DOTNET_MULTILEVEL_LOOKUP that can be used to opt-out of that behavior. 5) For reasons I don't understand, the previous logging configuration in Program.cs of MusicStore did not actually suppress logs below the warning level. This meant the key perf numbers had a few console screens of unneeded spew interspersed. The alternate overload of the logging filter works as you would expect it to.
noahfalk
added a commit
to noahfalk/JitBench
that referenced
this issue
Jan 13, 2018
I've been encountering a number of usability and reliability issues that I wanted to fix: 1) A number of issues are coming in because we are using inconsistent versions of different components. RunBenchmark.ps1 automates the entire setup and execution to help ensure that if you run it with the same arguments then you are getting the same component versions as others. By default it uses 2.0.0 RTM versions, but can be easily configured to use different in-development versions and private runtimes too. 2) There is a major perf issue in the released version of the dotnet store, filed as dotnet/sdk#1682. I have included a workaround because even after it is fixed the branch is still pinned to the older 2.0.0 SDK that has the bug. 3) There is a secondary less serious perf issue with dotnet store that causes msbuild to recursively enumerate every file in the directory containing CreateStore.proj every time it invokes crossgen.exe. Effectively this meant every file in the JitBench repo + every file in the new .dotnet and .store directories was enumerated dozens of times. I mitigated the issue by putting the project in a sub-directory. Together this fix and the previous one cut dotnet store time down to about 45 seconds on my machine. 4) Due to an (unforetunate) design choice, the dotnet.exe in the newly installed .dotnet directory wasn't actually isolated. It would scan and load files from the global dotnet directory if the global directory had more recent versions. This behavior could easily contribute to non-deterministic issues when running the scenario on different machines that have different content installed in the global location. Thankfully there is an environment variable DOTNET_MULTILEVEL_LOOKUP that can be used to opt-out of that behavior. 5) For reasons I don't understand, the previous logging configuration in Program.cs of MusicStore did not actually suppress logs below the warning level. This meant the key perf numbers had a few console screens of unneeded spew interspersed. The alternate overload of the logging filter works as you would expect it to.
|
Fixed by #3345 |
JL03-Yue
pushed a commit
that referenced
this issue
Mar 19, 2024
[main] Update dependencies from dotnet/arcade
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I think there is a functional issue here as well, but I discovered it just from the performance impact. To repro sync https://github.com/aspnet/jitbench and follow the first 3 steps in the README. In step 2 you don't need to replace anything. The important step is 3, it is generating the ASP.Net store from scratch using dotnet store command.
Expected: step 3 completes with only a modest perf overhead relative to the time it takes to crossgen the assemblies (apparently about ~45s on my machine)
Actual: step 3 is still running after 20 minutes, not sure how long it goes
I'm pretty sure the bug is right here:
sdk/src/Tasks/Microsoft.NET.Build.Tasks/build/Microsoft.NET.ComposeStore.targets
Line 108 in b5aa4a2
The StoreResolver target will be invoked N^2 times to make a full matrix of MSBuildProjectPathExtensionsPath on every project X StorePackageName on every project. I'm guessing the intent was to batch on properties of PackageReferencesToStore rather than to batch on PackageReference.
FWIW, while I was investigating the issue above I found that the scheme of invoking MSBuild on the current project to get target parallelism from project parallelism can have some nasty perf side-effects of its own. Before running the desired target MSBuild re-evaluates all the initial global items and properties. One of the initial global items in this project is the set of all source files which includes a globing pattern of **/*. This means in addition to the desired work, MSBuild also does a complete file system walk of the project directory to compute an item list that will never be used. For project directories containing M arbitrary files and N files to crossgen, together with the issue above performance is currently O(N^2 * M) whereas you'd hope for O(N) or O(N + M).
The text was updated successfully, but these errors were encountered: