From 0904f9b52c94e4e2499785a04f94d76348c12766 Mon Sep 17 00:00:00 2001 From: Ladi Prosek Date: Wed, 3 Jan 2024 14:23:44 +0100 Subject: [PATCH 1/4] Add NGEN documentation --- ...in-Propeties.md => Built-in-Properties.md} | 0 documentation/NETFramework-NGEN.md | 118 ++++++++++++++++++ 2 files changed, 118 insertions(+) rename documentation/{Built-in-Propeties.md => Built-in-Properties.md} (100%) create mode 100644 documentation/NETFramework-NGEN.md diff --git a/documentation/Built-in-Propeties.md b/documentation/Built-in-Properties.md similarity index 100% rename from documentation/Built-in-Propeties.md rename to documentation/Built-in-Properties.md diff --git a/documentation/NETFramework-NGEN.md b/documentation/NETFramework-NGEN.md new file mode 100644 index 00000000000..678a6f1b707 --- /dev/null +++ b/documentation/NETFramework-NGEN.md @@ -0,0 +1,118 @@ +# .NET Framework NGEN Considerations + +NGEN is the name of the legacy native AOT technology used in .NET Framework. Compared to its modern .NET counter-part, +NGEN has the following key characteristics: +- Native code is always stored in separate images located in a machine-wide cache. +- Native images are generated on user machines, typically during app installation, by an elevated process. +- Native images are specific for a given IL image (its identity, *not* its location) and its exact dependencies as they are bound to at run-time. + +Check the [Ngen.exe (Native Image Generator)](https://learn.microsoft.com/en-us/dotnet/framework/tools/ngen-exe-native-image-generator) Microsoft Learn article for an overview of how NGEN works. + +## NGEN in Visual Studio + +Visual Studio use NGEN for almost everything that ships in the box. The sheer amount of code which needs to be +compiled makes it impractical for native image generation to occur synchronously during installation. Instead, +VS installer queues up assemblies for deferred compilation by the NGEN service, which typically happens when the +machine is idle. To force native images to be generated, one can execute the following command in an elevated +terminal window: + +``` +C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen eqi +``` + +The .NET Framework build of MSBuild is inserted into VS and it registers itself for NGEN by including `vs.file.ngenApplications` +in the relevant [files.swr](https://github.com/dotnet/msbuild/blob/main/src/Package/MSBuild.VSSetup/files.swr) entries. MSBuild +is hosted in several processes, most notably the stand-alone command line tool `MSBuild.exe` and the main IDE process `devenv.exe`. +Because each process runs with slightly different dependencies - `MSBuild.exe` loads most of them from `[VS install dir]\MSBuild\Current\Bin` +or `[VS install dir]\MSBuild\Current\Bin\[arch]` while `devenv.exe` has its own set loaded from other parts of the VS installation - +we NGEN our code twice. This is encoded by multiple `vs.file.ngenApplications` entries for a single file in `files.swr`. +The special `[installDir]\Common7\IDE\vsn.exe` entry represents devenv. + +The bad thing about this is that the system is fragile and adding a dependency often results in having to tweak `files.swr` or +`devenv.exe.config`, the latter of which is generated from the file named `devenv.urt.config.tt` in the VS source tree. The good +thing is that regressions, be it a failure to compile an NGEN image or a failure to use an NGEN image, are reliably detected +by the VS PR gates so they are fixed before MSBuild is inserted into the product. + +## NGEN image loading rules + +The Common Language Runtime can be finicky about allowing a native image to load. We usually speak of "NGEN rejections" where a native +image has successfully been created but it cannot be used at run-time. When it happens, the CLR falls back to loading the IL assembly +and JITting code on demand, leading to sub-optimal performance. + +One major reason why a native image is rejected is loading into the LoadFrom context. The rules are excruciatingly complex, but suffice +it to say that when an assembly is loaded by `Assembly.LoadFrom`, it is automatically disqualified from having its native image used. +This is bad news for any app with an add-in system where extension assemblies are loaded by path. + +## SDK resolvers + +One class of assemblies loaded by MSBuild by path are SDK resolvers. MSBuild scans the `SdkResolvers` subdirectory to discover +the set of resolvers to use when evaluating projects. Extensible in theory, though in reality only a couple of resolvers actually +exist. Because resolvers ship as part of VS, it is not difficult to make sure their assemblies are properly NGENed. The hard part is +loading them with the regular `Assembly.Load` so the native images can be used. MSBuild cannot simply extend its probing path to +include the relevant subdirectories of `SdkResolvers` because 1) It is outside of the app directory cone for amd64 and arm64 versions +of `MSBuild.exe` and 2) Not all resolvers actually live under this directory; the system allows them to be placed anywhere. +It is unfortunately also not straightforward to add a binding redirect with a [`codeBase`](https://learn.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/codebase-element) +entry pointing to the right assemblies, because this requires knowing the exact assembly versions. + +### Microsoft.DotNet.MSBuildSdkResolver + +This is the base resolver, capable of resolving "in-box" SDKs that ship with the .NET SDK, and workloads. Since the resolver assembly +is located at a known path relative to MSBuild and has very few dependencies, none of which are used anywhere else, we have decided to +freeze the version of the resolver plus dependencies, so that their full names can be specified in MSBuild.exe.config, e.g. + +``` + + + + +``` + +Additionally, MSBuild.exe.config has the following entry, which enables us to refer to the resolver by simple name. + +``` + +``` + +This has a small advantage compared to hardcoding `Microsoft.DotNet.MSBuildSdkResolver, Version=8.0.100.0, Culture=neutral, PublicKeyToken=adb9793829ddae60` +directly in the code, as it can be modified to work in non-standard environments just by editing the app config appropriately. + +The resolver loading logic in MSBuild has been updated to call `Assembly.Load(AssemblyName)` where the `AssemblyName` specifies the +simple name of the assembly, e.g. `Microsoft.DotNet.MSBuildSdkResolver`, as well as its `CodeBase` (file path). This way the CLR assembly +loader will try to load the assembly into the default context first - a necessary condition for the native image to be used - and fall back +to LoadFrom if the simple name wasn't resolved. + +### Microsoft.Build.NuGetSdkResolver + +The NuGet resolver has many dependencies and its version is frequently changing. Due to the way Visual Studio is composed, MSBuild does +not know at its build time the exact version it will be loading at run-time. We would need a creative installer solution to be able to +have MSbuild.exe.config contain the right entries to load this resolver the same way as `Microsoft.DotNet.MSBuildSdkResolver``. A more +promising direction is loading this resolver into a separate AppDomain (see the section about NuGet.Frameworks below). + +The problem of JITting `Microsoft.Build.NuGetSdkResolver` remains unsolved for now. + +## NuGet.Frameworks + +When evaluating certain property functions, MSBuild requires functionality from `NuGet.Frameworks.dll`, which is not part of MSBuild proper. +The assembly is loaded lazily from a path calculated based on the environment where MSBuild is running and the functionality is invoked +via reflection. Similar to the NuGet resolver, the version is changing and it is not easy to know it statically at MSBuild's build time. +But, since there are only a handful of APIs used by MSBuild and they take simple types such as strings and versions, this has been +addressed by loading the assembly into a separate AppDomain. The AppDomain's config file is created in memory on the fly to contain the +right binding redirects, allowing MSBuild to use `Assembly.Load` and get the native image loaded if it exists. + +This approach has some small startup cost (building the config, creating AppDomain & a `MarshalByRefObject`) and a small run-time overhead +of cross-domain calls. The former is orders of magnitude smaller that the startup hit of JITting and the latter is negligible as long as +the types moved across the AppDomain boundary do not require expensive marshaling. + +## Task assemblies + +This is the proverbial elephant in the room. MSBuild learns about tasks dynamically as it parses project files. The `UsingTask` +element tends to specify the `AssemblyFile` attribute, pointing to the task assembly by path. Consequently MSBuild uses +`Assembly.LoadFrom` and no native images are loaded. Even task assemblies located in the SDK are problematic because MSBuild is +paired with an SDK on users machine at run-time. Unlike SDK resolvers and NuGet.Frameworks, which are part of the same installation +unit, this is a true dynamic inter-product dependency. Additionally, the task API is complex and involves a lot of functionality +provided to tasks via callbacks (e.g. logging) so the overhead of cross-domain calls may be significant. And that's assuming that +suitable native images exist in the first place, something that both VS and SDK installers would need to handle (task assemblies +in each installed SDK are NGENed against each installer version of VS). + +Hosting task assemblies in separate AppDomains looks like a major piece of work with uncertain outcome. We haven't tried it yet +and most task code is JITted. From f2fccb106dc1fe092c6005db245c71191874908a Mon Sep 17 00:00:00 2001 From: Ladi Prosek Date: Wed, 3 Jan 2024 21:48:21 +0100 Subject: [PATCH 2/4] Fix typos --- documentation/NETFramework-NGEN.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/NETFramework-NGEN.md b/documentation/NETFramework-NGEN.md index 678a6f1b707..19b963c30b8 100644 --- a/documentation/NETFramework-NGEN.md +++ b/documentation/NETFramework-NGEN.md @@ -85,7 +85,7 @@ to LoadFrom if the simple name wasn't resolved. The NuGet resolver has many dependencies and its version is frequently changing. Due to the way Visual Studio is composed, MSBuild does not know at its build time the exact version it will be loading at run-time. We would need a creative installer solution to be able to -have MSbuild.exe.config contain the right entries to load this resolver the same way as `Microsoft.DotNet.MSBuildSdkResolver``. A more +have MSbuild.exe.config contain the right entries to load this resolver the same way as `Microsoft.DotNet.MSBuildSdkResolver`. A more promising direction is loading this resolver into a separate AppDomain (see the section about NuGet.Frameworks below). The problem of JITting `Microsoft.Build.NuGetSdkResolver` remains unsolved for now. @@ -112,7 +112,7 @@ paired with an SDK on users machine at run-time. Unlike SDK resolvers and NuGet. unit, this is a true dynamic inter-product dependency. Additionally, the task API is complex and involves a lot of functionality provided to tasks via callbacks (e.g. logging) so the overhead of cross-domain calls may be significant. And that's assuming that suitable native images exist in the first place, something that both VS and SDK installers would need to handle (task assemblies -in each installed SDK are NGENed against each installer version of VS). +in each installed SDK are NGENed against each installed version of VS). Hosting task assemblies in separate AppDomains looks like a major piece of work with uncertain outcome. We haven't tried it yet and most task code is JITted. From 6f758c1cf86d1d5ae4015af9e63a424a3892cf4b Mon Sep 17 00:00:00 2001 From: Ladi Prosek Date: Thu, 4 Jan 2024 15:53:00 +0100 Subject: [PATCH 3/4] Update based on @jeffkl's feedback --- documentation/NETFramework-NGEN.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/documentation/NETFramework-NGEN.md b/documentation/NETFramework-NGEN.md index 19b963c30b8..5c0700de1e0 100644 --- a/documentation/NETFramework-NGEN.md +++ b/documentation/NETFramework-NGEN.md @@ -83,12 +83,13 @@ to LoadFrom if the simple name wasn't resolved. ### Microsoft.Build.NuGetSdkResolver -The NuGet resolver has many dependencies and its version is frequently changing. Due to the way Visual Studio is composed, MSBuild does -not know at its build time the exact version it will be loading at run-time. We would need a creative installer solution to be able to -have MSbuild.exe.config contain the right entries to load this resolver the same way as `Microsoft.DotNet.MSBuildSdkResolver`. A more -promising direction is loading this resolver into a separate AppDomain (see the section about NuGet.Frameworks below). +The NuGet resolver has many dependencies and its version is frequently changing, so the technique used for `Microsoft.DotNet.MSBuildSdkResolver` +does not apply in its current state. However, the NuGet team it looking to address this by: -The problem of JITting `Microsoft.Build.NuGetSdkResolver` remains unsolved for now. +1) ILMerge'ing the resolver with its dependencies into a single assembly. +2) Freezing the version of the assembly. + +When this happens, the cost of JITting `Microsoft.Build.NuGetSdkResolver` will be eliminated as well. ## NuGet.Frameworks From d42e1d2312ceb95cd76fc3c9d2dbe6a835396d12 Mon Sep 17 00:00:00 2001 From: Ladi Prosek Date: Mon, 8 Jan 2024 08:51:58 +0100 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: Rainer Sigwald --- documentation/NETFramework-NGEN.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/documentation/NETFramework-NGEN.md b/documentation/NETFramework-NGEN.md index 5c0700de1e0..c7ec1412d8d 100644 --- a/documentation/NETFramework-NGEN.md +++ b/documentation/NETFramework-NGEN.md @@ -56,27 +56,27 @@ entry pointing to the right assemblies, because this requires knowing the exact ### Microsoft.DotNet.MSBuildSdkResolver -This is the base resolver, capable of resolving "in-box" SDKs that ship with the .NET SDK, and workloads. Since the resolver assembly +This is the most-commonly-used resolver, capable of resolving "in-box" SDKs that ship with the .NET SDK and .NET SDK workloads. Since the resolver assembly is located at a known path relative to MSBuild and has very few dependencies, none of which are used anywhere else, we have decided to -freeze the version of the resolver plus dependencies, so that their full names can be specified in MSBuild.exe.config, e.g. +freeze the version of the resolver plus dependencies, so that their full names can be specified in `MSBuild.exe.config`, e.g. -``` +```xml ``` -Additionally, MSBuild.exe.config has the following entry, which enables us to refer to the resolver by simple name. +Additionally, `MSBuild.exe.config` has the following entry, which enables us to refer to the resolver by simple name. -``` +```xml ``` This has a small advantage compared to hardcoding `Microsoft.DotNet.MSBuildSdkResolver, Version=8.0.100.0, Culture=neutral, PublicKeyToken=adb9793829ddae60` directly in the code, as it can be modified to work in non-standard environments just by editing the app config appropriately. -The resolver loading logic in MSBuild has been updated to call `Assembly.Load(AssemblyName)` where the `AssemblyName` specifies the +The resolver loading logic in MSBuild [has been updated](https://github.com/dotnet/msbuild/pull/9439) to call `Assembly.Load(AssemblyName)` where the `AssemblyName` specifies the simple name of the assembly, e.g. `Microsoft.DotNet.MSBuildSdkResolver`, as well as its `CodeBase` (file path). This way the CLR assembly loader will try to load the assembly into the default context first - a necessary condition for the native image to be used - and fall back to LoadFrom if the simple name wasn't resolved. @@ -84,7 +84,7 @@ to LoadFrom if the simple name wasn't resolved. ### Microsoft.Build.NuGetSdkResolver The NuGet resolver has many dependencies and its version is frequently changing, so the technique used for `Microsoft.DotNet.MSBuildSdkResolver` -does not apply in its current state. However, the NuGet team it looking to address this by: +does not apply in its current state. However, the NuGet team is [looking to address this](https://github.com/NuGet/Home/issues/11441) by: 1) ILMerge'ing the resolver with its dependencies into a single assembly. 2) Freezing the version of the assembly. @@ -113,7 +113,7 @@ paired with an SDK on users machine at run-time. Unlike SDK resolvers and NuGet. unit, this is a true dynamic inter-product dependency. Additionally, the task API is complex and involves a lot of functionality provided to tasks via callbacks (e.g. logging) so the overhead of cross-domain calls may be significant. And that's assuming that suitable native images exist in the first place, something that both VS and SDK installers would need to handle (task assemblies -in each installed SDK are NGENed against each installed version of VS). +in each installed SDK would need to be NGENed against each installed version of VS). Hosting task assemblies in separate AppDomains looks like a major piece of work with uncertain outcome. We haven't tried it yet and most task code is JITted.