[3.0.102/3.1.101] various assemblies no longer r2r compiled since service release

[tmds]

@omajid noticed file sizes changed for various assemblies with the servicing release:

  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Antiforgery.dll: from 89600 to 43520 bytes (-51%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Authentication.Cookies.dll: from 87552 to 38400 bytes (-56%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Authentication.Core.dll: from 65536 to 31744 bytes (-51%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Authentication.OAuth.dll: from 76288 to 36864 bytes (-51%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Authentication.dll: from 128512 to 57344 bytes (-55%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Authorization.dll: from 68096 to 31232 bytes (-54%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Components.Forms.dll: from 43520 to 19968 bytes (-54%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Components.Server.dll: from 724992 to 391168 bytes (-46%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Components.Web.dll: from 94720 to 49664 bytes (-47%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Components.dll: from 355840 to 135680 bytes (-61%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Connections.Abstractions.dll: from 50176 to 27136 bytes (-45%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Cors.dll: from 65536 to 31232 bytes (-52%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Cryptography.Internal.dll: from 53760 to 31744 bytes (-40%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.DataProtection.dll: from 358912 to 145920 bytes (-59%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Diagnostics.dll: from 375808 to 237568 bytes (-36%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Hosting.dll: from 337408 to 150528 bytes (-55%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Http.Abstractions.dll: from 189952 to 81920 bytes (-56%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Http.Connections.dll: from 244224 to 102912 bytes (-57%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Http.Extensions.dll: from 68608 to 29184 bytes (-57%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Http.dll: from 197120 to 78848 bytes (-60%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.HttpOverrides.dll: from 50688 to 24576 bytes (-51%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Identity.dll: from 179200 to 74752 bytes (-58%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Localization.dll: from 43008 to 21504 bytes (-50%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Mvc.Abstractions.dll: from 178688 to 93696 bytes (-47%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Mvc.ApiExplorer.dll: from 61440 to 26624 bytes (-56%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Mvc.Core.dll: from 1762816 to 729600 bytes (-58%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Mvc.DataAnnotations.dll: from 91648 to 40960 bytes (-55%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Mvc.Formatters.Xml.dll: from 95232 to 39936 bytes (-58%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Mvc.Razor.dll: from 217600 to 100864 bytes (-53%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Mvc.RazorPages.dll: from 406016 to 165376 bytes (-59%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Mvc.TagHelpers.dll: from 258560 to 114688 bytes (-55%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Mvc.ViewFeatures.dll: from 736256 to 283136 bytes (-61%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Razor.Runtime.dll: from 51200 to 24064 bytes (-53%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Razor.dll: from 53248 to 25088 bytes (-52%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.ResponseCaching.dll: from 115712 to 58368 bytes (-49%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.ResponseCompression.dll: from 57856 to 29184 bytes (-49%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Rewrite.dll: from 186880 to 77824 bytes (-58%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Routing.dll: from 618496 to 233984 bytes (-62%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Server.HttpSys.dll: from 446976 to 183808 bytes (-58%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Server.IIS.dll: from 529408 to 231936 bytes (-56%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Server.Kestrel.Core.dll: from 1313792 to 526336 bytes (-59%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll: from 119808 to 51712 bytes (-56%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.Session.dll: from 66560 to 31744 bytes (-52%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.SignalR.Common.dll: from 60416 to 28160 bytes (-53%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.SignalR.Core.dll: from 344064 to 135168 bytes (-60%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.SignalR.Protocols.Json.dll: from 55808 to 24064 bytes (-56%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.StaticFiles.dll: from 131584 to 64512 bytes (-50%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.AspNetCore.WebUtilities.dll: from 187904 to 74752 bytes (-60%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.Caching.Memory.dll: from 47616 to 23552 bytes (-50%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.DependencyInjection.Abstractions.dll: from 65024 to 28672 bytes (-55%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.DependencyInjection.dll: from 167424 to 63488 bytes (-62%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.Diagnostics.HealthChecks.dll: from 79872 to 36864 bytes (-53%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.FileProviders.Physical.dll: from 59904 to 26624 bytes (-55%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.FileSystemGlobbing.dll: from 76288 to 30208 bytes (-60%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.Hosting.dll: from 69120 to 32256 bytes (-53%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.Http.dll: from 109056 to 48640 bytes (-55%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.Identity.Core.dll: from 432640 to 162304 bytes (-62%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.Identity.Stores.dll: from 66048 to 31744 bytes (-51%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.Localization.dll: from 41984 to 20480 bytes (-51%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.Logging.Abstractions.dll: from 95232 to 39424 bytes (-58%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.Logging.dll: from 55296 to 25088 bytes (-54%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.Options.dll: from 93184 to 41472 bytes (-55%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Extensions.Primitives.dll: from 67584 to 29696 bytes (-56%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.JSInterop.dll: from 70656 to 31232 bytes (-55%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/Microsoft.Net.Http.Headers.dll: from 193024 to 67584 bytes (-64%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/System.Diagnostics.EventLog.dll: from 71168 to 39936 bytes (-43%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/System.IO.Pipelines.dll: from 144384 to 56832 bytes (-60%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/System.Security.Cryptography.Pkcs.dll: from 749056 to 237568 bytes (-68%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/System.Security.Cryptography.Xml.dll: from 419328 to 151552 bytes (-63%)
  /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.{0|2}/System.Security.Permissions.dll: from 152064 to 83456 bytes (-45%)

Looking at these assemblies showed they are no longer being compiled ahead-of-time.
Microsoft service builds for 3.0 and 3.1 are still being compiled ahead-of-time.

ref: https://bugzilla.redhat.com/show_bug.cgi?id=1791462

cc @crummel @dagood @dseefeld @omajid

[dagood]

The following line was added by a5cd250, for #1260 (the 3.0.1 merge). This suppresses crossgen and causes the outputs not to be ready-to-run compiled (R2R). @crummel do you remember why you tried this change?

source-build/repos/aspnetcore.proj

Line 22 in 3085962

<BuildCommandArgs>$(BuildCommandArgs) /p:CrossgenOutput=false</BuildCommandArgs>

(FWIW, in my experience, AOT describes the more extensive efforts of e.g. CoreRT, distinct from R2R. In case it causes confusion down the line.)

[dagood]

AzDO deleted the build, but GitHub remembers this error from the previous commit in that PR:

Check failure on line 365 in /src/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/src/Framework/src/Microsoft.AspNetCore.App.Runtime.csproj

/ source-build-CI (centos71 Offline Portable)

/src/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/src/Framework/src/Microsoft.AspNetCore.App.Runtime.csproj#L365
/src/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/src/Framework/src/Microsoft.AspNetCore.App.Runtime.csproj(365,5): error MSB3073: (NETCORE_ENGINEERING_TELEMETRY=Build) The command ""/src/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/crossgen" -nologo -readytorun -in "/src/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/src/Components/Server/src/bin/Release/netcoreapp3.0/Microsoft.AspNetCore.Components.Server.dll" -out "/src/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/Microsoft.AspNetCore.Components.Server.dll" -jitpath "/src/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/libclrjit.so" "@/src/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/PlatformAssembliesPaths.rsp"" exited with code 33.

I think the logs right before the error would reveal more about the cause, but we'll have to run a new build to see what happens.

[dagood]

This is what I get when I enable it on v3.0.1-runtime:

Exec
    Assembly = Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
    CommandLineArguments = "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/crossgen" -nologo -readytorun -in "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/src/Components/Server/src/bin/Release/netcoreapp3.0/Microsoft.AspNetCore.Components.Server.dll" -out "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/Microsoft.AspNetCore.Components.Server.dll" -jitpath "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/libclrjit.so" "@/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/PlatformAssembliesPaths.rsp"
    Parameters
        Command = "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/crossgen" -nologo -readytorun -in "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/src/Components/Server/src/bin/Release/netcoreapp3.0/Microsoft.AspNetCore.Components.Server.dll" -out "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/Microsoft.AspNetCore.Components.Server.dll" -jitpath "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/libclrjit.so" "@/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/PlatformAssembliesPaths.rsp"
        EnvironmentVariables = COMPlus_PartialNGen=0
        IgnoreStandardErrorWarningFormat = True
        StandardOutputImportance = High
    Environment Variables passed to tool:
      COMPlus_PartialNGen=0
    Error: Could not load file or assembly 'Microsoft.Extensions.FileProviders.Embedded, Version=3.0.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
     (0x80070002)
    Error compiling /work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/src/Components/Server/src/bin/Release/netcoreapp3.0/Microsoft.AspNetCore.Components.Server.dll: Could not find or load a specific file. (0x80131621)
    Error: compilation failed for "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/src/Components/Server/src/bin/Release/netcoreapp3.0/Microsoft.AspNetCore.Components.Server.dll" (0x80131621)
    Errors
        /work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/src/Framework/src/Microsoft.AspNetCore.App.Runtime.csproj(365,5): error MSB3073: The command ""/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/crossgen" -nologo -readytorun -in "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/src/Components/Server/src/bin/Release/netcoreapp3.0/Microsoft.AspNetCore.Components.Server.dll" -out "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/Microsoft.AspNetCore.Components.Server.dll" -jitpath "/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/libclrjit.so" "@/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/netcoreapp3.0/linux-x64/crossgen/PlatformAssembliesPaths.rsp"" exited with code 33. [/work/bin/src/AspNetCore.22dedcb2f0de59022e0383e1f05c9caffc708522/src/Framework/src/Microsoft.AspNetCore.App.Runtime.csproj]

On NuGet.org, Microsoft.Extensions.FileProviders.Embedded 3.0.1 contains lib/netcoreapp3.0/Microsoft.Extensions.FileProviders.Embedded.dll. The source-built version (dotnet/extensions) has no lib directory at all. I don't see any clear reason for this, looking at the PackTask creating the nupkg. The DLL is at least passed in. I don't have a non-source-build extensions build on hand to directly compare.

[dagood]

Huh. I get the same weird lib-less nupkg if I clone dotnet/extensions@v3.0.1 and run ./build.sh, no args. (v3.0.2 is not buildable at all, it contains a non-public NuGet feed.) I've filed dotnet/extensions#2897 to ask for their help.

[dagood]

It turns out I filed dotnet/extensions#2897 under false pretense: checking out extensions@v3.0.0 and building it standalone also has the same packing problem. Based on that, it seems unrelated to the issue. I'll open a new issue once I get something more actionable for them.

Now I see that 3.0.0 has a patch called Fix packing on *nix systems that was deleted in afdf3d6. 😕 I'm running a build now with it restored on source-build@v3.0.1-runtime to see if that fixes the lib issue and lets crossgen run.

[dagood]

My build worked--I see crossgen happening in the binlog, the nupkgs are correct, and file sizes are back up. Opened PR #1458.

[tmds]

@dagood thanks a lot for looking into this, and fixing the issue!