`TypeLoadException`s on .NET 9 when referencing HttpSys on Linux

[MihaZupan]

YARP will try to grab the IServerDelegationFeature on startup to detect if the functionality is supported.
_serverDelegationFeature = server.Features?.Get<IServerDelegationFeature>();

We're seeing many users run into TypeLoadExceptions when upgrading their app from .NET 8 to 9:
dotnet/yarp#2652

System.TypeLoadException: Could not load type 'Microsoft.AspNetCore.Server.HttpSys.IServerDelegationFeature' from assembly 'Microsoft.AspNetCore.Server.HttpSys, Version=9.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
   at Yarp.ReverseProxy.Delegation.HttpSysDelegator..ctor(IServer server, ILogger`1 logger)

Reports mention both Ubuntu 22 and 24.
I was not able to reproduce the issue myself (on a VM nor WSL).

Possibly a recent regression since RC2?
dotnet/yarp#2652 (comment)

To give some extra confusion, we've been running our test envoirment on 9RC1 and 9RC2 when they came out, and didn't have the issue, they only came with the official release (I use ./dotnet-install.sh, no apt since it wasn't released there yet).

---

What sort of information would be needed to help diagnose this?

[Tratcher]

@HugoVG, @dario-l, @ebesavara, @mikoskinen, @shainegordon, can any of you provide a repro project and steps? Preferably as a github repo as this kind of issue depends a lot on the project file, dependencies, publish settings, etc.

[HugoVG]

making one right now

[HugoVG]

https://github.com/HugoVG/Dotnet9-linux-crash-sanityCheck

I put it in here with my stack trace
(keep in mind I've only grabbed the wsl one, I didn't have the other server at the moment)
WSL: Ubuntu 22.04
Test Servers: Ubuntu 22.04

Downloaded with ./dotnet-install.sh

[Tratcher]

Possible culprit: http://archive.ubuntu.com/ubuntu/pool/universe/d/dotnet9/aspnetcore-runtime-9.0_9.0.0-rtm-0ubuntu1~24.10.1_amd64.deb
https://ubuntu.pkgs.org/24.10/ubuntu-updates-universe-amd64/aspnetcore-runtime-9.0_9.0.0-rtm-0ubuntu1~24.10.1_amd64.deb.html

This deb package has an empty Microsoft.AspNetCore.Server.HttpSys.dll and different (invalid?) commit shaw a2bc464e40415d625118f38fbb0556d1803783ff. All the other versions I've seen have commit shaw af22eff.

Was this deb package built from source?

aspnetcore/src/Servers/HttpSys/src/Microsoft.AspNetCore.Server.HttpSys.csproj

Lines 36 to 38 in 764eb46

	<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
	<Compile Remove="**\*.cs" />
	</ItemGroup>

[sebastienros]

@richlander

[shainegordon]

Possible culprit: http://archive.ubuntu.com/ubuntu/pool/universe/d/dotnet9/aspnetcore-runtime-9.0_9.0.0-rtm-0ubuntu1~24.10.1_amd64.deb https://ubuntu.pkgs.org/24.10/ubuntu-updates-universe-amd64/aspnetcore-runtime-9.0_9.0.0-rtm-0ubuntu1~24.10.1_amd64.deb.html

This deb package has an empty Microsoft.AspNetCore.Server.HttpSys.dll and different (invalid?) commit shaw a2bc464e40415d625118f38fbb0556d1803783ff. All the other versions I've seen have commit shaw af22eff.

Was this deb package built from source?

aspnetcore/src/Servers/HttpSys/src/Microsoft.AspNetCore.Server.HttpSys.csproj

Lines 36 to 38 in 764eb46

I’m on 24.04, so am using the backports PPA to install .NET 9, rather than the one in the 24.10 distribution - but I’m not clued up as to how backports actually work

[adityamandaleeka]

Tagging @mthalman @MichaelSimons since this is likely source-build related based on the comments above.

[mthalman]

Was this deb package built from source?

aspnetcore/src/Servers/HttpSys/src/Microsoft.AspNetCore.Server.HttpSys.csproj

Lines 36 to 38 in 764eb46

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'"> 
  <Compile Remove="**\*.cs" /> 
</ItemGroup>

Yes, the deb package would have been built from source with DotNetBuildSourceOnly set to true.

Looking at the assembly, I can confirm that it is empty with no classes defined. So that matches with how the project is configured here. This was removed in #50685. cc @Tratcher

[mthalman]

Since this is a result of the configuration of the project file, folks from the ASP.NET Core team should investigate further. The source build team doesn't have the knowledge of how this should be resolved. But we're happy to help with any questions.

[Tratcher]

So why did it work for people in 9.0-rc2 and not rtm? Were they not consuming the source build package?

[mthalman]

Ubuntu never provided DEB packages until 9.0 GA. So any repro using RC2 was using a Microsoft-built installation (non-source build).

[adityamandaleeka]

Thanks @mthalman. I looked at the history here and it looks like we excluded this code from source build at the start of 9.0 because (a) it was being changed to include an external dependency on CsWin32 which isn't allowed in source build and (b) the logical assumption was that HttpSys wouldn't be relevant on non-Windows platforms anyway.

Given that the latter is evidently not true, we need to decide what to do about it while avoiding breaking changes.

It is unfortunate that this scenario never got any mileage on previews/RCs since those were using the MS-built bits...

@wtgodbe @Tratcher What do you think?

[wtgodbe]

it was being changed to include an external dependency on CsWin32 which isn't allowed in source build and (b) the logical assumption was that HttpSys wouldn't be relevant on non-Windows platforms anyway.

Not knowing a lot about this myself, shouldn't CsWin32 not be relevant on non-windows? That is, can we just condition the usage of CsWin32 on the build being on windows, rather than the build being DotNetSourceBuildOnly?