Generic linux-x64 RID for supported distros prevents shipping native dependencies in package

[bording]

I have a NuGet package that needs to ship a native library via the runtimes/<RID>/native mechanism, and I need to vary the library by linux distro due to the dependencies it has, OpenSSL being the primary one.

I want to be able to run on all of the supported linux distros listed here, but while testing them, I've noticed that when running dotnet --info, some of them show a specific RID, while others just have linux-x64 listed as the RID. This makes it impossible to ensure the correct version of the library will be chosen.

If I understand the dotnet CLI code correctly, it appears that the reason some of them are falling back to the generic RID is because the runtime.json file hasn't been updated for them.

Here is the list of distros I have tested so far, and the RIDs that are displayed via dotnet --info:

• CentOS 7 (centos.7-x64)
• Fedora 25 (fedora.25-x64)
• Fedora 26 (fedora.26-x64)
• Debian 9 (linux-x64)
• Ubuntu 14.04 (ubuntu.14.04-x64)
• Ubuntu 16.04 (ubuntu.16.04-x64)
• Ubuntu 17.04 (linux-x64)
• Linux Mint 18.2 (linux-x64)
• openSUSE 42.2 (linux-x64)

Was this just an oversight, or is this an intentional change to stop updating this file? If it was intentional, can you explain the reasoning?

I can't just ship a single library using the linux-x64 RID because, for example, I need to compile a separate library for Debian and openSUSE. Right now, it appears that I don't have a way to do that.

[karelz]

@janvorli can you please comment. cc @weshaggard

[weshaggard]

cc @ericstj @Petermarcu

@bording we are trying to switch to a more portable model to avoid an explosion of different distro's and versions. So as much as possible we are trying to use the portable/generic linux-x64 RID. If you need to have assets for distro's you can use the version-less RID for the distro which should be present in the list. If you need to use a differentiate by version and the version isn't present in https://github.com/dotnet/corefx/blob/master/pkg/Microsoft.NETCore.Platforms/runtime.json then there are two options. One you can try and submit a PR to https://github.com/dotnet/corefx/blob/master/pkg/Microsoft.NETCore.Platforms/runtime.json to add the necessary versions, although we have been avoiding that as much as possible, or you could also create your own runtime.json file for your packages and map the versions how you need to for your packages.

@bording would just targeting the versionless distro RID's be enough for your scenario if the generate/portable linux-x64 RID doesn't work.

[bording]

If you need to have assets for distro's you can use the version-less RID for the distro which should be present in the list.

@weshaggard How exactly would this work? When I look at dotnet --info, it only shows linux-x64 as the RID. Are you saying that there is actually another RID for those distros that will be considered? How do I determine what that value is? The only way I'm aware of to see what the RID will be is from dotnet --info. I was assuming that they would only use linux-x64 as the RID, so including assets under a more specific RID would be ignored.

If those distros do also use another RID that is more specific and maps to an existing entry in the runtime.json file, then that might be enough for my scenario, though I'll need to do a bit more research to confirm that.

or you could also create your own runtime.json file for your packages and map the versions how you need to for your packages.

This sounds like an interesting idea. Is there any documentation on how to do this? I wasn't aware that I could just ship my own file and it would be used.

[weshaggard]

you saying that there is actually another RID for those distros that will be considered? How do I determine what that value is?

You can see https://github.com/dotnet/core-setup/blob/master/src/corehost/common/pal.unix.cpp#L273 for how the dotnet host computes the RID for the current OS.

This sounds like an interesting idea. Is there any documentation on how to do this? I wasn't aware that I could just ship my own file and it would be used.

Unfortunately there isn't any documentation on this is there @ericstj? But all I believe is needed is to include a runtime.json in the root of your nuget package with the mappings and ensure your package is referenced in the project that needs the runtime specific assets.

[weshaggard]

Also to add a note about what you are seeing with "dotnet --info". See https://github.com/dotnet/cli/blob/af676518dd963814c76c45adb80d59adc1b387b8/src/dotnet/Program.cs#L272 it will display the RID that the CLI was built with if the calculated RID isn't supported by the shared framework.

[ericstj]

Unfortunately there isn't any documentation on this is there @ericstj?

AFAIK nuget didn't document it /cc @rrelyea . You can see the code here: https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.Packaging/RuntimeModel/RuntimeGraph.cs, https://github.com/NuGet/NuGet.Client/blob/4be29f39b0d9bb897f914e9607f14c31569239ed/src/NuGet.Core/NuGet.Commands/RestoreCommand/ProjectRestoreCommand.cs#L290-L366. NuGet will honor a runtime.json in the root of any package, so long as that package appears in the RID-less graph (it cannot be the result of a RID-specific package merged by a runtime.json). I don't think they NuGet has a great story for two packages providing conflicting mappings, so we need to make sure we make things consistent if you plan to eventually move mappings into MS.NETCore.Platforms.

We have our explanation of RIDs here: https://github.com/dotnet/corefx/blob/master/pkg/Microsoft.NETCore.Platforms/readme.md.

[bording]

But all I believe is needed is to include a runtime.json in the root of your nuget package with the mappings and ensure your package is referenced in the project that needs the runtime specific assets.

Does this work even if the package is referenced transitively? I would assume so, but I wanted to be sure.

Also to add a note about what you are seeing with "dotnet --info". See https://github.com/dotnet/cli/blob/af676518dd963814c76c45adb80d59adc1b387b8/src/dotnet/Program.cs#L272 it will display the RID that the CLI was built with if the calculated RID isn't supported by the shared framework.

Given that you're saying the actual calculated RID can still be used in that scenario, the current lack of discoverability of that RID seems like a problem to me. I think dotnet --info should display both, indicating that it will fall back to linux-x64.

It sounds like the combination of the the specific calculated RID being used and the ability to create a custom runtime.json file should unblock me, but to be honest, this still feels like something that shouldn't really be my responsibility to deal with.

As long as RIDs are a concept, and they can be used by NuGet to offer RID-specific assets in a package, I think the runtime.json file in the Microsoft.NETCore.Platforms package should be properly maintained for all of the supported platforms.

[ericstj]

Does this work even if the package is referenced transitively? I would assume so, but I wanted to be sure.

Yes

Given that you're saying the actual calculated RID can still be used in that scenario, the current lack of discoverability of that RID seems like a problem to me. I think dotnet --info should display both, indicating that it will fall back to linux-x64.

Do you think you can open a bug in dotnet/cli with your request? Please be specific about current vs desired behavior. /cc @eerhardt

It sounds like the combination of the the specific calculated RID being used and the ability to create a custom runtime.json file should unblock me, but to be honest, this still feels like something that shouldn't really be my responsibility to deal with.

I agree, I've opened NuGet/Home#5862 to see if we can get folks to provide a better solution.

As long as RIDs are a concept, and they can be used by NuGet to offer RID-specific assets in a package, I think the runtime.json file in the Microsoft.NETCore.Platforms package should be properly maintained for all of the supported platforms.

I agree, and as far as I know that's the current POR. We try to update as new versions are released and we also accept community contributions. https://github.com/dotnet/corefx/issues/23807 is tracking making those updates less error prone.

[bording]

Do you think you can open a bug in dotnet/cli with your request? Please be specific about current vs desired behavior. /cc @eerhardt

Will do.

I agree, and as far as I know that's the current POR. We try to update as new versions are released and we also accept community contributions. dotnet/corefx#23807 is tracking making those updates less error prone.

@weshaggard's comment in https://github.com/dotnet/corefx/issues/23699#issuecomment-326381966 made it sound like a PR to update the file might not be desired.

[eerhardt]

Just to clarify on the behavior of dotnet --info:

https://github.com/dotnet/cli/blob/80c6635fdbaf4b8a04a33be53de9fbb1adbe9ff9/src/dotnet/Program.cs#L264-L275

The call to RuntimeEnvironment.GetRuntimeIdentifier() performs that actual calculation of the RID by inspecting the current machine. It will return things like centos.7-x64, ubuntu.17.04-x64, no matter if the RID is known in a graph or not. The --info command then takes this RID and sees if the runtime it is running on supports this RID by looking for it in the RID-graph in the dotnet\shared\Microsoft.NETCore.App\<version>\Microsoft.NETCore.App.deps.json file. If it isn't supported by this runtime, chances are that it can't be used as a <RuntimeIdentifier> during restore and publish for self-contained apps. And certainly, that RID won't work at runtime on framework-dependent apps, because the host can't resolve it in the graph.

So, dotnet --info falls back to the RID it was built for: win10-x64, osx.10.12-x64, linux-x64 when displaying the RID, since the current machine's RID doesn't work with the current runtime.

See https://github.com/dotnet/cli/issues/4238 and dotnet/cli#4281 for more info.

[eerhardt]

And linking a related issue that explains more about

And certainly, that RID won't work at runtime on framework-dependent apps, because the host can't resolve it in the graph.

https://github.com/dotnet/core-setup/issues/1846

If we fixed that issue, then framework-dependent apps could work, if the runtime.json files were crafted correctly.

[bording]

@eerhardt What I'm getting from your comments here and your links to the other issues is that what @weshaggard and @ericstj have been suggesting here won't actually help me because it sounds like even if I authored my package with the specific calculated RIDs I need for my native assets, the fact that those RIDs don't exist in the shared framework RID, nothing would actually work?

If that's true, then what are my options? Is there anything I can do to get this working now with the existing .NET Core 2.0 release?

[eerhardt]

Using the approaches above, you should be able to make it work for self-contained applications. Let's take ubuntu.17.04-x64 as an example.

1. Create a runtime.json that maps ubuntu.17.04-x64 => linux-x64. Put it in the root of your NuGet package. Here's the file I used when testing it myself: runtime.json.zip
2. Inside your NuGet package, put native assets in runtimes/ubuntu.17.04-x64/native folder.
3. Create and publish a console application with <RuntimeIdentifier>ubuntu.17.04-x64</RuntimeIdentifier> that references your NuGet package.
   1. Note: in .NET Core 2.0, this would also work on a "RID-specific, framework-dependent" app. Meaning <RuntimeIdentifier>ubuntu.17.04-x64</RuntimeIdentifier> and <SelfContained>false</SelfContained>. The app will run against the shared framework, but will only contain Ubuntu 17 assets, thus it isn't considered "portable". See Enable RID-specific Apps on Shared Framework  sdk#1053 for more info.

This will allow your app to load the native Ubuntu 17 specific assets.

However, as far as I can tell, you won't be able to use the native ubuntu.17.04-x64 assets when publishing a "portable" application, because of the above referenced issue.

[bording]

@eerhardt Thanks for following up!

So, I have control over steps 1 and 2, but this is for a library that will be on nuget.org, so I have no control over how it's consumed, meaning step 3 won't be something I can enforce.

However, as far as I can tell, you won't be able to use the native ubuntu.17.04-x64 assets when publishing a "portable" application, because of the above referenced issue.

This combined with the fact that it sounds like certain properties need to be set in the consuming project file to make it work at all make me think that I don't have a way to solve this in a general way that will be usable by anyone who wants to use the package. 😢

I think this showing once again that the actual runtime.json file needs to be kept up to date with for all supported platforms, and shipping 2.0 without that having happened is a bug that needs to be fixed.

[weshaggard]

I think this showing once again that the actual runtime.json file needs to be kept up to date with for all supported platforms, and shipping 2.0 without that having happened is a bug that needs to be fixed.

Updating the runtime.json file will not help with this problem because we intentionally don't produce a shared framework for every RID, as that just doesn't scale. When you are in the world with custom RID's the only real option you have is to create standalone applications.

The combinatorics and scale issues are exactly why we are trying to move to more generic RID's and instead of producing binaries that can only run on one platform we produce native binaries that do runtime light-up for features that aren't common to our generic set of dependencies.