-
Notifications
You must be signed in to change notification settings - Fork 10.4k
ArPow stage 1: local source-build infrastructure #31235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
<Project> | ||
|
||
<PropertyGroup> | ||
<GitHubRepositoryName>aspnetcore</GitHubRepositoryName> | ||
<SourceBuildManagedOnly>true</SourceBuildManagedOnly> | ||
<CloneSubmodulesToInnerSourceBuildRepo>false</CloneSubmodulesToInnerSourceBuildRepo> | ||
</PropertyGroup> | ||
|
||
<Target Name="ApplySourceBuildPatchFiles" | ||
AfterTargets="PrepareInnerSourceBuildRepoRoot" | ||
BeforeTargets="RunInnerSourceBuildCommand"> | ||
|
||
<ItemGroup> | ||
<SourceBuildPatchFile Include="$(RepositoryEngineeringDir)source-build-patches\*.patch" /> | ||
</ItemGroup> | ||
|
||
<Exec | ||
Command="git apply --ignore-whitespace --whitespace=nowarn "%(SourceBuildPatchFile.FullPath)"" | ||
WorkingDirectory="$(InnerSourceBuildRepoRoot)" | ||
Condition="'@(SourceBuildPatchFile)' != ''" /> | ||
</Target> | ||
|
||
<!-- | ||
Init submodules - temporarary workaround for https://github.com/dotnet/sourcelink/pull/653 | ||
--> | ||
<Target Name="InitSubmodules" | ||
DependsOnTargets="PrepareInnerSourceBuildRepoRoot" | ||
BeforeTargets="RunInnerSourceBuildCommand"> | ||
|
||
<Exec | ||
Command="git submodule update --init --recursive" | ||
WorkingDirectory="$(InnerSourceBuildRepoRoot)" | ||
EnvironmentVariables="@(InnerBuildEnv)" /> | ||
</Target> | ||
dougbu marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
<!-- | ||
Remove inner source .globalconfig file as both the inner and outer config files get loaded and cause a conflict. | ||
Leaving the inner will cause all conflicting settings to be ignored. | ||
https://docs.microsoft.com/en-us/dotnet/fundamentals/code-analysis/configuration-files#general-options | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What's the TLDR info we need to understand "inner and outer" in this context as well as w/ respect to "inner builds"❔ I think inner builds will happen in one or more Docker containers after some setup on the "outer" build agent but am not sure… There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The Arcade Powered Source-Build infra re-clones the current repo somewhere in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In CI, this seems unnecessary. Why not patch the current repo in-place❔ Even in a local build, dev could There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Either way I think this problem will go away when we add our own CI and remove patches? I'm ready to merge this PR when this is clarified. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This was a new source-build problem the ArPow effort caused. The 5.0 source-build infra didn't encounter this because it doesn't have an inner clone - it clones the source in the source-build repo. I could have introduced a patch to address the problem but the long term goal is to eliminate patches and since this was a is problem I wanted to find a permanent solution. One thing to call out here is the patches only get applied to the inner clone. They get applied during each invocation of source-build. If a dev does 'git restore .' it isn't going to affect source-build because the patches aren't applied to the dev's repo. |
||
--> | ||
<Target Name="RemoveInnerGlobalConfig" | ||
DependsOnTargets="PrepareInnerSourceBuildRepoRoot" | ||
BeforeTargets="RunInnerSourceBuildCommand"> | ||
|
||
<Delete Files="$(InnerSourceBuildRepoRoot).globalconfig" /> | ||
</Target> | ||
|
||
<!-- | ||
Build RepoTasks - this is normally triggered via the build script but the inner ArPow source-build is run via msbuild | ||
--> | ||
<Target Name="BuildRepoTasks" | ||
Condition="'$(ArcadeInnerBuildFromSource)' == 'true'" | ||
BeforeTargets="Execute"> | ||
|
||
<Exec | ||
Command="./eng/build.sh --only-build-repo-tasks" | ||
WorkingDirectory="$(InnerSourceBuildRepoRoot)" | ||
EnvironmentVariables="@(InnerBuildEnv)" /> | ||
</Target> | ||
dougbu marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
<Target Name="CustomizeInnerBuildArgs" | ||
BeforeTargets="GetSourceBuildCommandConfiguration"> | ||
|
||
<PropertyGroup> | ||
<InnerBuildArgs>$(InnerBuildArgs) /p:SourceBuildRuntimeIdentifier=$(TargetRuntimeIdentifier)</InnerBuildArgs> | ||
</PropertyGroup> | ||
</Target> | ||
|
||
</Project> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
<UsageData> | ||
JunTaoLuo marked this conversation as resolved.
Show resolved
Hide resolved
|
||
<IgnorePatterns> | ||
<UsagePattern IdentityGlob="*/*" /> | ||
</IgnorePatterns> | ||
</UsageData> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
From 64d2efe24308d9e8b99fdea60c92ee1115259506 Mon Sep 17 00:00:00 2001 | ||
From: dseefeld <[email protected]> | ||
Date: Thu, 31 Oct 2019 20:38:26 +0000 | ||
Subject: [PATCH 02/14] Conditionally set PackAsToolShimRID | ||
|
||
--- | ||
Directory.Build.targets | 5 +++++ | ||
1 file changed, 5 insertions(+) | ||
|
||
diff --git a/Directory.Build.targets b/Directory.Build.targets | ||
index 4b72a52947..1a3b01e858 100644 | ||
--- a/Directory.Build.targets | ||
+++ b/Directory.Build.targets | ||
@@ -52,6 +52,11 @@ | ||
<SuppressDependenciesWhenPacking Condition="'$(SuppressDependenciesWhenPacking)' == '' AND '$(IsAnalyzersProject)' == 'true'">true</SuppressDependenciesWhenPacking> | ||
</PropertyGroup> | ||
|
||
+ <PropertyGroup Condition="'$(PackAsTool)' == 'true' AND '$(IsShippingPackage)' == 'true' AND '$(DotNetBuildFromSource)' != 'true'"> | ||
+ <!-- This is a requirement for Microsoft tool packages only. --> | ||
+ <PackAsToolShimRuntimeIdentifiers>win-x64;win-x86</PackAsToolShimRuntimeIdentifiers> | ||
+ </PropertyGroup> | ||
+ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems strange because this patch duplicates existing code in src/Tools/Directory.Build.targets. Should ensure (if we don't already) the tools don't build in the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The patches being introduced here are the current patches that exist in the current source-build. Our plan as described in the Arcade Powered Source-Build Implementation Plan, is for the repo teams to incorporate/eliminate/refactor these patches after getting the initial source-build infra in place. Some of these patches may be out of date, some may be better achieved via a different means, etc. The real key here is that we will have CI legs in each repo to ensure that the repo is always source-buildable. |
||
<PropertyGroup Condition=" '$(MSBuildProjectExtension)' == '.csproj' "> | ||
<PackageId Condition=" '$(PackageId)' == '' ">$(AssemblyName)</PackageId> | ||
<IsPackable | ||
-- | ||
2.18.0 | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
From 6f903f6870011b6a11074d93a03481faac6a35dc Mon Sep 17 00:00:00 2001 | ||
From: dseefeld <[email protected]> | ||
Date: Thu, 4 Feb 2021 15:18:11 +0000 | ||
Subject: [PATCH] Exclude some projects from source-build | ||
|
||
--- | ||
Directory.Build.props | 4 +++- | ||
1 file changed, 3 insertions(+), 1 deletion(-) | ||
|
||
diff --git a/Directory.Build.props b/Directory.Build.props | ||
index 50d30f12fd..85c2f2428b 100644 | ||
--- a/Directory.Build.props | ||
+++ b/Directory.Build.props | ||
@@ -27,7 +27,9 @@ | ||
'$(IsTestAssetProject)' == 'true' OR | ||
'$(IsBenchmarkProject)' == 'true' OR | ||
'$(IsUnitTestProject)' == 'true'">false</IsShipping> | ||
- | ||
+ <IsOtherTestProject Condition="$(MSBuildProjectName.Contains('IntegrationTesting')) or $(MSBuildProjectName.Contains('TestCommon'))">true</IsOtherTestProject> | ||
+ <ExcludeFromSourceBuild Condition="'$(IsSampleProject)' == 'true' or '$(IsTestAssetProject)' == 'true' or '$(IsBenchmarkProject)' == 'true' or '$(IsUnitTestProject)' == 'true' or '$(IsSpecificationTestProject)' == 'true' or '$(IsOtherTestProject)' == 'true'">true</ExcludeFromSourceBuild> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We shouldn't need this given https://github.com/dotnet/aspnetcore/blob/607e2164ec4a3fbe020bf3dda578ecb1ffd5f22e/Directory.Build.targets#L11-L19 and existing Good news is that as long as we use |
||
+ | ||
<!-- | ||
Following logic mimics core-setup approach as well as | ||
https://github.com/dotnet/arcade/blob/694d59f090b743f894779d04a7ffe11cbaf352e7/src/Microsoft.DotNet.Arcade.Sdk/tools/Publish.proj#L30-L31 | ||
-- | ||
2.21.3 | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
From 83c0f95c563fa6eb9b446f09166cac2c62ab6390 Mon Sep 17 00:00:00 2001 | ||
From: dseefeld <[email protected]> | ||
Date: Thu, 14 May 2020 16:29:28 +0000 | ||
Subject: [PATCH 04/14] Remove Yarn dependency - not used in source-build | ||
|
||
--- | ||
eng/targets/Npm.Common.targets | 3 --- | ||
global.json | 1 - | ||
.../src/Microsoft.Authentication.WebAssembly.Msal.csproj | 2 -- | ||
...oft.AspNetCore.Components.WebAssembly.Authentication.csproj | 2 -- | ||
src/Shared/E2ETesting/E2ETesting.targets | 3 --- | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I understand the global.json part of this patch but not why the other files (which shouldn't be involved in a source build) need to be touched. Could you explain❔ There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I can't say with certainty, it may have been the result of a global search. |
||
5 files changed, 11 deletions(-) | ||
|
||
diff --git a/eng/targets/Npm.Common.targets b/eng/targets/Npm.Common.targets | ||
index 696d23402b..6086dd0843 100644 | ||
--- a/eng/targets/Npm.Common.targets | ||
+++ b/eng/targets/Npm.Common.targets | ||
@@ -1,8 +1,5 @@ | ||
<Project DefaultTargets="Build" InitialTargets="_CheckForInvalidConfiguration"> | ||
|
||
- <!-- Version of this SDK is set in global.json --> | ||
- <Sdk Name="Yarn.MSBuild" /> | ||
- | ||
<PropertyGroup> | ||
<NormalizedPackageId>$(PackageId.Replace('@','').Replace('/','-'))</NormalizedPackageId> | ||
<PackageFileName>$(NormalizedPackageId)-$(PackageVersion).tgz</PackageFileName> | ||
diff --git a/global.json b/global.json | ||
index 501ba161bf..ee9dcaf4fa 100644 | ||
--- a/global.json | ||
+++ b/global.json | ||
@@ -29,7 +29,6 @@ | ||
"xcopy-msbuild": "16.5.0-alpha" | ||
}, | ||
"msbuild-sdks": { | ||
- "Yarn.MSBuild": "1.22.10", | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As far as I know, global.json and the Arcade SDK don't support conditional SDKs. Is this something the Arcade team is working on❔ Otherwise, what's the recommended approach to handling SDKs we need almost everywhere but in source builds❔ There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't have a suggestion. I'll bring this up in our next source-build sync and will reach out to Arcade on the possibility of supporting this scenario. |
||
"Microsoft.DotNet.Arcade.Sdk": "6.0.0-beta.21167.3", | ||
"Microsoft.DotNet.Helix.Sdk": "6.0.0-beta.21167.3" | ||
} | ||
diff --git a/src/Components/WebAssembly/Authentication.Msal/src/Microsoft.Authentication.WebAssembly.Msal.csproj b/src/Components/WebAssembly/Authentication.Msal/src/Microsoft.Authentication.WebAssembly.Msal.csproj | ||
index 939ba16f88..ea2c849ea3 100644 | ||
--- a/src/Components/WebAssembly/Authentication.Msal/src/Microsoft.Authentication.WebAssembly.Msal.csproj | ||
+++ b/src/Components/WebAssembly/Authentication.Msal/src/Microsoft.Authentication.WebAssembly.Msal.csproj | ||
@@ -1,7 +1,5 @@ | ||
<Project Sdk="Microsoft.NET.Sdk.Razor"> | ||
|
||
- <Sdk Name="Yarn.MSBuild" /> | ||
- | ||
<PropertyGroup> | ||
<TargetFramework>$(DefaultNetCoreTargetFramework)</TargetFramework> | ||
<Description>Authenticate your Blazor webassembly applications with Azure Active Directory and Azure Active Directory B2C</Description> | ||
diff --git a/src/Components/WebAssembly/WebAssembly.Authentication/src/Microsoft.AspNetCore.Components.WebAssembly.Authentication.csproj b/src/Components/WebAssembly/WebAssembly.Authentication/src/Microsoft.AspNetCore.Components.WebAssembly.Authentication.csproj | ||
index 21a2efba4e..b4a6a75446 100644 | ||
--- a/src/Components/WebAssembly/WebAssembly.Authentication/src/Microsoft.AspNetCore.Components.WebAssembly.Authentication.csproj | ||
+++ b/src/Components/WebAssembly/WebAssembly.Authentication/src/Microsoft.AspNetCore.Components.WebAssembly.Authentication.csproj | ||
@@ -1,7 +1,5 @@ | ||
<Project Sdk="Microsoft.NET.Sdk.Razor"> | ||
|
||
- <Sdk Name="Yarn.MSBuild" /> | ||
- | ||
<PropertyGroup> | ||
<TargetFramework>$(DefaultNetCoreTargetFramework)</TargetFramework> | ||
<Description>Build client-side authentication for single-page applications (SPAs).</Description> | ||
diff --git a/src/Shared/E2ETesting/E2ETesting.targets b/src/Shared/E2ETesting/E2ETesting.targets | ||
index 76ced2cce9..eaecb913a9 100644 | ||
--- a/src/Shared/E2ETesting/E2ETesting.targets | ||
+++ b/src/Shared/E2ETesting/E2ETesting.targets | ||
@@ -1,7 +1,4 @@ | ||
<Project> | ||
- <!-- Version of this SDK is set in global.json --> | ||
- <Sdk Name="Yarn.MSBuild" /> | ||
- | ||
<!-- Make sure the settings files get copied to the test output folder. --> | ||
<ItemGroup> | ||
<None Update="e2eTestSettings*.json"> | ||
-- | ||
2.18.0 | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
From adef97cd8923986914006454777f9f96d48c3a42 Mon Sep 17 00:00:00 2001 | ||
From: Chris Rummel <[email protected]> | ||
Date: Sun, 28 Jun 2020 18:41:22 +0000 | ||
Subject: [PATCH 09/14] Use source-built version of MSBuild. | ||
|
||
--- | ||
eng/Versions.props | 1 + | ||
eng/tools/RepoTasks/RepoTasks.csproj | 6 +++--- | ||
2 files changed, 4 insertions(+), 3 deletions(-) | ||
|
||
diff --git a/eng/Versions.props b/eng/Versions.props | ||
index 90bb9be..2266be5 100644 | ||
--- a/eng/Versions.props | ||
+++ b/eng/Versions.props | ||
@@ -186,6 +186,7 @@ | ||
<!-- Partner teams --> | ||
<MicrosoftBclAsyncInterfacesVersion>1.0.0</MicrosoftBclAsyncInterfacesVersion> | ||
<MicrosoftBuildVersion>16.9.0</MicrosoftBuildVersion> | ||
+ <MicrosoftBuildFrameworkPackageVersion>16.9.0</MicrosoftBuildFrameworkPackageVersion> | ||
<MicrosoftAzureSignalRVersion>1.2.0</MicrosoftAzureSignalRVersion> | ||
<MicrosoftBuildFrameworkVersion>16.9.0</MicrosoftBuildFrameworkVersion> | ||
<MicrosoftBuildLocatorVersion>1.2.6</MicrosoftBuildLocatorVersion> | ||
diff --git a/eng/tools/RepoTasks/RepoTasks.csproj b/eng/tools/RepoTasks/RepoTasks.csproj | ||
index 7fa9957..3664366 100644 | ||
--- a/eng/tools/RepoTasks/RepoTasks.csproj | ||
+++ b/eng/tools/RepoTasks/RepoTasks.csproj | ||
@@ -20,9 +20,9 @@ | ||
</ItemGroup> | ||
|
||
<ItemGroup Condition="'$(TargetFramework)' == '$(DefaultNetCoreTargetFramework)'"> | ||
- <PackageReference Include="Microsoft.Build.Framework" Version="16.9.0" /> | ||
- <PackageReference Include="Microsoft.Build.Tasks.Core" Version="16.9.0" /> | ||
- <PackageReference Include="Microsoft.Build.Utilities.Core" Version="16.9.0" /> | ||
+ <PackageReference Include="Microsoft.Build.Framework" Version="$(MicrosoftBuildFrameworkPackageVersion)" /> | ||
+ <PackageReference Include="Microsoft.Build.Tasks.Core" Version="$(MicrosoftBuildFrameworkPackageVersion)" /> | ||
+ <PackageReference Include="Microsoft.Build.Utilities.Core" Version="$(MicrosoftBuildFrameworkPackageVersion)" /> | ||
</ItemGroup> | ||
|
||
<ItemGroup Condition="'$(TargetFramework)' == 'net472'"> | ||
-- | ||
2.18.0 | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
From d67aa83be72cbb1ca37aba992633da87235f397b Mon Sep 17 00:00:00 2001 | ||
From: Tomas Weinfurt <[email protected]> | ||
Date: Wed, 13 Nov 2019 22:32:02 -0800 | ||
Subject: [PATCH 06/14] Add FreeBSD | ||
|
||
--- | ||
Directory.Build.props | 1 + | ||
eng/Common.props | 1 + | ||
2 files changed, 2 insertions(+) | ||
|
||
diff --git a/Directory.Build.props b/Directory.Build.props | ||
index cd2dabcdba..53d90f2903 100644 | ||
--- a/Directory.Build.props | ||
+++ b/Directory.Build.props | ||
@@ -159,6 +159,7 @@ | ||
win-arm64; | ||
osx-x64; | ||
osx-arm64; | ||
+ freebsd-x64; | ||
linux-musl-x64; | ||
linux-musl-arm; | ||
linux-musl-arm64; | ||
diff --git a/eng/Common.props b/eng/Common.props | ||
index 7c489b15a9..a9a69bde9f 100644 | ||
--- a/eng/Common.props | ||
+++ b/eng/Common.props | ||
@@ -3,6 +3,7 @@ | ||
<TargetOsName Condition=" '$(TargetOsName)' == '' AND $([MSBuild]::IsOSPlatform('Windows'))">win</TargetOsName> | ||
<TargetOsName Condition=" '$(TargetOsName)' == '' AND $([MSBuild]::IsOSPlatform('OSX'))">osx</TargetOsName> | ||
<TargetOsName Condition=" '$(TargetOsName)' == '' AND $([MSBuild]::IsOSPlatform('Linux'))">linux</TargetOsName> | ||
+ <TargetOsName Condition=" '$(TargetOsName)' == '' AND $([MSBuild]::IsOSPlatform('FreeBSD'))">freebsd</TargetOsName> | ||
<TargetArchitecture Condition="'$(TargetArchitecture)' == ''">x64</TargetArchitecture> | ||
<TargetRuntimeIdentifier Condition="'$(TargetRuntimeIdentifier)' == ''">$(TargetOsName)-$(TargetArchitecture)</TargetRuntimeIdentifier> | ||
</PropertyGroup> | ||
-- | ||
2.18.0 | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
From ca706cd8670d14fae05b36ee83de716fa19d95ba Mon Sep 17 00:00:00 2001 | ||
pranavkm marked this conversation as resolved.
Show resolved
Hide resolved
|
||
From: Omair Majid <[email protected]> | ||
Date: Wed, 23 Oct 2019 15:43:57 -0400 | ||
Subject: [PATCH 07/14] Support global.json on arm64 as well | ||
|
||
arcade uses the runtime section of global.json to decide which | ||
architecture + runtime combination needs to be installed. | ||
|
||
With https://github.com/dotnet/arcade/pull/4132 arcade can install | ||
foreign SDKs in separate locations correctly. | ||
|
||
This change, suggested by @dougbu, makes arcade always install the | ||
dougbu marked this conversation as resolved.
Show resolved
Hide resolved
|
||
runtime for the local architecture (which means it should work on arm64 | ||
and x64) as well as the x86 architecture (skipped on Linux). | ||
|
||
This gets us a working SDK/Runtime combo on arm64. | ||
--- | ||
global.json | 5 +---- | ||
1 file changed, 1 insertion(+), 4 deletions(-) | ||
|
||
diff --git a/global.json b/global.json | ||
index ee9dcaf4fa..053e92ebee 100644 | ||
--- a/global.json | ||
+++ b/global.json | ||
@@ -5,13 +5,10 @@ | ||
"tools": { | ||
"dotnet": "6.0.100-preview.3.21168.19", | ||
"runtimes": { | ||
- "dotnet/x64": [ | ||
+ "dotnet": [ | ||
"2.1.25", | ||
"$(MicrosoftNETCoreBrowserDebugHostTransportVersion)" | ||
], | ||
- "dotnet/x86": [ | ||
- "$(MicrosoftNETCoreBrowserDebugHostTransportVersion)" | ||
- ], | ||
"aspnetcore/x64": [ | ||
"3.1.13" | ||
] | ||
-- | ||
2.18.0 | ||
|
Uh oh!
There was an error while loading. Please reload this page.