Skip to content

Cleanup Helix infrastructure slightly #32554

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 11, 2021
Merged

Cleanup Helix infrastructure slightly #32554

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
96ec3cd
Remove unused `$(HelixUseArchive)` property
dougbu May 9, 2021
4db1fe8
Move helix.proj output into artifacts/bin/helix/
dougbu May 9, 2021
4992128
Support added `msbuild` arguments in RunHelix.ps1
dougbu May 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 28 additions & 25 deletions eng/helix/helix.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
<Import Project="..\targets\Helix.Common.props" />
<Import Project="..\Build.props" />
<Import Project="..\Versions.props" />

<!-- Microsoft.DotNet.Helix.Sdk.MultiQueue.targets splits $(HelixTargetQueues) into @(HelixTargetQueue) items. -->
<ItemGroup Condition=" '$(HelixTargetQueues)' == '' ">
<HelixTargetQueue Condition="'$(_UseHelixOpenQueues)' == 'true'" Include="@(HelixAvailableTargetQueue)" />
Expand All @@ -26,13 +26,11 @@
<HelixAccessToken Condition="'$(_UseHelixOpenQueues)' != 'true'">$(HelixApiAccessToken)</HelixAccessToken>
<IncludeDotNetCli>true</IncludeDotNetCli>
<DotNetCliPackageType>sdk</DotNetCliPackageType>
<DotNetCliVersion>$(NETCoreSdkVersion)</DotNetCliVersion>
<DotNetCliVersion>$(NETCoreSdkVersion)</DotNetCliVersion>
<DotNetCliChannel>Current</DotNetCliChannel>

<!-- Copied from Microsoft.NET.DefaultOutputPaths.targets in the .NET SDK. The Helix SDK contains nothing similar. -->
<BaseOutputPath Condition="'$(BaseOutputPath)' == ''">bin\</BaseOutputPath>
<OutputPath Condition="'$(OutputPath)' == '' and '$(PlatformName)' == 'AnyCPU'">$(BaseOutputPath)$(Configuration)\</OutputPath>
<OutputPath Condition="'$(OutputPath)' == '' and '$(PlatformName)' != 'AnyCPU'">$(BaseOutputPath)$(PlatformName)\$(Configuration)\</OutputPath>
<!-- Similar to ProjectLayout.props in the Arcade SDK. The Helix SDK contains nothing similar. -->
<OutputPath Condition=" '$(OutputPath)' == '' ">$(RepoRoot)artifacts\bin\$(MSBuildProjectName)\</OutputPath>
</PropertyGroup>

<!-- Specify the runtime we need which will be included as a correlation payload -->
Expand All @@ -44,7 +42,7 @@
<PackageType>runtime</PackageType>
</AdditionalDotNetPackage>
</ItemGroup>

<PropertyGroup Condition=" '$(ContinuousIntegrationBuild)' == 'true' ">
<HelixType>ci</HelixType>
<!-- Creator is not valid for internal queues -->
Expand Down Expand Up @@ -86,30 +84,35 @@

<PropertyGroup>
<SharedFxVersion>@(_ResolvedProductVersionInfo->'%(PackageVersion)')</SharedFxVersion>
</PropertyGroup>

</PropertyGroup>

<!-- Use package because .dotnet/ folder doesn't contain RuntimeList.xml file and SharedFxTests checks that. -->
<Unzip Condition="Exists('$(RepoRoot)artifacts\packages\$(Configuration)\Shipping\Microsoft.AspNetCore.App.Runtime.$(TargetRuntimeIdentifier).$(SharedFxVersion).nupkg')"
SourceFiles="$(RepoRoot)artifacts\packages\$(Configuration)\Shipping\Microsoft.AspNetCore.App.Runtime.$(TargetRuntimeIdentifier).$(SharedFxVersion).nupkg"
DestinationFolder="$(OutputPath)\AspNetCoreAppRuntimeHelixRaw" />
SourceFiles="$(RepoRoot)artifacts\packages\$(Configuration)\Shipping\Microsoft.AspNetCore.App.Runtime.$(TargetRuntimeIdentifier).$(SharedFxVersion).nupkg"
DestinationFolder="$(OutputPath)\SharedFx.Unzip" />
<ItemGroup>
<_appRuntimeFiles Include="$(OutputPath)\AspNetCoreAppRuntimeHelixRaw\**\*.txt" />
<_appRuntimeFiles Include="$(OutputPath)\AspNetCoreAppRuntimeHelixRaw\**\*.json" />
<_appRuntimeFiles Include="$(OutputPath)\AspNetCoreAppRuntimeHelixRaw\**\*.dll" />
<_appRuntimeFiles Include="$(OutputPath)\AspNetCoreAppRuntimeHelixRaw\**\*.xml" />
<_appRuntimeFiles Include="$(OutputPath)\SharedFx.Unzip\**\*.txt" />
<_appRuntimeFiles Include="$(OutputPath)\SharedFx.Unzip\**\*.json" />
<_appRuntimeFiles Include="$(OutputPath)\SharedFx.Unzip\**\*.dll" />
<_appRuntimeFiles Include="$(OutputPath)\SharedFx.Unzip\**\RuntimeList.xml" />
</ItemGroup>
<Copy SourceFiles="@(_appRuntimeFiles)" DestinationFolder="$(OutputPath)\AspNetCoreAppRuntimeHelix\shared\Microsoft.AspNetCore.App\$(SharedFxVersion)" />
<Unzip Condition="Exists('$(RepoRoot)artifacts\packages\$(Configuration)\Shipping\Microsoft.AspNetCore.App.Ref.$(SharedFxVersion).nupkg')"
SourceFiles="$(RepoRoot)artifacts\packages\$(Configuration)\Shipping\Microsoft.AspNetCore.App.Ref.$(SharedFxVersion).nupkg"
DestinationFolder="$(OutputPath)\AspNetCoreAppRuntimeHelix\packs\Microsoft.AspNetCore.App.Ref\$(SharedFxVersion)" />
<ZipDirectory Condition="Exists('$(OutputPath)\AspNetCoreAppRuntimeHelix')"
SourceDirectory="$(OutputPath)\AspNetCoreAppRuntimeHelix"
DestinationFile="$(OutputPath)\AspNetCoreAppRuntimeHelix.zip" Overwrite="true" />

<Copy SourceFiles="@(_appRuntimeFiles)"
DestinationFolder="$(OutputPath)\SharedFx.Layout\shared\Microsoft.AspNetCore.App\$(SharedFxVersion)" />
<Copy SourceFiles="$(OutputPath)\SharedFx.Unzip\Microsoft.AspNetCore.App.versions.txt"
DestinationFiles="$(OutputPath)\SharedFx.Layout\shared\Microsoft.AspNetCore.App\$(SharedFxVersion)\.version" />

<Unzip Condition="Exists('$(RepoRoot)artifacts\packages\$(Configuration)\Shipping\Microsoft.AspNetCore.App.Ref.$(SharedFxVersion).nupkg')"
SourceFiles="$(RepoRoot)artifacts\packages\$(Configuration)\Shipping\Microsoft.AspNetCore.App.Ref.$(SharedFxVersion).nupkg"
DestinationFolder="$(OutputPath)\SharedFx.Layout\packs\Microsoft.AspNetCore.App.Ref\$(SharedFxVersion)" />
<ZipDirectory Condition="Exists('$(OutputPath)\SharedFx.Layout')"
SourceDirectory="$(OutputPath)\SharedFx.Layout"
DestinationFile="$(OutputPath)\SharedFx.Layout.zip" Overwrite="true" />

<ItemGroup>
<HelixCorrelationPayload Include="$(OutputPath)\AspNetCoreAppRuntimeHelix.zip" Destination="$(DotNetCliDestination)" />
<HelixCorrelationPayload Include="$(OutputPath)\SharedFx.Layout.zip" Destination="$(DotNetCliDestination)" />
</ItemGroup>
</Target>

<Target Name="Gather" BeforeTargets="Build">
<MSBuild Projects="@(ProjectToBuild)"
Targets="CreateHelixPayload"
Expand Down
18 changes: 16 additions & 2 deletions eng/scripts/RunHelix.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,27 @@
Redhat.7.Amd64.Open
.PARAMETER RunQuarantinedTests
By default quarantined tests are not run. Set this to $true to run only the quarantined tests.
.PARAMETER TargetArchitecture
The CPU architecture to build for (x64, x86, arm). Default=x64
.PARAMETER MSBuildArguments
Additional MSBuild arguments to be passed through.
#>
[CmdletBinding(PositionalBinding = $false)]
param(
[Parameter(Mandatory=$true)]
[string]$Project,

[string]$HelixQueues = "Windows.10.Amd64.Open",
[switch]$RunQuarantinedTests,

[ValidateSet('x64', 'x86', 'arm', 'arm64')]
[string]$TargetArchitecture = "x64",
[bool]$RunQuarantinedTests = $false

# Capture the rest
[Parameter(ValueFromRemainingArguments = $true)]
[string[]]$MSBuildArguments
)

$ErrorActionPreference = 'Stop'
$ProgressPreference = 'SilentlyContinue' # Workaround PowerShell/PowerShell#2138

Expand All @@ -43,4 +56,5 @@ Write-Host -ForegroundColor Yellow "And if packing for a different platform, add
$HelixQueues = $HelixQueues -replace ";", "%3B"
dotnet msbuild $Project /t:Helix /p:TargetArchitecture="$TargetArchitecture" /p:IsRequiredCheck=true `
/p:IsHelixDaily=true /p:HelixTargetQueues=$HelixQueues /p:RunQuarantinedTests=$RunQuarantinedTests `
/p:_UseHelixOpenQueues=true /p:CrossgenOutput=false /p:ASPNETCORE_TEST_LOG_DIR=artifacts/log
/p:_UseHelixOpenQueues=true /p:CrossgenOutput=false /p:ASPNETCORE_TEST_LOG_DIR=artifacts/log `
@MSBuildArguments
1 change: 0 additions & 1 deletion eng/targets/Helix.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@
<IsMacHelixQueue>false</IsMacHelixQueue>
<IsMacHelixQueue Condition="$(HelixTargetQueue.Contains('OSX')) or $(HelixTargetQueue.Contains('macOs'))">true</IsMacHelixQueue>
<HelixTestName>$(MSBuildProjectName)--$(TargetFramework)</HelixTestName>
<HelixUseArchive>false</HelixUseArchive>
<LoggingTestingDisableFileLogging Condition="'$(IsHelixJob)' == 'true'">false</LoggingTestingDisableFileLogging>
<NodeVersion>10.15.3</NodeVersion>
<TestDependsOnAspNetPackages>false</TestDependsOnAspNetPackages>
Expand Down
10 changes: 2 additions & 8 deletions eng/targets/Helix.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,11 +120,6 @@
</PropertyGroup>
</Target>

<Target Name="_PublishHelixArchive" DependsOnTargets="Publish" >
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I vaguely recall IIS functional tests using this, or at least this is why it was added, did they move to something else?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no mentions of this target in the repo after removing this. And, nothing remained that set $(HelixUseArchive) to true before I started. @jkotalik was the "archiving for IIS tests on Helix" requirement removed at some point❔

<ZipDirectory Condition="'$(HelixUseArchive)' == 'true'" SourceDirectory="$(PublishDir)" DestinationFile="$(PublishDir)../$(HelixTestName).zip" Overwrite="true" />
<RemoveDir Condition="'$(HelixUseArchive)' == 'true'" Directories="$(PublishDir)" />
</Target>

<Target Name="_CreateHelixWorkItem" Condition="$(BuildHelixPayload)">
<PropertyGroup>
<!-- Extract the effective queue name from this format "(name)host@dockerimage". This is sometimes used in test code to skip tests. -->
Expand All @@ -133,7 +128,7 @@
</PropertyGroup>

<!-- Important: If HelixTargetQueue is not removed here, then Publish will occur for every single queue type. And since Publish shouldn't depend on the queue we can just publish once -->
<MSBuild Projects="$(MSBuildProjectFullPath)" Targets="_PublishHelixArchive" RemoveProperties="HelixTargetQueue;TestRunId" />
<MSBuild Projects="$(MSBuildProjectFullPath)" Targets="Publish" RemoveProperties="HelixTargetQueue;TestRunId" />

<!-- Isolate Helix tests from whatever happens to be in a parent directory. -->
<Copy SourceFiles="$(RepoRoot)eng\helix\content\Directory.Build.empty.in"
Expand Down Expand Up @@ -176,8 +171,7 @@
</ConvertToAbsolutePath>
<ItemGroup>
<HelixWorkItem Include="$(HelixTestName)">
<PayloadArchive Condition="'$(HelixUseArchive)' == 'true'">$(PublishAbsoluteDir)../$(HelixTestName).zip</PayloadArchive>
<PayloadDirectory Condition="'$(HelixUseArchive)' == 'false'">$(PublishAbsoluteDir)</PayloadDirectory>
<PayloadDirectory>$(PublishAbsoluteDir)</PayloadDirectory>
<TestAssembly>$(TargetFileName)</TestAssembly>
<PreCommands>@(HelixPreCommand)</PreCommands>
<PostCommands>@(HelixPostCommand)</PostCommands>
Expand Down