-
Notifications
You must be signed in to change notification settings - Fork 313
ESRP publishing #3065
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
danieljurek
wants to merge
67
commits into
main
Choose a base branch
from
djurek/esrp-publish
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
ESRP publishing #3065
Changes from all commits
Commits
Show all changes
67 commits
Select commit
Hold shift + click to select a range
7a4dbaa
Remove yank job; ESRP doesn't have direct programmatic support for that
danieljurek 840dfd2
Rough changes to Pack-Crates.ps1 to produce .crate files
danieljurek 10d7fec
First cut at ESRP publishing
danieljurek 5776bcc
Use *.crate package pattern, adjust Pack-Crates.ps1 to create package…
danieljurek 4609efe
Extract package information from .crate file
danieljurek 9361718
Syntax
danieljurek 439fb75
Syntax
danieljurek 219b615
Syntax
danieljurek 67cfc1b
Syntax
danieljurek 5f99257
Compress-ReleaseCrate.ps1, wire into archetype-rust-release.yml
danieljurek 26e43bd
Use managed identity
danieljurek a9bb270
Redundant line
danieljurek 3fb1dff
Deployment scripts must be inline (no checkout)
danieljurek 17492cc
Disable repo tagging to iterate on release, inline compression
danieljurek 29056a6
Create output directory
danieljurek f8e8b61
:
danieljurek 66b58b2
EsrpRelease@10
danieljurek f28db0a
Remove redundant keys
danieljurek 8eae845
DomainTenantId
danieljurek 5529ee9
Don't nest in zip?
danieljurek 228f56e
Remove toolchain config
danieljurek fe40fea
Add KV and signing info
danieljurek 4346cbc
Use ESRPRELPACMANTEST
danieljurek 5a9a328
Check that crate is publishable
danieljurek 5088f09
Pack-Crates.ps1: Remove -PackageInfoDirectory, add -RequireDependenci…
danieljurek 45ee30c
Wire up release intent
danieljurek 6f4ea85
each artifact
danieljurek 8293f62
Parameters
danieljurek 8680b7d
Only specify an environment if not in a test pipeline
danieljurek 056885a
Use environment: none if in TestPipeline
danieljurek 56fb5ed
Remove Environment param
danieljurek 90faf9b
(test) Depend on unreleased core
danieljurek a0447ec
Remove azure_canary_core's dependency on azure_core
danieljurek f6ccd4d
Check existence
danieljurek f70e12b
Naming
danieljurek bdc03d9
Log file paths
danieljurek 867b1c2
backtick
danieljurek 3aa7bf4
Remove extra logging
danieljurek 922fc54
Test ESRP idempotency
danieljurek 014370f
Remove test step
danieljurek f342ccd
Update ci.yml files with release parameters
danieljurek 3a697ac
Also require dependency order
danieljurek 68cffcb
Wire up CHANGELOG.md and README.md
danieljurek 4735144
Test: remove package verison updating from release. Should fail Packing.
danieljurek 8f8ebc3
Better testing: turn off TestPipeline
danieljurek f43a964
Undo test changes
danieljurek fe472fc
Move RequireDependencies into pack.yml
danieljurek 6423c8d
Move RequireDependencies into pack.yml
danieljurek 247de8e
Pack-Crates.ps1: Use PackageInfoDirectory
danieljurek 726d7ac
-RequireDependencies
danieljurek 5997f54
Syntax
danieljurek 37ea97a
Formatting/cleanup
danieljurek fe8cf19
cspell: allowCompoundWords: true
danieljurek 20bed97
Remove formatting
danieljurek debd2f2
Spelling
danieljurek 455288c
Review feedback: No artifacts selected means "just build all artifact…
danieljurek fa4ab88
convertToJson
danieljurek da1fda7
Only create the stage if there are artifacts to release
danieljurek 1a298a0
'False'
danieljurek d3c1de1
More error throwing
danieljurek da077ef
Review feedback: Don't enable allowCompoundWords, also release_ names…
danieljurek 11ad80d
Fix typespec_macros
danieljurek 8e71351
Artifact ordering for Pack-Crates.ps1
danieljurek 9caec0d
Add support for outputting release order of specified packages
danieljurek ba0db82
Use artifact ordering from artifacts
danieljurek db2fb91
Log index
danieljurek 9cfa228
One PR per-release
danieljurek File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,16 +14,13 @@ parameters: | |
| - name: DevFeedName | ||
| type: string | ||
| default: 'public/azure-sdk-for-rust' | ||
| - name: Environment | ||
| type: string | ||
| default: 'cratesio' | ||
|
|
||
| stages: | ||
| - ${{ if eq(variables['System.TeamProject'], 'internal') }}: | ||
| - ${{ if in(variables['Build.Reason'], 'Manual', '') }}: | ||
| - ${{ each artifact in parameters.Artifacts }}: | ||
| - stage: Release_${{artifact.safeName}} | ||
| displayName: "Release: ${{artifact.name}}" | ||
| - ${{ if gt(length(parameters.Artifacts), 0) }}: | ||
| - stage: Release_Batch | ||
| displayName: "Releasing: ${{length(parameters.Artifacts)}} crates" | ||
| dependsOn: ${{parameters.DependsOn}} | ||
| condition: and(succeeded(), ne(variables['SetDevVersion'], 'true'), ne(variables['Skip.Release'], 'true'), ne(variables['Build.Repository.Name'], 'Azure/azure-sdk-for-rust-pr')) | ||
| variables: | ||
|
|
@@ -50,16 +47,17 @@ stages: | |
|
|
||
| - template: /eng/common/pipelines/templates/steps/retain-run.yml | ||
|
|
||
| - script: | | ||
| echo "##vso[build.addbuildtag]${{artifact.name}}" | ||
| displayName: Add build tag '${{artifact.name}}' | ||
| - ${{ each artifact in parameters.Artifacts }}: | ||
| - script: | | ||
| echo "##vso[build.addbuildtag]${{artifact.name}}" | ||
| displayName: Add build tag '${{artifact.name}}' | ||
|
|
||
| - template: /eng/common/pipelines/templates/steps/create-tags-and-git-release.yml | ||
| parameters: | ||
| ArtifactLocation: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}/${{artifact.name}} | ||
| PackageRepository: Crates.io | ||
| ReleaseSha: $(Build.SourceVersion) | ||
| WorkingDirectory: $(Pipeline.Workspace)/_work | ||
| - template: /eng/common/pipelines/templates/steps/create-tags-and-git-release.yml | ||
| parameters: | ||
| ArtifactLocation: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}/${{artifact.name}} | ||
| PackageRepository: Crates.io | ||
| ReleaseSha: $(Build.SourceVersion) | ||
| WorkingDirectory: $(Pipeline.Workspace)/_work | ||
|
|
||
| - deployment: PublishPackage | ||
| displayName: "Publish to Crates.io" | ||
|
|
@@ -71,7 +69,10 @@ stages: | |
| - input: pipelineArtifact # Required, type of the input artifact | ||
| artifactName: ${{parameters.PipelineArtifactName}} # Required, name of the pipeline artifact | ||
| targetPath: $(Pipeline.Workspace)/drop # Optional, specifies where the artifact is downloaded to | ||
| environment: ${{parameters.Environment}} | ||
| ${{if parameters.TestPipeline}}: | ||
| environment: none | ||
| ${{else}}: | ||
| environment: cratesio | ||
| # This timeout shouldn't be necessary once we're able to parallelize better. Right now, | ||
| # this is here to ensure larger areas (30+) libraries don't time out. | ||
| timeoutInMinutes: 120 | ||
|
|
@@ -84,33 +85,77 @@ stages: | |
| runOnce: | ||
| deploy: | ||
| steps: | ||
| - template: /eng/pipelines/templates/steps/use-rust.yml@self | ||
| parameters: | ||
| Toolchain: stable | ||
|
|
||
| - pwsh: | | ||
| $additionalOwners = @('heaths', 'hallipr') | ||
| $token = $env:CARGO_REGISTRY_TOKEN | ||
| $crateName = '${{artifact.name}}' | ||
|
|
||
| $manifestPath = "$(Pipeline.Workspace)/drop/$crateName/contents/Cargo.toml" | ||
| Write-Host "> cargo publish --manifest-path `"$manifestPath`"" | ||
| cargo publish --manifest-path $manifestPath | ||
| if (!$?) { | ||
| Write-Error "Failed to publish package: '$crateName'" | ||
| exit 1 | ||
| } | ||
|
|
||
| $existingOwners = (cargo owner --list $crateName) -replace " \(.*", "" | ||
| $missingOwners = $additionalOwners | Where-Object { $existingOwners -notcontains $_ } | ||
|
|
||
| foreach ($owner in $missingOwners) { | ||
| Write-Host "> cargo owner --add $owner $crateName" | ||
| cargo owner --add $owner $crateName | ||
| } | ||
| displayName: Publish Crate | ||
| env: | ||
| CARGO_REGISTRY_TOKEN: $(azure-sdk-cratesio-token) | ||
| - pwsh: | | ||
| Write-Host "##vso[task.setvariable variable=ArtifactIndex]0" | ||
| displayName: Set ArtifactIndex to 0 | ||
|
|
||
| - ${{ each artifact in parameters.Artifacts }}: | ||
| - pwsh: | | ||
| # Read artifact release order from release-order.json | ||
| # and use ArtifactIndex to select the right one | ||
| $index = [int]'$(ArtifactIndex)' | ||
| $artifacts = Get-Content '$(Pipeline.Workspace)/drop/release-order.json' | ConvertFrom-Json | ||
| if ($index -ge $artifacts.Count) { | ||
| Write-Error "ArtifactIndex $index is out of range (0..$($artifacts.Count - 1))" | ||
| exit 1 | ||
| } | ||
|
|
||
| $artifactName = $artifacts[$index] | ||
| Write-Host "Releasing artifact $artifactName (index $index)" | ||
|
|
||
| $artifactRootPath = '$(Pipeline.Workspace)/drop' | ||
| $outDir = '$(Pipeline.Workspace)/esrp-release' | ||
|
|
||
| if (Test-Path $outDir) { | ||
| Write-Host "Cleaning output directory: $outDir" | ||
| Remove-Item -Path $outDir -Recurse -Force | ||
| } | ||
| New-Item -ItemType Directory -Path $outDir -Force | Out-Null | ||
|
|
||
| Write-Host "Artifact name: $artifactName" | ||
|
|
||
| $packageMetadataPath = "$artifactRootPath/PackageInfo/$artifactName.json" | ||
| if (!(Test-Path $packageMetadataPath)) { | ||
| Write-Error "Package metadata file not found: $packageMetadataPath" | ||
| exit 1 | ||
| } | ||
|
|
||
| $packageMetadata = Get-Content -Raw $packageMetadataPath | ConvertFrom-Json | ||
| $packageVersion = $packageMetadata.version | ||
| Write-Host "Package version: $packageVersion" | ||
|
|
||
| $cratePath = "$artifactRootPath/$artifactName/$artifactName-$packageVersion.crate" | ||
| Copy-Item ` | ||
| -Path $cratePath ` | ||
| -Destination $outDir | ||
| Write-Host "Contents of $outDir" | ||
| Get-ChildItem -Path $outDir | ForEach-Object { Write-Host $_.FullName } | ||
| displayName: 'Copy crate for ESRP' | ||
|
|
||
| - task: EsrpRelease@10 | ||
| displayName: 'ESRP Release' | ||
| inputs: | ||
| connectedservicename: 'Azure SDK PME Managed Identity' | ||
| ClientId: '5f81938c-2544-4f1f-9251-dd9de5b8a81b' | ||
| DomainTenantId: '975f013f-7f24-47e8-a7d3-abc4752bf346' | ||
| Usemanagedidentity: true | ||
| KeyVaultName: 'kv-azuresdk-codesign' | ||
| SignCertName: 'azure-sdk-esrp-release-certificate' | ||
| intent: 'packagedistribution' | ||
| contenttype: 'Rust' | ||
| contentsource: 'Folder' | ||
| folderlocation: '$(Pipeline.Workspace)/esrp-release' | ||
| waitforreleasecompletion: true | ||
| owners: ${{ coalesce(variables['Build.RequestedForEmail'], '[email protected]') }} | ||
| approvers: ${{ coalesce(variables['Build.RequestedForEmail'], '[email protected]') }} | ||
| serviceendpointurl: 'https://api.esrp.microsoft.com/' | ||
| mainpublisher: 'ESRPRELPACMANTEST' | ||
|
|
||
| - pwsh: | | ||
| $index = [int]'$(ArtifactIndex)' + 1 | ||
| Write-Host "Setting ArtifactIndex to $index" | ||
| Write-Host "##vso[task.setvariable variable=ArtifactIndex]$index" | ||
| displayName: Increment ArtifactIndex | ||
|
|
||
| - job: UpdatePackageVersion | ||
| displayName: "API Review and Package Version Update" | ||
|
|
@@ -130,69 +175,32 @@ stages: | |
| displayName: Download ${{parameters.PipelineArtifactName}} artifact | ||
| artifact: ${{parameters.PipelineArtifactName}} | ||
|
|
||
| - template: /eng/common/pipelines/templates/steps/create-apireview.yml | ||
| parameters: | ||
| ArtifactPath: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}} | ||
| Artifacts: ${{parameters.Artifacts}} | ||
| ConfigFileDir: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}/PackageInfo | ||
| MarkPackageAsShipped: true | ||
| ArtifactName: ${{parameters.PipelineArtifactName}} | ||
| SourceRootPath: $(System.DefaultWorkingDirectory) | ||
| PackageName: ${{artifact.name}} | ||
|
|
||
| # Apply the version increment to each library, which updates the Cargo.toml and changelog files. | ||
| - task: PowerShell@2 | ||
| displayName: Increment ${{artifact.name}} version | ||
| inputs: | ||
| targetType: filePath | ||
| filePath: $(Build.SourcesDirectory)/eng/scripts/Update-PackageVersion.ps1 | ||
| arguments: > | ||
| -ServiceDirectory '${{parameters.ServiceDirectory}}' | ||
| -PackageName '${{artifact.name}}' | ||
| - ${{each artifact in parameters.Artifacts }}: | ||
| - template: /eng/common/pipelines/templates/steps/create-apireview.yml | ||
| parameters: | ||
| ArtifactPath: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}} | ||
| Artifacts: ${{parameters.Artifacts}} | ||
| ConfigFileDir: $(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}/PackageInfo | ||
| MarkPackageAsShipped: true | ||
| ArtifactName: ${{parameters.PipelineArtifactName}} | ||
| SourceRootPath: $(System.DefaultWorkingDirectory) | ||
| PackageName: ${{artifact.name}} | ||
|
|
||
| # Apply the version increment to each library, which updates the Cargo.toml and changelog files. | ||
| - task: PowerShell@2 | ||
| displayName: Increment ${{artifact.name}} version | ||
| inputs: | ||
| targetType: filePath | ||
| filePath: $(Build.SourcesDirectory)/eng/scripts/Update-PackageVersion.ps1 | ||
| arguments: > | ||
| -ServiceDirectory '${{parameters.ServiceDirectory}}' | ||
| -PackageName '${{artifact.name}}' | ||
|
|
||
| - template: /eng/common/pipelines/templates/steps/create-pull-request.yml | ||
| parameters: | ||
| PRBranchName: increment-package-version-${{parameters.ServiceDirectory}}-$(Build.BuildId) | ||
| CommitMsg: "Increment package version after release of ${{ artifact.name }}" | ||
| CommitMsg: "Increment package version after release of ${{ join(', ', parameters.Artifacts.*.name) }}" | ||
| PRTitle: "Increment versions for ${{parameters.ServiceDirectory}} releases" | ||
| CloseAfterOpenForTesting: '${{parameters.TestPipeline}}' | ||
| ${{ if startsWith(variables['Build.SourceBranch'], 'refs/pull/') }}: | ||
| BaseBranchName: main | ||
|
|
||
| - ${{ if eq(parameters.TestPipeline, true) }}: | ||
| - job: ManualApproval | ||
| displayName: "Manual approval" | ||
| dependsOn: PublishPackage | ||
| condition: ne(variables['Skip.PublishPackage'], 'true') | ||
| pool: server | ||
| timeoutInMinutes: 120 # 2 hours | ||
| steps: | ||
| - task: ManualValidation@1 | ||
| timeoutInMinutes: 60 # 1 hour | ||
| inputs: | ||
| notifyUsers: '' # Required, but empty string allowed | ||
| allowApproversToApproveTheirOwnRuns: true | ||
| instructions: "Approve yank of ${{ artifact.name }}" | ||
| onTimeout: 'resume' | ||
|
|
||
| - job: YankCrates | ||
| displayName: "Yank Crates" | ||
| dependsOn: ManualApproval | ||
| condition: and(succeeded(), ne(variables['Skip.PublishPackage'], 'true')) | ||
| steps: | ||
| - template: /eng/common/pipelines/templates/steps/sparse-checkout.yml | ||
|
|
||
| - download: current | ||
| displayName: Download ${{parameters.PipelineArtifactName}} artifact | ||
| artifact: ${{parameters.PipelineArtifactName}} | ||
|
|
||
| - task: PowerShell@2 | ||
| displayName: Yank Crates | ||
| env: | ||
| CARGO_REGISTRY_TOKEN: $(azure-sdk-cratesio-token) | ||
| inputs: | ||
| targetType: filePath | ||
| filePath: $(Build.SourcesDirectory)/eng/scripts/Yank-Crates.ps1 | ||
| arguments: | ||
| -CrateNames '${{artifact.name}}' | ||
| -PackageInfoDirectory '$(Pipeline.Workspace)/${{parameters.PipelineArtifactName}}/PackageInfo' | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why the double negative here?