ESRP publishing #3065

danieljurek · 2025-09-29T17:59:08Z

Do I need to review this as a service owner?

Probably not. This PR is for EngSys. You're probably seeing this review because I updated your ci.yml file to use "release intent" which allows you to choose which packages to release at release build time. We in EngSys will merge this after reviewing.

Notes

Fixes #3035

Release packages using ESRP
Release intent (check boxes to release packages)
You can run live tests without releasing packages
Packages are released in dependency order
Dependency enforcement

Updated runs with release intent:

Sample pipeline run with 2 release crates: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=5405078&view=results

Sample pipeline run with no releases and live tests: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=5405113&view=results -- In this case note that necessary dependencies were built (e.g. typespec) even though it was outside the scope of the service.

Sample pipeline run with no releases (also no live tests): https://dev.azure.com/azure-sdk/internal/_build/results?buildId=5405079&view=results

Sample releases:

Open questions

Right now Pack-Crates.ps1 fails in release scenarios when releasing a package that has unreleased dependencies (e.g. dependency_crate = { workspace = true }. The expectation here is that dependencies will go from workspace = true to { version = "1.2.3" } when releasing. Is this a correct expectation?

.vscode/cspell.json

danieljurek · 2025-09-29T22:56:21Z

eng/scripts/Pack-Crates.ps1

+    $orderMatches = $true
+    for ($i = 0; $i -lt $PackageNames.Count; $i++) {
+      if ($packages[$i].name -ne $PackageNames[$i]) {
+        $orderMatches = $false


This change uses Artifact names coming from the service's sdk/<service>/ci.yml file to supply the -PackageNames parameter. The sorting of those artifact names must match the sorted order created by this script for a successful release (while we're releasing one package per-ESRP step).

It's possible to change the sorting by altering the placement of workspace.members in the root Cargo.tml file.

If the sorting is incorrect, you'll get an error at release time that the order of packages in -PackageNames is not correct. To fix, update the order of the service's ci.yml file to match the order that the script provides and then run the release.

This validation currently only happens during release because the pipeline yaml templates aren't aware of the various libraries' dependencies in the state of their Cargo.toml files (e.g. workspace dependencies vs. released dependencies).

This should probably be documented somewhere - as a comment in this file, or in some release docs somewhere in the repo.

Heath and I expanded on this a few moments ago:

Relying on the user to get the order right in ci.yml is going to be a source of problems. Instead, we should see if the ordering can be done "dynamically" in the job. This will require some more YAML

Copilot

Pull Request Overview

This PR implements ESRP (Enterprise Software Signing Release Pipeline) publishing for Azure SDK for Rust packages, replacing the previous cargo publish approach. The changes introduce release package selection parameters and dependency enforcement mechanisms to ensure packages are released in the correct order.

Adds release intent parameters to each service CI file to allow selective package releases
Implements dependency enforcement and order validation in the Pack-Crates.ps1 script
Replaces cargo publish with ESRP release tasks in the pipeline templates
Updates package artifact handling to work with .crate files instead of Cargo.toml manifests

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
sdk/*/ci.yml	Adds release intent parameters for each package in the service directories
eng/scripts/Pack-Crates.ps1	Adds dependency validation and order enforcement for release builds
eng/scripts/Language-Settings.ps1	Updates package pattern and artifact processing to handle .crate files
eng/pipelines/templates/stages/archetype-sdk-client.yml	Filters artifacts based on release intent parameters
eng/pipelines/templates/stages/archetype-rust-release.yml	Replaces cargo publish with ESRP release workflow
eng/pipelines/templates/jobs/pack.yml	Adds dependency requirement enforcement for manual builds
.vscode/cspell.json	Minor formatting cleanup and adds spell check exceptions

sdk/canary/ci.yml

eng/scripts/Pack-Crates.ps1

eng/scripts/Language-Settings.ps1

hallipr · 2025-09-30T00:18:00Z

eng/scripts/Pack-Crates.ps1


+  if ($RequireDependencies) {
+    $unspecifiedPackages = $packages.name | Where-Object { $_ -notin $PackageNames }
+    if ($unspecifiedPackages.Count -gt 0) { 


This doesn't seem to allow for packages outside of this pipeline run to be accommodate the current release. e.g. :
azure_core_amqp depends on azure_core
azure_core depends on typespec
We're building core, but typespec isn't shipped in the core pipeline

We should be able to build with $PackageNames = ('azure_core', 'azure_core_amqp'), even though there's that lingering typespec dependency because we know that, before releasing those packages, we'll be releasing typespec in a separate pipeline.

This is different from $PackageNames = ('azure_core_amqp') where we know azure_core_amqp depends on azure_core, we know this is the pipeline that should build it and we know we're not building it.

I'm proposing that we call Pack-Crates.ps1 with a list of all the artifacts this pipeline is responsible for, along with a list of packages we intend on releasing. If any dependency is in the list we're responsible for but aren't releasing, then we error.

Allowed

Pack-Crates.ps1 -Packages 'azure_core' -PipelinePackages 'azure_core_amqp', 'azure_core'

Should error

Pack-Crates.ps1 -Packages 'azure_core_amqp' -PipelinePackages 'azure_core_amqp', 'azure_core'

In the scenario where we're releasing azure_core_amqp and there's a dependency on typespec how does azure_core_amqp's Cargo.toml file express the dependency on typespec?

https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/core/azure_core_amqp/Cargo.toml#L33

As it's set up in this PR, the core pipeline would fail if it's depending on a release of typespec that hasn't yet completed. So you need to queue and release typespec and then you can queue and release your packages in core that depend on typespec.

For packing/publishing cargo will elevate workspace dependencies and remove any path properties from the dependencies, leaving their other properties:

given the workspace dependency in the root toml:

[workspace.dependencies.typespec] default-features = false path = "sdk/typespec" version = "0.9.0"

azure_core_amqp's dependency would process as:

typespec = { workspace = true, features = ["amqp"] } => typespec = { path = "sdk/typespec", version = "0.9.0", features = ["amqp"] } => typespec = { version = "0.9.0", features = ["amqp"] }

Where typespec 0.9.0 is currently unreleased.

If we queue and release typespec, we need to uptake the version bump PR to rev the unreleased, path version to 0.10.0.

Doing this would cause azure_core_amqp to depend on the new, unreleased 0.10.0

@hallipr and I went over this on the whiteboard and it works out to mostly YAML changes:

Check boxes in the pipeline queuing UI signals release intent

If no boxes are checked (basically, run live tests but don't release anything) and if a pipeline is run manually then treat it as Pack-Crates.ps1 -OutputPath '...' -PackageNames all,packages,in,build -RequireDependencies:$false

If boxes are checked, that signals release intent, then use -RequireDependencies:$true

@hallipr - Can you have another look? I updated the YAML. I'm debating pulling the inline PowerShell into a file so we do local iteration but it's probably not going to change much as written.

analogrelay

Approving from Cosmos

…s in the service and run live tests"

sdk/core/ci.yml

heaths · 2025-10-01T23:28:08Z

eng/scripts/Pack-Crates.ps1

+    $orderMatches = $true
+    for ($i = 0; $i -lt $PackageNames.Count; $i++) {
+      if ($packages[$i].name -ne $PackageNames[$i]) {
+        $orderMatches = $false


This should probably be documented somewhere - as a comment in this file, or in some release docs somewhere in the repo.

.vscode/cspell.json

… can include name separators

hallipr · 2025-10-02T20:13:40Z

eng/pipelines/templates/jobs/pack.yml

+    - pwsh: |
+        $artifacts = '${{ convertToJson(parameters.Artifacts) }}' | ConvertFrom-Json
+        $requireDependencies = $true
+        $artifactsToBuild = $artifacts | Where-Object { $_.releaseInBatch -ne 'False' }


Why the double negative here?

github-actions bot added Azure.Core The azure_core crate Azure.Identity The azure_identity crate Cosmos The azure_cosmos crate KeyVault Key Vault Storage Storage Service (Queues, Blobs, Files) labels Sep 29, 2025

github-project-automation bot added this to Azure SDK for Key Vault and Azure Identity SDK Improvements Sep 29, 2025

github-project-automation bot moved this to Untriaged in Azure SDK for Key Vault Sep 29, 2025

github-project-automation bot moved this to Untriaged in Azure Identity SDK Improvements Sep 29, 2025

danieljurek force-pushed the djurek/esrp-publish branch from 7549c10 to 13de88a Compare September 29, 2025 18:03

danieljurek changed the title ~~ESRP publishing~~ [DRAFT DO NOT REVIEW] ESRP publishing Sep 29, 2025

danieljurek commented Sep 29, 2025

View reviewed changes

.vscode/cspell.json Outdated Show resolved Hide resolved

danieljurek commented Sep 29, 2025

View reviewed changes

danieljurek marked this pull request as ready for review September 29, 2025 23:00

danieljurek requested review from hallipr, weshaggard and heaths as code owners September 29, 2025 23:00

Copilot AI review requested due to automatic review settings September 29, 2025 23:00

danieljurek requested review from RickWinter, LarryOsterman, vincenttran-msft, jalauzon-msft, a team, chlowell and ronniegeraghty as code owners September 29, 2025 23:00

Copilot AI reviewed Sep 29, 2025

View reviewed changes

sdk/canary/ci.yml Outdated Show resolved Hide resolved

eng/scripts/Pack-Crates.ps1 Show resolved Hide resolved

eng/scripts/Language-Settings.ps1 Outdated Show resolved Hide resolved

danieljurek changed the title ~~[DRAFT DO NOT REVIEW] ESRP publishing~~ ESRP publishing Sep 29, 2025

hallipr reviewed Sep 30, 2025

View reviewed changes

analogrelay approved these changes Sep 30, 2025

View reviewed changes

danieljurek added 13 commits October 1, 2025 02:50

Test: remove package verison updating from release. Should fail Packing.

4735144

Better testing: turn off TestPipeline

8f8ebc3

Undo test changes

f43a964

Move RequireDependencies into pack.yml

fe472fc

Move RequireDependencies into pack.yml

6423c8d

Pack-Crates.ps1: Use PackageInfoDirectory

247de8e

-RequireDependencies

726d7ac

Syntax

5997f54

Formatting/cleanup

37ea97a

cspell: allowCompoundWords: true

fe8cf19

Remove formatting

20bed97

Spelling

debd2f2

Review feedback: No artifacts selected means "just build all artifact…

455288c

…s in the service and run live tests"

danieljurek force-pushed the djurek/esrp-publish branch from 36c0bf2 to 455288c Compare October 1, 2025 03:03

danieljurek added 3 commits October 1, 2025 03:06

convertToJson

fa4ab88

Only create the stage if there are artifacts to release

da1fda7

'False'

1a298a0

danieljurek self-assigned this Oct 1, 2025

danieljurek added this to Azure SDK EngSys 🍔🌭 Oct 1, 2025

github-project-automation bot moved this to 🤔 Triage in Azure SDK EngSys 🍔🌭 Oct 1, 2025

danieljurek moved this from 🤔 Triage to 🔬 Dev in PR in Azure SDK EngSys 🍔🌭 Oct 1, 2025

More error throwing

d3c1de1

heaths requested changes Oct 1, 2025

View reviewed changes

github-project-automation bot moved this from Untriaged to In Progress in Azure Identity SDK Improvements Oct 1, 2025

github-project-automation bot moved this from Untriaged to In Progress in Azure SDK for Key Vault Oct 1, 2025

danieljurek added 3 commits October 2, 2025 03:44

Review feedback: Don't enable allowCompoundWords, also release_ names…

da077ef

… can include name separators

Fix typespec_macros

11ad80d

Artifact ordering for Pack-Crates.ps1

8e71351

hallipr reviewed Oct 2, 2025

View reviewed changes

heaths self-requested a review October 2, 2025 22:53

ESRP publishing #3065

Are you sure you want to change the base?

ESRP publishing #3065

Uh oh!

Conversation

danieljurek commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Do I need to review this as a service owner?

Notes

Open questions

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danieljurek Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danieljurek Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

analogrelay left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danieljurek commented Sep 29, 2025 •

edited

Loading

danieljurek Sep 30, 2025 •

edited

Loading

danieljurek Sep 30, 2025 •

edited

Loading