Unbound variable error from the latest version of dotnet-install.sh

[ujpandey]

The script exits with error due to an unbound variable. The code there seems to be assuming that the variable is set but that's only true if the download failed as of this commit which unsets the variables explicitly when starting the download.

On MacOS:

./dotnet-install.sh -c Current
dotnet-install: Note that the intended use of this script is for Continuous Integration (CI) scenarios, where:
dotnet-install: - The SDK needs to be installed without user interaction and without admin rights.
dotnet-install: - The SDK installation doesn't need to persist across multiple CI runs.
dotnet-install: To set up a development environment or to run apps, use installers rather than this script. Visit https://dotnet.microsoft.com/download to get the installer.

dotnet-install: Downloading primary link https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.102/dotnet-sdk-5.0.102-osx-x64.tar.gz
./dotnet-install.sh: line 908: http_code: unbound variable

On Linux:

./dotnet-install.sh -c Current
dotnet-install: Note that the intended use of this script is for Continuous Integration (CI) scenarios, where:
dotnet-install: - The SDK needs to be installed without user interaction and without admin rights.
dotnet-install: - The SDK installation doesn't need to persist across multiple CI runs.
dotnet-install: To set up a development environment or to run apps, use installers rather than this script. Visit https://dotnet.microsoft.com/download to get the installer.

dotnet-install: Downloading primary link https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.102/dotnet-sdk-5.0.102-linux-x64.tar.gz
./dotnet-install.sh: line 908: http_code: unbound variable

[BruceForstall]

@bozturkMSFT

[danmoseley]

@bozturkMSFT it would be good to understand how this bug made it through validation, as it is taking out our builds, what are your thoughts? Perhaps there's some more validation we can add.

[dougbu]

/fyi @dotnet/aspnet-build

[dougbu]

Offline is fine but I would appreciate a root cause analysis, not of the bug itself but of the rollout containing the bug.

[dave-yotta]

This has broken our deployments and halted building images in dev - when will the fix be rolled out? Very lucky production was not due to go out today! We use roslyn in production so the sdk is neccesary.

[dave-yotta]

@bozturkMSFT don't suppose you know any ETA when this will reach https://dot.net/v1/dotnet-install.sh? Tried just now still not working. Sorry if I'm atting the wrong person :)

[bekir-ozturk]

Hi @dave-yotta,
We don't own the website and the deployment is done by our colleagues in the US. Once started, it should take around 30 minutes. Though I'm afraid they won't be available to start it until morning (which is in 5-6 hours).
I am still looking for ways to expedite this.

[ManickaP]

Hi @bozturkMSFT, I hit this problem in the middle of conversion of our repo from master to main. I have already disabled maestro subscription for our repo. Do I need to re-enable it in order to get the fixed script? I know it hasn't been published yet, I'm just asking if getting the fix depends on working maestro subscription?

[dave-yotta]

@bozturkMSFT is there a way we can pin to a specific version of dotnet-install ?

[bekir-ozturk]

Hi @ManickaP,
Re-enabling maestro subscription shouldn't be necessary. The URL to download the install script stays the same, we only update the content.

[bekir-ozturk]

@dave-yotta For the time being, no. you can update your pipeline to fetch the script from GitHub though.

In that case, you would be using
https://raw.githubusercontent.com/dotnet/install-scripts/master/src/dotnet-install.sh
instead of
https://dot.net/v1/dotnet-install.sh

[ElvenSpellmaker]

Why doesn't MS give us a link which is pinned to a known version, CI builds need to be repeatable and giving us a link that is effectively tied to master is almost useless.

There have been many similar problems with RVM in the past whereby the linked installer breaks and breaks CI builds and it causes people to have to tie things to commits.

[dave-yotta]

Can we pin to the git ref here at least by doing https://raw.githubusercontent.com/dotnet/install-scripts/7a9d5dcab92cf131fc2d8977052f8c2c2d540e22/src/dotnet-install.sh or something?

[rarkins]

"Not breaking things in the first place" is nice, but not always possible. Unless the team who writes and publishes this script can control when it's deployed to https://dot.net, then it shouldn't be the recommended way to install.

[dave-yotta]

@bozturkMSFT Any update? This has begun taking down production clusters on EBS because it restarts them by building from dockerfile...

Edit: Devops team are hotfixing all services to pin this to the git ref.

[bekir-ozturk]

No updates yet, but hopefully soon.

[christopherdalton]

Can we pin to the git ref here at least by doing https://raw.githubusercontent.com/dotnet/install-scripts/7a9d5dcab92cf131fc2d8977052f8c2c2d540e22/src/dotnet-install.sh or something?

Is this script going to be available after the issue has been resolved? We don't want to use it and then have services go offline because the script is no longer available?

[bekir-ozturk]

Yes. The URL should continue to work regardless if the script was published to https://dot.net/v1/dotnet-install.sh or not.

[dave-yotta]

I'm not sure if the commit 7a9d5dc would be tidied up by git garbage collection on github at some point - I think this only happens if it becomes unreachable from any HEAD, but I don't know enough about git to be sure on this one.

[bekir-ozturk]

We expect the fix to be available at https://dot.net/v1/dotnet-install.sh within the next 30 minutes.

[AArnott]

@dave-yotta I found I could pin an older version from before the regression by downloading from https://raw.githubusercontent.com/dotnet/install-scripts/1ebb108764c092e7a314ff3fe1388f582cbcf89a/src/dotnet-install.ps1

AArnott/Library.Template#91

[bekir-ozturk]

We are waiting for the final approval in the CD pipeline, so it looks like we will pass the 30 minute mark. I will update here once the publishing is complete.

[bekir-ozturk]

Fix is now live. It may still take a few minutes for the caches to update.

Please let me know if the issue still persists.

[dave-yotta]

@dave-yotta I found I could pin an older version from before the regression by downloading from https://raw.githubusercontent.com/dotnet/install-scripts/1ebb108764c092e7a314ff3fe1388f582cbcf89a/src/dotnet-install.ps1

AArnott/Library.Template#91

@AArnott I though the same in my above comment - but there's a good chance 1ebb108764c092e7a314ff3fe1388f582cbcf89a will be cleaned up by git at some point in the future, so be careful. A good step would be tags added to this repo, perhaps we can ask for this at least.

[AArnott]

@dave-yotta Nah. GitHub cannot remove commits that are referenced by refs. The commit I chose is in the history of the default branch, so unless they do a force push and remove the commit I selected from history (which seems very unlikely), I'm safe.

[dave-yotta]

@AArnott true - or if it gets rebased at some point but again that'd be a forced change to a head - I don't know, we're definitely too scared to do that here based on what's already happened that shouldn't really have happened :)

[am11]

Back in dotnet/cli days, these scripts used to live in that repo and the workflow used to be: make a PR (don't directly push to master no matter how urgent it is..) -> all CI legs get green -> PR gets reviewed/approved/merged -> changes get deployed to staging -> folks used to run smoke test -> deploy to production (dot.net). This workflow ensured that the commonly used parts are always in working state.

Note, these public-facing scripts are used beyond github/dotnet and github/microsoft orgs; they are used in the wild by other CI providers and in miscellaneous dev workflows; so extra precautions are necessary. During the downtime of merely few hours, thousands of build failed in many CI systems.

[bekir-ozturk]

@am11 The process is still the same as before. In fact, the tests from the the original location of the scripts were also brought into this repo together with the scripts. Although we added some new tests as we made improvements, there seems to be some areas not yet covered by the tests. We are working on this and the pipelines will be updated in the coming days.

[dave-yotta]

@am11 The process is still the same as before. In fact, the tests from the the original location of the scripts were also brought into this repo together with the scripts. Although we added some new tests as we made improvements, there seems to be some areas not yet covered by the tests. We are working on this and the pipelines will be updated in the coming days.

This is good - although I don't recall seeing any GitHub checks linked on the PR that fixed this issue? That would add some reassurance at least (or maybe I am misremembering EDIT: They are just not displayed inline with the PR like I'm used to, they are on the checks tab :) ).

I would add - this could have been picked up by simply running the installer once (on a debian environment I guess). Such tests would be extremely simple to add, and although not functional/unit tests, do directly test the use-cases we have.

For example:

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim

RUN curl -sSL https://dot.net/v1/dotnet-install.sh > dotnet-install.sh
RUN chmod u+x dotnet-install.sh
RUN ./dotnet-install.sh --channel 3.1 --install-dir /usr/share/dotnet

This is all that would have been needed to repro the problem we got here (obviously not the dot.net hostname part but you get my drift :)).