Skip to content

Prevent Windows GHA CI disk exhausation #468

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 15, 2021
Merged

Conversation

DellaBitta
Copy link
Contributor

@DellaBitta DellaBitta commented Jun 15, 2021

Windows BoringSSL builds are now larger due to #454. The checkout & build partition (D:) hits capacity when attempting to install the Google Cloud SDK, which is what we use to upload the final integraion_test build artifacts to storage.

This change moves the arhives destined for GCS to the C: drive, then installs the Cloud SDK after they're off of D:. The migration of the integration tests only occurs on Windows, for now.

Some notes:

  • We can’t set the build_testapps.py’s --output_directory flag to directly output to the C: directory because python’s relpath operations fail when working across different drives.
  • The github.workspace env var can be changed to override check-out & installation paths, but the GHA team notes potential unknown side effects. Therefore I didn’t test this.
  • We couldn't use mv the results to C: as Firestore's build results include symlinks. Using tar to ferry the files instead.

@google-cla google-cla bot added the cla: yes label Jun 15, 2021
@DellaBitta DellaBitta added tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). and removed cla: yes labels Jun 15, 2021
@google-cla google-cla bot added the cla: yes label Jun 15, 2021
@github-actions github-actions bot added tests: in-progress This PR's integration tests are in progress. and removed tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). labels Jun 15, 2021
@github-actions github-actions bot added the tests: failed This PR's integration tests failed. label Jun 15, 2021
@firebase-workflow-trigger firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Jun 15, 2021
@DellaBitta DellaBitta added tests-requested: quick Trigger a quick set of integration tests. and removed tests: failed This PR's integration tests failed. labels Jun 15, 2021
@github-actions github-actions bot added tests: in-progress This PR's integration tests are in progress. and removed tests-requested: quick Trigger a quick set of integration tests. labels Jun 15, 2021
@github-actions
Copy link

github-actions bot commented Jun 15, 2021

✅  Integration test succeeded!

Requested by @DellaBitta on commit d7b7a13
Last updated: Tue Jun 15 12:06:11 PDT 2021
View integration test results

@firebase-workflow-trigger firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Jun 15, 2021
@github-actions github-actions bot added the tests: succeeded This PR's integration tests succeeded. label Jun 15, 2021
vimanyu
vimanyu previously approved these changes Jun 15, 2021
Copy link
Contributor

@vimanyu vimanyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking care of this! One minor note but LGTM otherwise.

set -x
mkdir -p ${{env.GCS_UPLOAD_DIR}}/scripts/gha/integration_testing
mkdir -p ${{env.GCS_UPLOAD_DIR}}/scripts/gha-encrypted
tar -chf ${{env.GCS_UPLOAD_DIR}}/ta.tar ta
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for readability, could you put a comment here what the "ta" directory is? Others are easy to follow.

@vimanyu
Copy link
Contributor

vimanyu commented Jun 15, 2021

Just a note, the boringssl change (download tarball -> checkout specific tag in repo) increased the boringssl source code directory size from 100M to 280M.

@jonsimantov
Copy link
Contributor

jonsimantov commented Jun 15, 2021

Just a note, the boringssl change (download tarball -> checkout specific tag in repo) increased the boringssl source code directory size from 100M to 280M.

Would checking out with --depth 1 help this?

@DellaBitta
Copy link
Contributor Author

I'm going to try out some new git cleanup code that I've found to see if I can reduce the size of the checkout, but I'd like to merge this for now so that we can effectively use integration tests moving forward and I can roll back this change if it's not necessary for now.

@DellaBitta DellaBitta merged commit 7ae9569 into main Jun 15, 2021
@DellaBitta DellaBitta deleted the fix/windows_disk_exhaustion branch June 15, 2021 21:03
@firebase firebase locked and limited conversation to collaborators Jul 16, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla: yes tests: succeeded This PR's integration tests succeeded.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants