Skip to content

Prevent Windows GHA CI disk exhausation #468

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 15, 2021
Merged

Prevent Windows GHA CI disk exhausation #468

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f25df67
move ta artifacts to c for windows uploads
DellaBitta Jun 14, 2021
7308aa7
attempt to rework cancelled clause
DellaBitta Jun 14, 2021
4a69666
cancelled
DellaBitta Jun 14, 2021
02c6950
comment describing the ta directory contents
DellaBitta Jun 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 54 additions & 17 deletions .github/workflows/integration_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,20 +253,20 @@ jobs:
with:
ref: ${{needs.prepare_matrix.outputs.github_ref}}
submodules: true
- name: Set env vars (ubuntu)
- name: Set env vars (Linux)
if: startsWith(matrix.os, 'ubuntu')
run: echo "VCPKG_TRIPLET=x64-linux" >> $GITHUB_ENV
- name: Set env vars (macos)
- name: Set env vars (macOS)
if: startsWith(matrix.os, 'macos')
run: echo "VCPKG_TRIPLET=x64-osx" >> $GITHUB_ENV
- name: Set env vars (windows)
- name: Set env vars (Windows)
shell: bash
if: startsWith(matrix.os, 'windows')
run: echo "VCPKG_TRIPLET=x64-windows-static" >> $GITHUB_ENV
- name: Set env vars(all)
- name: Set env vars (all)
shell: bash
run: echo "VCPKG_RESPONSE_FILE=external/vcpkg_${{ env.VCPKG_TRIPLET }}_response_file.txt" >> $GITHUB_ENV
- name: Add msbuild to PATH (windows)
- name: Add msbuild to PATH (Windows)
if: startsWith(matrix.os, 'windows')
uses: microsoft/[email protected]
- name: Cache vcpkg C++ dependencies
Expand All @@ -293,12 +293,12 @@ jobs:
run: |
pip install -r scripts/gha/requirements.txt
python scripts/gha/restore_secrets.py --passphrase "${{ secrets.TEST_SECRET }}"
- name: Install OpenSSL (windows)
- name: Install OpenSSL (Windows)
if: matrix.ssl_variant == 'openssl' &&
startsWith(matrix.os, 'windows')
run: |
choco install openssl -r
- name: Install OpenSSL (MacOS)
- name: Install OpenSSL (macOS)
if: matrix.ssl_variant == 'openssl' &&
startsWith(matrix.os, 'macos')
run: |
Expand Down Expand Up @@ -341,7 +341,7 @@ jobs:
--artifact_name "desktop-${{ matrix.os }}-${{ matrix.ssl_variant }}" \
--noadd_timestamp \
--short_output_paths \
${additional_flags[*]}
${additional_flags[*]}
- name: Upload Desktop integration tests artifact
uses: actions/[email protected]
if: ${{ !cancelled() }}
Expand All @@ -356,6 +356,39 @@ jobs:
name: log-artifact
path: build-results-desktop-${{ matrix.os }}-${{ matrix.ssl_variant }}*
retention-days: ${{ env.artifactRetentionDays }}
- name: Cleanup Local Copies of Uploaded Artifacts
shell: bash
run: |
rm -rf testapps-desktop-${{ matrix.os }}-${{ matrix.ssl_variant }}
- name: Configure GCS Upload Directory (Linux, macOS)
if: (startsWith(matrix.os, 'ubuntu') || startsWith(matrix.os, 'macos')) && !cancelled()
shell: bash
run: echo "GCS_UPLOAD_DIR=${{ github.workspace }}" >> $GITHUB_ENV
- name: Configure GCS Upload Directory (Windows)
# For Windows move the uploads to C: to create enough space for the
# Cloud SDK installation on D:.
if: startsWith(matrix.os, 'windows') && !cancelled()
shell: bash
run: echo "GCS_UPLOAD_DIR=/c/fpl-upload" >> $GITHUB_ENV
- name: Move GCS Artifacts to GCS Upload Dir
# Copy the artifacts and also the upload scripts since python's relpath
# produces errors when called across two different drives. Use tar to
# move the files as it neatly deepcopies firestore's symlinks.
if: ${{ (env.GCS_UPLOAD_DIR != github.workspace) && !cancelled() }}
shell: bash
run: |
set -x
mkdir -p ${{env.GCS_UPLOAD_DIR}}/scripts/gha/integration_testing
mkdir -p ${{env.GCS_UPLOAD_DIR}}/scripts/gha-encrypted
# The ta directory contains the integration testapps to archive.
tar -chf ${{env.GCS_UPLOAD_DIR}}/ta.tar ta
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for readability, could you put a comment here what the "ta" directory is? Others are easy to follow.

rm -rf ta
tar xf ${{env.GCS_UPLOAD_DIR}}/ta.tar -C ${{env.GCS_UPLOAD_DIR}}
rm ${{env.GCS_UPLOAD_DIR}}/ta.tar
cp scripts/gha/gcs_uploader.py ${{env.GCS_UPLOAD_DIR}}/scripts/gha-encrypted/
cp scripts/gha/integration_testing/gcs.py ${{env.GCS_UPLOAD_DIR}}/scripts/gha/integration_testing/
cp scripts/gha/gcs_uploader.py ${{env.GCS_UPLOAD_DIR}}/scripts/gha/
cp scripts/gha-encrypted/gcs_key_file.json ${{env.GCS_UPLOAD_DIR}}//scripts/gha-encrypted/
- name: Set CLOUDSDK_PYTHON (Windows)
shell: bash
if: startsWith(matrix.os, 'windows') && !cancelled()
Expand All @@ -364,8 +397,12 @@ jobs:
if: ${{ !cancelled() }}
uses: google-github-actions/setup-gcloud@master
- name: Upload Desktop Artifacts to GCS
shell: bash
if: ${{ !cancelled() }}
run: python scripts/gha/gcs_uploader.py --testapp_dir ta --key_file scripts/gha-encrypted/gcs_key_file.json
run: |
pushd ${{env.GCS_UPLOAD_DIR}}
python scripts/gha/gcs_uploader.py --testapp_dir ta --key_file scripts/gha-encrypted/gcs_key_file.json
popd
- name: Add failure label
# We can mark a failure as soon as any one test fails.
if: ${{ needs.check_trigger.outputs.should_update_labels && failure() && !cancelled() }}
Expand Down Expand Up @@ -431,20 +468,20 @@ jobs:
with:
ref: ${{needs.prepare_matrix.outputs.github_ref}}
submodules: true
- name: Set env vars (ubuntu)
- name: Set env vars (Linux)
if: startsWith(matrix.os, 'ubuntu')
run: echo "VCPKG_TRIPLET=x64-linux" >> $GITHUB_ENV
- name: Set env vars (macos)
- name: Set env vars (macOS)
if: startsWith(matrix.os, 'macos')
run: echo "VCPKG_TRIPLET=x64-osx" >> $GITHUB_ENV
- name: Set env vars (windows)
- name: Set env vars (Windows)
shell: bash
if: startsWith(matrix.os, 'windows')
if: startsWith(matrix.os, 'Windows')
run: echo "VCPKG_TRIPLET=x64-windows-static" >> $GITHUB_ENV
- name: Set env vars(all)
- name: Set env vars (all)
shell: bash
run: echo "VCPKG_RESPONSE_FILE=external/vcpkg_${{ env.VCPKG_TRIPLET }}_response_file.txt" >> $GITHUB_ENV
- name: Add msbuild to PATH (windows)
- name: Add msbuild to PATH (Windows)
if: startsWith(matrix.os, 'windows')
uses: microsoft/[email protected]
- name: Cache NDK
Expand Down Expand Up @@ -575,9 +612,9 @@ jobs:
with:
ref: ${{needs.prepare_matrix.outputs.github_ref}}
submodules: true
- name: Set env vars (macos)
- name: Set env vars (macOS)
run: echo "VCPKG_TRIPLET=x64-osx" >> $GITHUB_ENV
- name: Set env vars(all)
- name: Set env vars (all)
shell: bash
run: echo "VCPKG_RESPONSE_FILE=external/vcpkg_${{ env.VCPKG_TRIPLET }}_response_file.txt" >> $GITHUB_ENV
- name: Setup python
Expand Down