deployments (55 functions) failing with message "Build failed with status: EXPIRED..."

[wojtek-iwarranty]

[REQUIRED] Environment info

firebase-tools:
13.8.0 or 13.10.2

Platform:
ubuntu-latest, via GitHubAction

[REQUIRED] Test case

run firebase deploy --force --only functions,firestore with a codebase containing 55 2nd Cloud Functions.
In my case it is ran via action w9jds/firebase-action@v13.10.2 (or w9jds/firebase-action@v13.8.0)

[REQUIRED] Steps to reproduce

Same as test case

[REQUIRED] Expected behavior

I would expect this deployment to go through since it is below the deployment quota of 80 functions per minute.

[REQUIRED] Actual behavior

Some of the functions fail to deploy with build with the following error:

Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted].
The build logs are empty and don't contain any useful information other than the build expiring.

I can see in Cloud Build console that some of the builds failed:

This happens for about 15 of the 55 functions. Here is a truncated output of the full deploy:

Run w9jds/firebase-action@v13.8.0
  with:
    args: deploy --force --config=firebase.json --only functions,firestore
  env:
    GCP_SA_KEY: ***
  
    PROJECT_ID: [redacted]
/usr/bin/docker run --name w9jdsfirebaseactionv1380_9e59c3 --label f3784f --workdir /github/workspace --rm -e  [redacted] -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/rabbit/rabbit":"/github/workspace" w9jds/firebase-action:v13.8.0 deploy --force --config=firebase.json --only functions,firestore
Storing GCP_SA_KEY in /opt/gcp_key.json
Exporting GOOGLE_APPLICATION_CREDENTIALS=/opt/gcp_key.json
setting firebase project to [redacted]
Now using alias [redacted] ([redacted])

=== Deploying to '[redacted]'...

i  deploying firestore, functions
i  firestore: reading indexes from apps/[redacted]/firestore.indexes.json...
i  cloud.firestore: checking apps/[redacted]/firestore.rules for compilation errors...
✔  cloud.firestore: rules file apps/[redacted]/firestore.rules compiled successfully
i  functions: preparing codebase [redacted]-functions for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
⚠  functions: package.json indicates an outdated version of firebase-functions. Please upgrade using npm install --save firebase-functions@latest in your functions directory.
⚠  functions: Please note that there will be breaking changes when you upgrade.
i  functions: Loading and analyzing source code for codebase [redacted]-functions to determine what to deploy
Serving at port 8640

Firebase Admin Version: 11.11.0

i  functions: preparing dist/apps/[redacted]-functions directory for uploading...
i  functions: packaged /github/workspace/dist/apps/[redacted]-functions (some MB) for uploading
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API eventarc.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: required API storage.googleapis.com is enabled
i  functions: generating the service identity for pubsub.googleapis.com...
i  functions: generating the service identity for eventarc.googleapis.com...
i  firestore: deploying indexes...
i  firestore: latest version of apps/[redacted]/firestore.rules already up to date, skipping upload...
✔  firestore: deployed indexes in apps/[redacted]/firestore.indexes.json successfully for (default) database
✔  functions: dist/apps/[redacted]-functions folder uploaded successfully
✔  firestore: released rules apps/[redacted]/firestore.rules to cloud.firestore
i  functions: updating Node.js [2](https://github.com/iwarranty-io/rabbit/actions/runs/[redacted]#step:9:2)0 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success]full(europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
i  functions: updating Node.js 20 (2nd Gen) function [redacted]-functions:[redacted-success](europe-west1)...
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success]full(europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=europe-west1/AA
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.
✔  functions[[redacted]-functions:[redacted-success](europe-west1)] Successful update operation.

Functions deploy had errors with the following functions:
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
	[redacted]-functions:[redacted](europe-west1)
i  functions: cleaning up build files...
⚠  functions: Unhandled error cleaning up build images. This could result in a small monthly bill if not corrected. You can attempt to delete these images by redeploying or you can delete them manually at https://console.cloud.google.com/artifacts/docker/[redacted]-[redacted]/europe-west1/gcf-artifacts
Error: There was an error deploying functions:
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1
- Error Failed to update function [redacted] in region europe-west1

It's actually reopening for #5967 ...

[charifield]

Following from previous ticket. Encountering the same issues

[gersonlimadev]

The same errors are happening here to me

[SherpaMiguel]

Blocking issue since last week.

[toddmotto]

It amazes me how this is going to badly wrong again @joehan the previous 'fix' suggested by running firebase experiments:enable functionsv2deployoptimizations now does not work, what are we supposed to do to deploy vital function updates?

[raymond-tetteh]

Experiencing the same issues. Can't deploy my functions.

[onn-software]

Same here, 51 functions, and it worked for years. Now I have 'Build failed with status: EXPIRED and message: An unexpected error occurred'.

Edit: Also, Gitlab build went from a successful deploy in ±5 minutes, to a failing deploy in ±12 minutes. Which might be correlated to the EXPIRED error.

[halafi]

same issue, also sometimes deployment is skipped when there are changes to be deployed, thinking to --force deploy each function manually

[Endran]

We encounter the same. Breaking up the build in doesn't work either, each of the groups takes forever.

firebase deploy --only functions:group1
firebase deploy --only functions:group2
firebase deploy --only functions:group3
firebase deploy --except functions

Each function group contains around 10 functions, and takes over 10 minutes, where it used to deploy all in a few minutes.

The last successful deploy was on May 30th, first failing deploy June 2nd. What did the Firebase/GCP team change in the meantime?

[wesleygonalv]

We are having the same issue: we have over 100 functions, but we can only deploy a few of them.

We considered breaking them into smaller groups, but as @Endran mentioned, it unfortunately did not solve the problem.

We have been experiencing this issue since the date mentioned by others and are still waiting for updates.

[aalej]

Hey everyone, thanks for raising this and apologies for the issue this has caused. Our engineering team has been notified about this issue.

While waiting for our engineering team’s response, I’m currently trying to replicate this by deploying ~60 v2 functions, but so far I haven’t been able to encounter the same error. I would like to ask if anyone could share the ff:

1. Do you have the experiment functionsv2deployoptimizations enabled?
2. Which region are you deploying the function to? E.g. europe-west1, us-central1, etc.
3. What functions are being deployed? E.g. onRequest, beforeUserSignedIn, onDocumentWritten, etc.