"Cannot parse Firebase url" on snapshot reference access when using RTDB via firebase functions. #746

filipesilva · 2020-07-22T15:51:06Z

[REQUIRED] Describe your environment

Operating System version: Windows 10
Browser version: Chrome 84
Firebase SDK version: 7.16.1
Firebase Product: database

[REQUIRED] Describe the problem

It's currently not possible to access the data snapshot reference when using RTDB via firebase functions.

I believe this is due to the check in https://github.com/firebase/firebase-js-sdk/blob/6af4c27743372ba531e8ce3d046ae2f81e8f5be1/packages/database/src/core/util/libs/parser.ts#L83-L91

That piece of code checks that parsedUrl.domain is localhost, but it seems to be emulator-test-1.localhost instead.

Steps to reproduce:

git clone https://github.com/filipesilva/firebase-emulator-parse-url
cd firebase-emulator-parse-url
yarn
yarn emulators
# open http://localhost:5001/emulator-test-1/us-central1/posts in the browser

You should see the following error log in the console:

i  functions: Beginning execution of "postsPushHandler"
>  [2020-07-22T15:49:53.227Z]  @firebase/database: FIREBASE FATAL ERROR: Cannot parse Firebase url. Please use https://<YOUR FIREBASE>.firebaseio.com
!  functions: Error: FIREBASE FATAL ERROR: Cannot parse Firebase url. Please use https://<YOUR FIREBASE>.firebaseio.com
    at fatal (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:341:11)
    at parseRepoInfo (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:1296:9)
    at RepoManager.databaseFromApp (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:14990:25)
    at Object.initStandalone (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15389:45)
    at DatabaseService.getDatabase (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:67:23)
    at FirebaseApp.database (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-app.js:232:24)
    at DataSnapshot.get ref [as ref] (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-functions\lib\providers\database.js:293:34)
    at D:\sandbox\firebase-emulator-parse-url\functions\index.js:17:24
    at cloudFunction (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-functions\lib\cloud-functions.js:132:23)
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:573:20
!  Your function was killed because it raised an unhandled error.

Relevant Code:

exports.postsPushHandler = functions.database.ref('/posts/{postId}').onCreate(snapshot => {
  console.log(snapshot.ref)
});

The text was updated successfully, but these errors were encountered:

filipesilva · 2020-07-22T15:52:02Z

cc @samtstern (because I've reported other emulator related issues to you)

samtstern · 2020-07-22T15:57:55Z

@filipesilva what version of firebase-functions are you using? This should be fixed in version 3.8.0 (assuming newest version of the CLI as well)

filipesilva · 2020-07-22T16:00:48Z

The repro (https://github.com/filipesilva/firebase-emulator-parse-url) is using [email protected]. Everything there is the latest as of today, I believe.

samtstern · 2020-07-22T16:50:56Z

@filipesilva sorry I should have clicked into that

samtstern · 2020-07-23T12:36:12Z

Ok a few things here:

I was able to reproduce this
If I change the initialization block to just admin.initializeApp() this error goes away.

samtstern · 2020-07-23T12:46:36Z

It looks like parseRepoInfo() is being called with a URL like:

https://emulator-test-1.localhost

This is a firebase-functions issue, transferring.

samtstern · 2020-07-23T12:50:51Z

Ok so I added some logging and extractInstanceAndPath is being called like this:

>  extractInstanceAndPath(projects/_/instances/emulator-test-1/refs/posts/-MCvh_klBc9LYP7JIlsJ, localhost)
>  return [https://emulator-test-1.localhost,/posts/-MCvh_klBc9LYP7JIlsJ]

The dataConstructor in onCreate is being given this raw.context:

{
  "eventType": "google.firebase.database.ref.create",
  "params": {
    "postId": "-MCviHJhohdsNE5uAJWZ"
  },
  "domain": "localhost",
  "resource": {
    "service": "firebaseio.com",
    "name": "projects/_/instances/emulator-test-1/refs/posts/-MCviHJhohdsNE5uAJWZ"
  },
  "timestamp": "2020-07-23T12:49:58.150Z",
  "eventId": "5/3OSuz1TP5pWvc/Lt0o9RDcN2M=",
  "authType": "ADMIN"
}

So the port is lost ... gonna have to think about this one.

samtstern · 2020-07-23T13:00:26Z

@filipesilva question for you: when you're running inside the Functions emulator and you also have the RTDB emulator running, but you do this:

admin.initializeApp({
   databaseURL: "https://emulator-test-1.firebaseio.com/",
   credential: admin.credential.applicationDefault()
});

const db = admin.database()

Do you still expect db to point to the emulator? To me it looks like you're explicitly trying to access production, otherwise you could do admin.initializeApp() to accept the defaults.

So I'm trying to decide if the actual bug is that your posts function writes to the emulators at all.

filipesilva · 2020-07-23T14:04:15Z

@samtstern I don't quite care about setting credential and would be happy enough if that is just defaulted.

I do care about setting databaseURL because I test a sharded RTDB and it is important that functions I have can be routed to the correct shard, and that the shard is using the emulators. I think this is what happens today because, in my particular work setup, when I save security rules, it will update rules for all 100 shards I use, and they show up in the emulator UI. I could use a URL that's pointing to the emulators by encoding that as a build time variable.

I also care about setting databaseAuthVariableOverride as well because I want my writes from firebase functions to still obey validation rules.

Attempting to set it with no databaseURL (e.g. admin.initializeApp({databaseAuthVariableOverride: "something"}); ) results in a runtime error:

i  functions: Beginning execution of "posts"
!  functions: Error: Can't determine Firebase Database URL.
    at FirebaseDatabaseError.FirebaseError [as constructor] (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\utils\error.js:43:28)       
    at new FirebaseDatabaseError (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\utils\error.js:204:23)
    at DatabaseService.ensureUrl (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:89:15)
    at DatabaseService.getDatabase (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:56:26)
    at FirebaseApp.database (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-app.js:232:24)
    at Proxy.fn (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-namespace.js:280:45)
    at D:\sandbox\firebase-emulator-parse-url\functions\index.js:8:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:583:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:558:19
    at Generator.next (<anonymous>)
!  Your function was killed because it raised an unhandled error.

Attempting to set it with databaseURL but no credential also results in a runtime error:

i  functions: Beginning execution of "posts"
!  functions: Error: Only objects are supported for option databaseAuthVariableOverride
    at new Repo (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:12513:27)
    at RepoManager.createRepo (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15049:16)
    at RepoManager.databaseFromApp (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15014:25)
    at Object.initStandalone (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15389:45)
    at DatabaseService.getDatabase (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:67:23)
    at FirebaseApp.database (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-app.js:232:24)
    at Proxy.fn (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-namespace.js:280:45)
    at D:\sandbox\firebase-emulator-parse-url\functions\index.js:10:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:583:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:558:19
!  Your function was killed because it raised an unhandled error.

So in my concrete work example it does not seem possible to just go with the defaults. I must set credential and databaseURL in order to set databaseAuthVariableOverride. In some cases I must also set databaseURL to access shards.

samtstern · 2020-07-24T13:30:30Z

@filipesilva thanks for clarifying all of that ... there's a lot going on here and I'll have to think of the best way to solve it.

filipesilva · 2020-12-30T20:26:58Z

@samtstern heya, did you have time to think about what a resolution for this looks like?

samtstern · 2020-12-31T12:30:26Z

@filipesilva thanks for bumping this issue! I took another look today and think I found a really simple fix:
#838

Here are the functions I am using:

const functions = require('firebase-functions');
const admin = require('firebase-admin');

const instance = "fir-dumpster-secondary";

admin.initializeApp({
  databaseURL: `https://${instance}.firebaseio.com/`,
  credential: admin.credential.applicationDefault()
});

exports.posts = functions.https.onRequest(async (request, response) => {
  const db = admin.database();
  const ref = await db.ref('posts').push({
    date: new Date().toISOString()
  });
  response.send(`Added: ${ref}`);
});

exports.postsPushHandler = functions.database
  .instance(instance)
  .ref('/posts/{postId}')
  .onCreate(snapshot => {
    console.log("onCreate:", snapshot.ref.toString());
    return true;
  });

And here are the logs I get:

i  functions: Beginning execution of "posts"
i  functions: Finished "posts" in ~1s
i  functions: Beginning execution of "postsPushHandler"
>  onCreate: http://localhost:9000/posts/-MPslO6MOO53iuSWFUJL
i  functions: Finished "postsPushHandler" in ~1s

In the Emulator UI it all seems wired up correctly:

Would you mind testing this for me? You can check out my branch of this repo and then run npm run build. Then in your functions repo run npm install --save /path/to/your/clone/firebase-functions which should let you use the local version.

filipesilva · 2021-01-01T10:22:22Z

@samtstern thanks for getting back to me so quickly! I tested your branch build on https://github.com/filipesilva/firebase-emulator-parse-url and can confirm I no longer get the URL error, and that the write correctly goes through. I think that fixes it!

filipesilva · 2021-04-05T18:05:17Z

@samtstern today I was integrating your changes into our codebase, and I found a follow up problem with the fix you submitted.

I've updated the original repository with the most recent firebase versions and code to repro this problem in https://github.com/filipesilva/firebase-emulator-parse-url/commit/979d5c65808b15a423ee29aff0c33748c3c9a9eb.

The repro steps are still the same:

git clone https://github.com/filipesilva/firebase-emulator-parse-url
cd firebase-emulator-parse-url
yarn
yarn emulators
# open http://localhost:5001/emulator-test-1/us-central1/posts in the browser

This time the output is:

i  functions: Beginning execution of "posts"
i  functions: Finished "posts" in ~1s
i  functions: Beginning execution of "postsPushHandler"
>  snapshot ref http://localhost:9000/posts/-MXYBOwrxE0SXK0MOAmN
>  admin app options {
>    databaseURL: 'https://emulator-test-1.firebaseio.com/',
>    credential: RefreshTokenCredential {
>      httpAgent: undefined,
>      implicit: true,
>      refreshToken: RefreshToken {
>        clientId: '563584335869-fgrhgmd47bqnekij5i8b5pr03ho849e6.apps.googleusercontent.com',
>        clientSecret: 'j9iVZfS8kkCEFUPaAeJV0sAi',
>        refreshToken: '1//0dffIHm3tl_W7CgYIARAAGA0SNwF-L9IrkirUzBA22L9-rvZHCDa4jcM47eAhdK2USE-8miFIGJkYRn39CoFzhFTErl82HejNLJU',
>        type: 'authorized_user'
>      },
>      httpClient: HttpClient { retry: [Object] }
>    }
>  }
>  [2021-04-05T18:00:42.083Z]  @firebase/database: FIREBASE FATAL ERROR: Database initialized multiple times. Please make sure the format of the database URL matches with each database() call.
⚠  functions: Error: FIREBASE FATAL ERROR: Database initialized multiple times. Please make sure the format of the database URL matches with each database() call.
    at fatal (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:341:11)
    at RepoManager.createRepo (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15218:13)
    at RepoManager.databaseFromApp (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15185:25)
    at initStandalone (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15462:45)
    at Object.initStandalone$1 [as initStandalone] (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15594:12)
    at DatabaseService.getDatabase (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/lib/database/database-internal.js:80:23)
    at FirebaseApp.database (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/lib/firebase-app.js:158:24)
    at /Users/filipesilva/sandbox/firebase-emulator-parse-url/functions/index.js:19:40
    at cloudFunction (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-functions/lib/cloud-functions.js:134:23)
    at /Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:592:16
⚠  Your function was killed because it raised an unhandled error.

This logging comes from the updated handler:

exports.postsPushHandler = functions.database.ref('/posts/{postId}').onCreate(snapshot => {
   console.log("snapshot ref", snapshot.ref.toString())
   console.log("admin app options", adminApp.options)
   console.log("admin app db", adminApp.database())
 });

Attempting to access the database for the initialised admin app results in an error. There is no error if console.log("snapshot ref", snapshot.ref.toString()) is commented out.

It seems that using the snapshot ref initialises a database for the same url, but with a different URL.

Can this issue be reopened, or should I open a new issue?

samtstern · 2021-04-08T12:47:34Z

@filipesilva sure let's reopen this issue. So from that commit you sent it looks like there's a regression in one of the recent SDK updates. Could be firebase-admin, could be firebase-functions, etc. Do you think you could help me narrow it down by figuring out which of those changes in your commit leads to the regression?

filipesilva · 2021-04-22T14:58:59Z

Hi @samtstern! I tried going back to the original package versions for everything but firebase-functions in https://github.com/filipesilva/firebase-emulator-parse-url/commit/9ea2e64603b4f4d123fb248cce2cff012e964f95. The new problem still repros so I think it's related to firebase-functions.

I'm not even sure it's a regression though. I didn't test this originally with your fix because it required a larger time investment in updating our codebase, so it's possible that it was never working either.

samtstern self-assigned this Jul 22, 2020

samtstern transferred this issue from firebase/firebase-js-sdk Jul 23, 2020

google-oss-bot added the api: database label Jul 23, 2020

samtstern mentioned this issue Dec 31, 2020

Fix emulated database URL parse issue #838

Merged

samtstern closed this as completed in #838 Jan 11, 2021

samtstern reopened this Apr 8, 2021

samtstern removed their assignment Jul 16, 2021

cabljac added this to [Cloud] Extensions + Functions May 1, 2025

github-project-automation bot moved this to Needs Triage in [Cloud] Extensions + Functions May 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

"Cannot parse Firebase url" on snapshot reference access when using RTDB via firebase functions. #746

"Cannot parse Firebase url" on snapshot reference access when using RTDB via firebase functions. #746

filipesilva commented Jul 22, 2020

filipesilva commented Jul 22, 2020

Uh oh!

samtstern commented Jul 22, 2020

Uh oh!

filipesilva commented Jul 22, 2020

Uh oh!

samtstern commented Jul 22, 2020

Uh oh!

samtstern commented Jul 23, 2020 •

edited

Loading

Uh oh!

samtstern commented Jul 23, 2020

Uh oh!

samtstern commented Jul 23, 2020

Uh oh!

samtstern commented Jul 23, 2020

Uh oh!

filipesilva commented Jul 23, 2020

Uh oh!

samtstern commented Jul 24, 2020

Uh oh!

filipesilva commented Dec 30, 2020

Uh oh!

samtstern commented Dec 31, 2020

Uh oh!

filipesilva commented Jan 1, 2021

Uh oh!

filipesilva commented Apr 5, 2021

Uh oh!

samtstern commented Apr 8, 2021

Uh oh!

filipesilva commented Apr 22, 2021

Uh oh!

"Cannot parse Firebase url" on snapshot reference access when using RTDB via firebase functions. #746

"Cannot parse Firebase url" on snapshot reference access when using RTDB via firebase functions. #746

Comments

filipesilva commented Jul 22, 2020

[REQUIRED] Describe your environment

[REQUIRED] Describe the problem

Steps to reproduce:

Relevant Code:

filipesilva commented Jul 22, 2020

Uh oh!

samtstern commented Jul 22, 2020

Uh oh!

filipesilva commented Jul 22, 2020

Uh oh!

samtstern commented Jul 22, 2020

Uh oh!

samtstern commented Jul 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samtstern commented Jul 23, 2020

Uh oh!

samtstern commented Jul 23, 2020

Uh oh!

samtstern commented Jul 23, 2020

Uh oh!

filipesilva commented Jul 23, 2020

Uh oh!

samtstern commented Jul 24, 2020

Uh oh!

filipesilva commented Dec 30, 2020

Uh oh!

samtstern commented Dec 31, 2020

Uh oh!

filipesilva commented Jan 1, 2021

Uh oh!

filipesilva commented Apr 5, 2021

Uh oh!

samtstern commented Apr 8, 2021

Uh oh!

filipesilva commented Apr 22, 2021

Uh oh!

samtstern commented Jul 23, 2020 •

edited

Loading