Query by value with special characters / numbers fails

[mark-hoffmann]

[REQUIRED] Step 2: Describe your environment

• Firebase SDK version: 2.12.0
• Firebase Product: database

[REQUIRED] Step 3: Describe the problem

Querying breaks on certain strings. Setting up a query such as:

db.reference("ParentNode").order_by_child("key").equal_to("value_to_match")

Works when your values are pure strings such as "Todd", but fail to return results when your strings are like in my case of a phone number following the following pattern "+15555555".

This behavior is working in the javascript sdk, but does not in the python version.

[hiranya911]

I was able to reproduce the problem. Looks like the REST API doesn't handle filters with special characters well. I'm using the following test:

db.reference('aParent').order_by_child('key').equal_to('+123456789').get()

This is the request made by the SDK:

GET /aParent.json?equalTo=%22+123456789%22&orderBy=%22key%22 HTTP/1.1
Host: xxxxxxxx.firebaseio.com
User-Agent: Firebase/HTTP/2.12.0/3.6/AdminPython
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
authorization: Bearer xxxxxx

This doesn't return any results. @rockwotj any idea what might be happening? Any other value in the filter (e.g. equal_to('foo')) works fine.

[rockwotj]

Same issue as #165

I really want to get this resolved, but we need to make sure the fix doesn't break people in production.

[hiranya911]

Reproducible with curl too:

Success case:

curl -v -H "Authorization: Bearer $TOKEN" "https://xxxx.firebaseio.com/aParent.json?equalTo=%22foo%22&orderBy=%22key%22"

< HTTP/1.1 200 OK
< Server: nginx
< Date: Fri, 10 Aug 2018 21:04:32 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 24
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Cache-Control: no-cache
< Strict-Transport-Security: max-age=31556926; includeSubDomains; preload
< 
{"child1":{"key":"foo"}}

Failure case:

curl -v -H "Authorization: Bearer $TOKEN" "https://xxxx.firebaseio.com/aParent.json?equalTo=%22+123456789%22&orderBy=%22key%22"

< HTTP/1.1 200 OK
< Server: nginx
< Date: Fri, 10 Aug 2018 21:07:56 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 2
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Cache-Control: no-cache
< Strict-Transport-Security: max-age=31556926; includeSubDomains; preload
< 
{}

[mark-hoffmann]

For others who stumble onto this issue in the meantime before a fix is implemented here. Pyrebase (https://github.com/thisbejim/Pyrebase) has fit all of my requirements very nicely including not having this problem and has only minor syntax differences.

The official sdk here is probably a better idea once all the small kinks have been worked out, but this is a nice solution in the interim.

[hiranya911]

Pyrebase works around the problem by double encoding the special characters:

Admin SDK: GET /aParent.json?equalTo=%22+1234567890%22&orderBy=%22key%22
Pyrebase: GET /aParent.json?orderBy=%2522key%2522&equalTo=%2522%252B1234567890%2522

@rockwotj is this the right fix? Looks like, if we fix the issue in the back-end server, this can potentially break.

[hoonk]

I've implemented a workaround for this problem with phone number entries like '+16505555555'.

For main db.py, add these imports:

from string import replace
from urllib import urlencode

In the Query class, I added:

    @property
    def _queryurl(self):
        if self._params:
            return Query._escapeQueryChars(urlencode(self._params))
        else:
            return None

    @staticmethod
    def _escapeQueryChars(text):
        text = replace(text, '%22', '"')
        text = replace(text, '%27', "'")
        text = replace(text, '%24', '$')
        text = replace(text, '%2B', '\u002B')
        return text

For get(), I made the following edits:

    def get(self):
        """Executes this Query and returns the results.

        The results will be returned as a sorted list or an OrderedDict.

        Returns:
          object: Decoded JSON result of the Query.

        Raises:
          ApiCallError: If an error occurs while communicating with the remote database server.
        """
        url = self._pathurl
        queryurl = self._queryurl
        if queryurl is not None:
            url = "%s?%s" % (url, queryurl)

        result = self._client.body('get', url=url)
        if isinstance(result, (dict, list)) and self._order_by != '$priority':
            return _Sorter(result, self._order_by).get()
        return result

[rockwotj]

So yes the bug is that query parameters are url decoded twice. We really want to fix this, but having customers double encoding requests make it impossible to fix this correctly.

We could add a special header to the server to not URL Decode twice as a quick first hack. We want to analyze and see who depends on this behavior and reach out to them to fix their usage so the server can do the "right" thing.

[rockwotj]

If you include the header X-Firebase-Decoding: 1 we will do correct URL Decoding. Once we've worked with our customers and fixed this issue, you won't need to specify this header. But we can't break users who are relying on the broken behavior.

[rockwotj]

This should be fixed in #206

[hiranya911]

Thanks @rockwotj