Query on existing field and value fails with AttributeError: 'numpy.bool_' object has no attribute 'empty'

[michaelhyatt]

Trying to query for an existing value in "winlogbeat-*", getting AttributeError

/usr/local/lib/python3.6/dist-packages/eland/query.py in exists(self, field, must)
     39         """
     40         if must:
---> 41             if self._query.empty():
     42                 self._query = NotNull(field)
     43             else:

AttributeError: 'numpy.bool_' object has no attribute 'empty'

Code:

import eland as ed
import numpy as np
import pandas as pd

from elasticsearch import Elasticsearch

es = Elasticsearch(cloud_id="XYZ", http_auth=('user', 'pwd'))
ed_df = ed.DataFrame(client=es, index_pattern="winlogbeat-*", columns=["@timestamp", "event.code", "user.name"])
ed_df.query('"user.name" == "ADMIN"')

[stevedodson]

Reproducible on test data:

    def test_issue_146(self):
        ed_ecommerce = self.ed_ecommerce()[["total_quantity", "geoip.country_iso_code"]]

        print(ed_ecommerce.query('"geoip.country_iso_code" == "EG"'))

Testing started at 12:50 ...
/Users/steve/anaconda3/envs/eland/bin/python /Applications/PyCharm.app/Contents/helpers/pycharm/_jb_pytest_runner.py --target test_query_pytest.py::TestDataFrameQuery.test_issue_146
Launching pytest with arguments test_query_pytest.py::TestDataFrameQuery::test_issue_146 in /Users/steve/PycharmProjects/eland/eland/tests/dataframe

============================= test session starts ==============================
platform darwin -- Python 3.7.5, pytest-5.3.0, py-1.8.0, pluggy-0.13.0 -- /Users/steve/anaconda3/envs/eland/bin/python
cachedir: .pytest_cache
rootdir: /Users/steve/PycharmProjects/eland
plugins: nbval-0.9.3
collecting ... collected 1 item

test_query_pytest.py::TestDataFrameQuery::test_issue_146 FAILED          [100%]
eland/tests/dataframe/test_query_pytest.py:131 (TestDataFrameQuery.test_issue_146)
self = <eland.tests.dataframe.test_query_pytest.TestDataFrameQuery object at 0x115fe6610>

    def test_issue_146(self):
        ed_ecommerce = self.ed_ecommerce()[["total_quantity", "geoip.country_iso_code"]]
    
>       print(ed_ecommerce.query('"geoip.country_iso_code" == "EG"'))

test_query_pytest.py:135: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../dataframe.py:417: in __repr__
    if len(self) > max_rows:
../../ndframe.py:148: in __len__
    return len(self.index)
../../index.py:63: in __len__
    return self._query_compiler._index_count()
../../query_compiler.py:347: in _index_count
    return self._operations.index_count(self, self.index.index_field)
../../operations.py:666: in index_count
    body.exists(field, must=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <[AttributeError("'bool' object has no attribute 'empty'") raised in repr()] Query object at 0x115fe24d0>
field = '_id', must = True

    def exists(self, field, must=True):
        """
        Add exists query
        https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html
        """
        if must:
>           if self._query.empty():
E           AttributeError: 'bool' object has no attribute 'empty'

../../query.py:41: AttributeError

Assertion failed

Assertion failed

Assertion failed

Assertion failed

Assertion failed

Assertion failed

Assertion failed

Assertion failed


=================================== FAILURES ===================================
______________________ TestDataFrameQuery.test_issue_146 _______________________

self = <eland.tests.dataframe.test_query_pytest.TestDataFrameQuery object at 0x115fe6610>

    def test_issue_146(self):
        ed_ecommerce = self.ed_ecommerce()[["total_quantity", "geoip.country_iso_code"]]
    
>       print(ed_ecommerce.query('"geoip.country_iso_code" == "EG"'))

test_query_pytest.py:135: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../dataframe.py:417: in __repr__
    if len(self) > max_rows:
../../ndframe.py:148: in __len__
    return len(self.index)
../../index.py:63: in __len__
    return self._query_compiler._index_count()
../../query_compiler.py:347: in _index_count
    return self._operations.index_count(self, self.index.index_field)
../../operations.py:666: in index_count
    body.exists(field, must=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <[AttributeError("'bool' object has no attribute 'empty'") raised in repr()] Query object at 0x115fe24d0>
field = '_id', must = True

    def exists(self, field, must=True):
        """
        Add exists query
        https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html
        """
        if must:
>           if self._query.empty():
E           AttributeError: 'bool' object has no attribute 'empty'

../../query.py:41: AttributeError
============================== 1 failed in 1.74s ===============================

Process finished with exit code 0

Assertion failed

Assertion failed

Assertion failed

Assertion failed

[stevedodson]

pandas.DataFrame.query doesn't support '"user.name" == "ADMIN"' in 0.25.3 (pandas-dev/pandas#27017). Therefore, we don't support it.

This may be fixed in pandas 1.1.0, and we will then support it.

To work around for now, use boolean filters:

ed_df[ed_df['user.name'] == "ADMIN"]

[sethmlarson]

Even in pandas v1 I'm seeing an issue on the pandas side:

df = ed.DataFrame(es, 'flights')
pd_df = df._to_pandas()
pd_df.query('"Carrier" == "JetBeats"')  # Raises a similar error "KeyError: False"

Are you able to reproduce the same issue in just pandas?

[sethmlarson]

Closing this as not an issue. Let me know if this is still an issue and I'll reopen.