ENH: remove restrictions to numexpr to allow `where` etc.

[jonas-eschle]

Is your feature request related to a problem?

the evaluation of a query is currently limited to the list _mathops while numexpr would support more, most notably a where (that would also solve other issues simple).

I do not see any reason for this restriction. In fact, simply adding the where runs (at least for my use case). Why is this restriction in place? Why can't we enlarge it/directly pass it through to numexpr?

Describe the solution you'd like

Allow the full operator set that numexpr supports in the pd.eval

API breaking implications

Nothing

Alternatives

Using .where is an option if you can access the dataframe directly (although suboptimal). However, if your selection of data is based on passing a selection string around instead of the df (several reasons for this), the latter is not feasible.

The following doesn't work:

import pandas as pd

data = {'a': [1, 2, 3]}                                                 

df = pd.DataFrame({'a': [1, 2, 3]})
df.eval('where(a>2, 42, 0)')

whereas in numexpr it does

numexpr.evaluate('where(a>2, 42, 0)', local_dict=data)

we expect this to return [0, 0, 42]

[TomAugspurger]

Can you add a minimal example? http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

[jonas-eschle]

Well, yeah. I've updated it. But it does not make so much sense I think to provide an example for eval/query? Or what exactly do you think is unclear?

Again, in short: eval/query use numexpr to forward a string for evaluation but (to my view) unnecessarily limit the allowed expressions to a subset of what numexpr supports.

Let me know if things are confusing and thanks a lot for taking a look at it!

[TomAugspurger]

Thanks. And can you add the expected output?

Do you know if we have other expressions that would be supported by only one engine (numexpr in this case)?

[Liam3851]

I was about to report this issue but realized [edit: I would have created] a dupe. numexpr supports the following functions missing from the definitions in pandas.core.computation.ops:

• where
• tan (oddly, we support sin, cos, tanh, arctan, and arctan2, but not tan)
• conj, real, imag, complex (though I'm not sure the extent to which pandas supports complex numbers in general so this might be fine)
• contains (for strings)

Expectation is that

import pandas as pd

data = {'a': [1, 2, 3]}                                                 

df = pd.DataFrame({'a': [1, 2, 3]})
df.eval('where(a>2, 42, 0)')

should return the same as

pd.DataFrame({'a':[0, 0, 42]})

[Liam3851]

My apologies @mayou36 can you reopen? I meant that I would have created a dupe. This is not a dupe, I think it's a legit issue that pandas doesn't support where or tan when numexpr does.