ENH: allow in-line expression assignment with df.eval

[jreback]

This was a relatively easy extension of eval to allow in-line creation/assignment.

Allows one to basically use formulas to do things (pandas conquers excel!!!!)

• docs
• tests non-frame

In [11]: df = DataFrame(dict(a = range(5), b = range(5,10)))

In [12]: df
Out[12]: 
   a  b
0  0  5
1  1  6
2  2  7
3  3  8
4  4  9

In [13]: df.eval('c=a+b')

In [14]: df.eval('d=a+b+c')

In [15]: df.eval('a=1')

In [16]: df
Out[16]: 
   a  b   c   d
0  1  5   5  10
1  1  6   7  14
2  1  7   9  18
3  1  8  11  22
4  1  9  13  26

You can do this (this could maybe have a bit better syntax though)

In [31]: df = DataFrame(dict(a = range(5), b = range(5,10)))

In [32]: formulas = Series(['c=a+b','d=a*b'],index=['a','b'])

In [33]: df.apply(lambda x: df.eval(formulas[x.name]))
Out[33]: 
a    None
b    None
dtype: object

In [34]: df
Out[34]: 
   a  b   c   d
0  0  5   5   0
1  1  6   7   6
2  2  7   9  14
3  3  8  11  24
4  4  9  13  36

[jreback]

@cpcloud what do you think?

[cpcloud]

Any reason not to put this in a visit_Assign method?

[jreback]

I didn't try that because we have one which basically switches the Assign to a Compare and I think its needed for that? (e.g. if its on the rhs of an expression). I also wanted to ensure only a single assignment, e.g. a = b = c is invalid

[cpcloud]

I think that's only for PyTables, though. Not meaning to be nitpicky.

[cpcloud]

What happens with overlapping locals?

[jreback]

you mean like

a = 5
df.eval('a = b + c')

[cpcloud]

yep

[jreback]

I actually don't think its a problem because in this case the local doesn't matter

my question to use is, I am using env.resolvers[0] to figure out whom to do the assignment to, (its there when you call df.eval, but not when you call pd.eval). is that the 'right' way?

[jreback]

I think this works as expected (put tests in for this)

In [1]: df = DataFrame(np.random.randn(5, 2), columns=list('ab'))

In [2]: a = 1

In [3]: df.eval('a=1+b')

In [4]: df
Out[4]: 
          a         b
0 -0.641769 -1.641769
1  1.342992  0.342992
2  1.688316  0.688316
3  0.321509 -0.678491
4  1.652393  0.652393

In [5]: df.eval('a=a+b')

NameResolutionError: resolvers and locals overlap on names ['a']

In [6]: df.eval('a=@a+b')

In [7]: df
Out[7]: 
          a         b
0 -0.641769 -1.641769
1  1.342992  0.342992
2  1.688316  0.688316
3  0.321509 -0.678491
4  1.652393  0.652393

[cpcloud]

If you pass in resolvers explicitly (I actually think this should be disallowed and added if people want it), then resolvers[0] will not work in some cases. One thing that it sort of breaks is that variable look ups occur in the order of the resolvers, so shouldn't assignment be attempted until it either succeeds or exhausts all of the resolvers?

[cpcloud]

I wonder if an assign(name, value) method on Scope would be useful here?

[cpcloud]

I suppose __setitem__ could be enlisted to do that.

[cpcloud]

FWIW I would like to eventually gut Scope and use ChainMap (in Python 3 collections) that is essentially Scope, but much less hacky and basically behaves like dict with a few bells a whistles to control the order of lookup. I'll do that for 0.14.

[cpcloud]

At some point I should also add vars(pd.compat.builtin) to locals so that builtin constants are there.

[cpcloud]

ChainMap is pretty cool.

[jreback]

ok...now use visit_Assign pretty easy, just had to figure out that _visitor actually has the data..

[jreback]

on the resolvers.

I think that I need to have an explicit way of having the target recorded by df.eval, maybe i'll just add a target ky to Scope?

[jreback]

@cpcloud all updated now....take a look when you have a chance, added a short note in the docs too.

[cpcloud]

Should this be switched with test right below? This is not valid Python syntax while the one below is valid Python syntax, but not valid eval syntax.

[jreback]

I'll check but iirc it fails in the assign block with a left hand side that is a list with a length of 2

[cpcloud]

strange ... that should simply throw the usual syntax error and not do any parsing outside of Python

[jreback]

you are right 'd c = a + b' raises a SyntaxError in fix_missing_locations....I think I meant the a = b = c, which ends up having multiple assignment nodes (which I raise a Syntax Error); though in theory in the future you could handle

[cpcloud]

can u change the test to make sure the multiple assignment fails? again not meaning to be a troll....just want to make sure the correct error is checked

[jreback]

otherwise look ok @cpcloud ?

[cpcloud]

yep looks okay

[jreback]

@cpcloud did you mean something additional besides this?

[cpcloud]

oh nope! sorry didn't see that 👍