subset keyword argument will include last column if incorrect keys are given

[arijun]

If you give a non-existent key to the subset argument it will default to the last column. For example:

In [3]: d = pd.DataFrame({'a':[1],'b':[2],'c':[np.nan]})
In [4]: len(d.dropna(subset=['x']))
Out[4]: 0

It seems the problem is in the line where dropna takes the subset:

agg_obj = self.take(ax.get_indexer_for(subset),axis=agg_axis)

Here, ax.get_indexer_for(subset) will return -1 as a sentinel value for any keys that were not found in subset, and self.take interprets the -1 as a request for the last column.

[jreback]

yep would call that a buggy....pull-request welcome!

[mcjcode]

Would it be best for dropna to raise a KeyError exception, like d['x'] would in the example above? or to return all of the rows, the thinking being that column 'x' contains no nan's whatsoever (in fact it contains no values whatsoever, since 'x' doesn't even exist).

[jreback]

no this is very easy to fix
just filter the -1 from the indexer before passing to take

[mcjcode]

Easy enough, but does this mean we don't support location based indices for the subset argument? I mean, what if someone actually wrote d.dropna(subset=[-1]) intending to drop the rows where the last column is nan?

[jreback]

only support labels - you have the whole complication the of having to have 2 methods to disambiguate

and get indexer converts labels to locations

[jreback]

use your example (with subset = [-1]) as a test
that will KeyError I think

[mcjcode]

Yes, I like just supporting column labels here. I do note though, that when I do this

import pandas as pd
import numpy as np
print(pd.version.version)
d = pd.DataFrame({'a':[1,2],'b':[np.nan,3],'c':[4,np.nan]})
print(d.dropna(subset=[-1]))

I get this (i.e. it works as if it was dropping rows where the last were nan)

0.14.1-429-g0eb9023
   a   b  c
0  1 NaN  4

[jreback]

the indexer returns -1 (meaning it can't find the key) and take returns the last value

so you will need to filter out -1 or maybe if any -1 should raise
(if u have 1 good subset then should it work if another is not found)

we usually do this (eg in loc)

though I think in this case prob should just a raise of any columns are not found
as prob a user error