iloc with boolean mask

[hayd]

Currently masking by boolean vectors it doesn't matter which syntax you use:

df[mask]
df.iloc[mask]
df.loc[mask]

are all equivalent. Should mask df.iloc[mask] mask by position? (this makes sense if mask is integer index).

This SO question.

[jreback]

normally the generated masks have the same index as to what you are doing, e.g in your example in the SO question.

I think .iloc could/should do this, does make sense. There is an alignment step for the mask in .where

[snth]

Here's an example to summarise the current behaviours:

locs = np.arange(4)
nums = 2**locs
reps = map(bin, nums)
df = pd.DataFrame({'locs':locs, 'nums':nums}, reps)
print df
for idx in [None, 'index', 'locs']:
    mask = (df.nums>2).values
    if idx:
        mask = pd.Series(mask, list(reversed(getattr(df, idx))))
    for method in ['', '.loc', '.iloc']:
        try:
            if method:
                accessor = getattr(df, method[1:])
            else:
                accessor = df
            ans = bin(accessor[mask]['nums'].sum())
        except Exception, e:
            ans = str(e)
        print "{:>5s}: df{}[mask].sum()=={}".format(idx, method, ans)

with output

        locs  nums
0b1        0     1
0b10       1     2
0b100      2     4
0b1000     3     8

 None: df[mask].sum()==0b1100
 None: df.loc[mask].sum()==0b1100
 None: df.iloc[mask].sum()==0b1100
index: df[mask].sum()==0b11
index: df.loc[mask].sum()==0b11
index: df.iloc[mask].sum()==0b11
 locs: df[mask].sum()==Unalignable boolean Series key provided
 locs: df.loc[mask].sum()==Unalignable boolean Series key provided
 locs: df.iloc[mask].sum()==Unalignable boolean Series key provided

If I'm understanding your discussion correctly then in the last line the output should also be 0b11.

[jreback]

yes...that looks right, if you happen to think of a simple example where you would actually use this pls post that

keep in mind that masks are the same length as what you are indexing, so I don't think there is ever ambiguity, but I could be wrong

[snth]

Fair enough, I can't actually think of anything. It just seems that by symmetry the behaviour should be there.

Similarly, it doesn't seem quite right that in line 6 of my example above, df.iloc[mask] is actually realigning the mask based on mask.index rather than throwing an error.

Given these observations, I would probably vote for the following behaviour:

• df[mask] should only work with .values in the order they're given and do no realignment. Perhaps for performance reasons.
• df.loc[mask] should work as it currently does and realign based on mask.index if present. If mask.index is of the wrong type it should throw an Error.
• df.iloc[mask] should realign based on mask.index if present and of integer type. Otherwise throw an error.

It's true that I haven't thought about this for more than 10 minutes though so if there are many use cases where this causes a problem then nevermind.

[hayd]

I think it's possible there could be an ambiguity, if the index is in a different order (e.g. was taken from somewhere else where it may well mean the location rather than the label).

Also, I'm given a /core/frame.py:1943: UserWarning: Boolean Series key will be reindexed to match DataFrame index. somewhere along the way messing with integer index booleans.

Correct me if I wrong, but isn't "realign based on mask.index" different from location?

[jreback]

Still thinking about this, but I think a nice feature of boolean masking is that since we don't ever have an ambiguity whether its label or position based (as the mask must be the same length as what you are indexing), then it can be used in either .loc or .iloc. I don't think restricting this is a good idea.

The basic question is do we drop the index effectively (and make it not matter) when its the right length?

e.g. should

df[mask_with_index] and df[mask_ndarray] be the same?

or the first align (currently I retract my earlier statement, I don't think there is an alignment on the index itself)

but this may depend, e.g. iloc/loc don't align, but I think [] might....

[jreback]

@hayd you make a good point, but to my knowledge that is the issue with an unlabeled index, the user has to make sure its in the right order, pandas cannot help (but the case we are talking about it could help, by making sure the index is aligned)

[snth]

@jreback With regards to the alignment, see my example above. I threw in the reversed(...) to see whether it realigns based on the labels or not. The results above show that [], .loc and .iloc all perform an alignment step based on the labels when these are in a different order. Therefore I think the fact that the length is the same is lulling you into a false sense of security.

Apologies if my example isn't clear. I thought the binary representation was a nice way of concisely summarising which items were selected or not. The bottom line is that the results differ between ndarray and pd.Series because in the pd.Series case the .index is used to do an alignment step first.

Also there's mistake in my output because it should read:

df[mask]['nums'].sum()==...

[jreback]

@snth you are right....I was eye-balling the code....as its a failry tricky path

ok....so bottom line is wether to make iloc align on the labels or position (for a boolean mask), but that itself is somewhat of an issue, the is certainly probably and likely that the index is not integer-based, so then iloc should throw an error

So from the original SO question, this should then raise?

In [1]: df = pd.DataFrame(range(5), list('ABCDE'), columns=['a'])

In [2]: mask = (df.a%2 == 0)

In [3]: mask
Out[3]: 
A     True
B    False
C     True
D    False
E     True
Name: a, dtype: bool

In [4]: df.iloc[mask]
Out[4]: 
   a
A  0
C  2
E  4

What about this?

In [5]: mask.nonzero()
Out[5]: (array([0, 2, 4]),)

In [6]: mask.nonzero()[0]
Out[6]: array([0, 2, 4])

In [7]: df.iloc[mask.nonzero()[0]]
Out[7]: 
   a
A  0
C  2
E  4

In [8]: df.iloc[Series(mask.nonzero()[0])]
Out[8]: 
   a
A  0
C  2
E  4

In [9]: Series(mask.nonzero()[0])
Out[9]: 
0    0
1    2
2    4
dtype: int64

I there is NO alignment happening, instead its using the values to actually index

[hayd]

I think iloc should throw an error if it's not integer based, it should definitely use position.

These integers needn't be in order, just like the labels needn't be in order.

Putting an array to iloc should do what it does now, but a boolean Series is a different beast:

msk = pd.Series([True, True, True, False, False])
df.iloc[msk] == df.iloc[0:3]

?

[snth]

@jreback I agree that in your first example by my reasoning In[4] should raise an Exception rather than realign based on labels within in .iloc.

I don't understand your second example. There's no boolean indexing involved there and they just seem to be examples of .iloc. Is that what's happening in the source code?

[jreback]

@snth I was trying to have it index with a Series that had a different index, (that is essentially what boolean masking does)

[hayd]

Ok, if you had the one above, there is an easy workaround (with arrays):

msk = pd.Series([True, True, True, False, False])
df.iloc[msk.values]  # instead of df.iloc[msk]

But if your index was not in order:

msk = pd.Series([True, True, True, False, False], index=[1, 2, 3, 4, 0])
df.iloc[msk] == df[1:4]

?

[snth]

@hayd In terms of workarounds you could probably do

df.iloc[msk.index][msk.values]

[jreback]

Nothing else broke...I'll put up the PR in a minute....pls try out

In [3]: df = DataFrame(range(5), list('ABCDE'), columns=['a'])

In [4]: mask = (df.a%2 == 0)

In [5]: df.iloc[mask]
ValueError: iLocation based boolean indexing can only have an integer index [string] was inferred