BUG: Allow assignment by indexing with duplicate column names

[gfyoung]

closes #12344

in which assignment to columns in DataFrame with duplicate column names caused all columns with the same name to be reassigned. The bug was located here.

When you try to index into the DataFrame using .iloc, pandas will find the corresponding column name (or key) first before setting that key with the given value. Unfortunately, since all of your columns have the same key name, pandas ends up choosing all of the columns corresponding to that name.

This PR introduces a _set_item_by_index method for DataFrame objects that allows you to bypass that issue by using the indices of the columns to set the columns themselves whenever there are duplicates involved.

[shoyer]

Could we use a more directly method for constructing this test case? I found this setup very confusing (using index to do implicit broadcasting across rows).

For the record, this is the resulting dataframe:

     Same   Same  Same
0     NaN    NaN     1
1     NaN    NaN     1
2     NaN    NaN     1

[gfyoung]

Done. Would you mind also cancelling my first Travis build then (it hasn't started yet but fixing this test added a new build)?

[TomAugspurger]

I canceled https://travis-ci.org/pydata/pandas/builds/112754541, hopefully that was the correct one.

[gfyoung]

@TomAugspurger : Yep, that's the right one! The one on top should always be the latest build. Thanks!

[jreback]

you are adding WAY too much machinery. This is a pretty simple fix, though I don't have time at the moment to look.

[jreback]

this should be handled in internals.

[gfyoung]

@jreback : yes, it is currently being handled there, but I probably took a slightly more circuitous root than needed to get there. I'll see what I can simplify.

[gfyoung]

Simplified the internal machinery to route directly to internals.py, and Travis is happy. Should be good to merge if there is nothing else.

[jreback]

still needs some work. you are adding function. use existing functionaility. This just makes it much harder on future readers.

[gfyoung]

@jreback : I don't quite see how the existing functionality can properly handle duplicate columns in this context IMOH. The new functionality I have added is just five lines of code here. The rest of it is refactoring. Besides essentially taking the internals of the function I wrote and placing them in the location where the function is called, I'm not sure what else you mean by "using existing functionality"

[gfyoung]

If some could cancel this build #18021 (it's an old build), that would be great. Thanks!

[jreback]

this still needs work. too much specific if/thening

[gfyoung]

What do you mean "too much specific if/thening"?

[jreback]

this is too much specific if thening. you are trying to catch a very specific case and not solving it in a more general way.

[jreback]

diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
index f0f5507..0313bb2 100644
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -541,7 +541,7 @@ class _NDFrameIndexer(object):
                 if (len(indexer) > info_axis and
                         is_integer(indexer[info_axis]) and
                         all(is_null_slice(idx) for i, idx in enumerate(indexer)
-                            if i != info_axis)):
+                            if i != info_axis) and item_labels.is_unique ):
                     self.obj[item_labels[indexer[info_axis]]] = value
                     return

This is a more idiomatic way of creating the data

df = pd.DataFrame(np.arange(9).reshape(3,3).T)
df.columns = list('AAA')

df.iloc[:, 0] = df.iloc[:, 0].fillna(df.iloc[:, 1])

also need to test mixed setting (e.g. use int/float/string block)

The .fillna doesn't matter here at all.

[gfyoung]

@jreback : Alright, I definitely did not understand what you meant by "using existing functionality". Thanks for the patch! I think I was too focused on resolving the issue within that block itself and didn't quite pay attention to the other conditionals listed underneath it.

[gfyoung]

@jreback : Tests are passing with the patch you provided. Should be good to merge now.

[jreback]

thanks