Skip to content

BUG: null group spills into final group when grouping on a categorical #10014

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 29, 2015
Merged

BUG: null group spills into final group when grouping on a categorical #10014

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
238e30e
BUG: null group spills into final group when grouping on a categorical
evanpw Apr 28, 2015
d194c99
Fix missing index name in test
Apr 29, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v0.16.1.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ Bug Fixes
- Bug in csv parser causing lines with initial whitespace plus one non-space character to be skipped. (:issue:`9710`)



- Bug causing elements with a null group to spill into the final group when grouping by a ``Categorical`` (:issue:`9603`)


- Bug in invalid attribute access on a ``TimedeltaIndex`` incorrectly raised ``ValueError`` instead of ``AttributeError`` (:issue:`9680`)
Expand Down
20 changes: 12 additions & 8 deletions pandas/lib.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1306,9 +1306,10 @@ def duplicated(ndarray[object] values, take_last=False):

def generate_slices(ndarray[int64_t] labels, Py_ssize_t ngroups):
cdef:
Py_ssize_t i, group_size, n, lab, start
Py_ssize_t i, group_size, n, start
int64_t lab
object slobj
ndarray[int64_t] starts
ndarray[int64_t] starts, ends

n = len(labels)

Expand All @@ -1318,13 +1319,16 @@ def generate_slices(ndarray[int64_t] labels, Py_ssize_t ngroups):
start = 0
group_size = 0
for i in range(n):
group_size += 1
lab = labels[i]
if i == n - 1 or lab != labels[i + 1]:
starts[lab] = start
ends[lab] = start + group_size
start += group_size
group_size = 0
if lab < 0:
start += 1
else:
group_size += 1
if i == n - 1 or lab != labels[i + 1]:
starts[lab] = start
ends[lab] = start + group_size
start += group_size
group_size = 0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The null group is represented with a label of -1, so it was getting put in the final spot of starts / ends. In the usual case, it would then get overwritten by the last group, but when grouping on a categorical, the last group may be empty.

return starts, ends

Expand Down
7 changes: 7 additions & 0 deletions pandas/tests/test_categorical.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1841,6 +1841,13 @@ def f(x):
tm.assert_frame_equal(df.groupby(c).transform(sum), df[['a']])
tm.assert_frame_equal(df.groupby(c).transform(lambda xs: np.sum(xs)), df[['a']])

# GH 9603
df = pd.DataFrame({'a': [1, 0, 0, 0]})
c = pd.cut(df.a, [0, 1, 2, 3, 4])
result = df.groupby(c).apply(len)
expected = pd.Series([1, 0, 0, 0], index=c.values.categories)
tm.assert_series_equal(result, expected)

def test_pivot_table(self):

raw_cat1 = Categorical(["a","a","b","b"], categories=["a","b","z"], ordered=True)
Expand Down