Skip to content

BUG: DataFrame.first_valid_index() fails if there is no valid entry. #17488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 23, 2017
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v0.21.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -399,6 +399,7 @@ Indexing
- Bug in ``CategoricalIndex`` reindexing in which specified indices containing duplicates were not being respected (:issue:`17323`)
- Bug in intersection of ``RangeIndex`` with negative step (:issue:`17296`)
- Bug in ``IntervalIndex`` where performing a scalar lookup fails for included right endpoints of non-overlapping monotonic decreasing indexes (:issue:`16417`, :issue:`17271`)
- Bug in ``DataFrame.first_valid_index`` and ``DataFrame.last_valid_index`` when no valid entry (:issue:`17400`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can do

:meth:`DataFrame.first_valid_index` and so on


I/O
^^^
Expand Down
82 changes: 78 additions & 4 deletions pandas/core/frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -4069,23 +4069,97 @@ def update(self, other, join='left', overwrite=True, filter_func=None,
# ----------------------------------------------------------------------
# Misc methods

def _get_valid_indices(self):
is_valid = self.count(1) > 0
return self.index[is_valid]

def first_valid_index(self):
"""
Return label for first non-NA/null value
Return index for first non-NA/null value.
If all elements are non-NA/null, returns None.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add these 2 last sentences to a Notes section

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a Returns section (it returns a scalar)

Also returns None for empty DataFrame.

Examples
--------

When no null value in a DataFrame, returns first index.

>>> df = DataFrame({'A': [1, 2, 3], 'B': ['a', 'b', 'f']})
>>> df.first_valid_index()
0

When all elements in first row are null, returns second index.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally would like to move these doc-strings to core/generic.py and use _shared_docs. can you do that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreback
Thank you for your advice.
Modified the doc-strings with _shared_docs.
Not sure to the manner of pandas doc-strings, please check it.
If it's ok, I will squash the 2nd commit to 1st one.

Note: I have no idea to share the doc-strings between Series and DataFrames with Examples, so I removed the examples.

>>> df = DataFrame({'A': [None, 2, 3], 'B': [None, 'b', 'f']})
>>> df.first_valid_index()
1

When only part of elements in first row are null, recognized as valid.

>>> df = DataFrame({'A': [1, 2, 3], 'B': [None, 'b', 'f']})
>>> df.first_valid_index()
0

When all elements in a dataframe are null, returns None

>>> df = DataFrame({'A': [None, None, None], 'B': [None, None, None]})
>>> df.first_valid_index()

Returns None for empty DataFrame

>>> df = DataFrame()
>>> df.first_valid_index()

"""
if len(self) == 0:
return None

return self.index[self.count(1) > 0][0]
valid_indices = self._get_valid_indices()
return valid_indices[0] if len(valid_indices) else None

def last_valid_index(self):
"""
Return label for last non-NA/null value
Return index for last non-NA/null value.
If all elements are non-NA/null, returns None.
Also returns None for empty DataFrame.

Examples
--------

When no null value in a DataFrame, returns last index.

>>> df = DataFrame({'A': [1, 2, 3], 'B': ['a', 'b', 'f']})
>>> df.last_valid_index()
2

When all elements in last row are null, returns second from last index.

>>> df = DataFrame({'A': [1, 2, None], 'B': ['a', 'b', None]})
>>> df.last_valid_index()
1

When only part of elements in last row are null, recognized as valid.

>>> df = DataFrame({'A': [1, 2, 3], 'B': ['a', 'b', None]})
>>> df.last_valid_index()
2

When all elements in a dataframe are null, returns None

>>> df = DataFrame({'A': [None, None, None], 'B': [None, None, None]})
>>> df.last_valid_index()

Returns None for empty DataFrame

>>> df = DataFrame()
>>> df.last_valid_index()

"""
if len(self) == 0:
return None

return self.index[self.count(1) > 0][-1]
valid_indices = self._get_valid_indices()
return valid_indices[-1] if len(valid_indices) else None

# ----------------------------------------------------------------------
# Data reshaping
Expand Down
62 changes: 60 additions & 2 deletions pandas/core/series.py
Original file line number Diff line number Diff line change
Expand Up @@ -2840,7 +2840,36 @@ def dropna(self, axis=0, inplace=False, **kwargs):

def first_valid_index(self):
"""
Return label for first non-NA/null value
Return index for first non-NA/null value.
If all elements are non-NA/null, returns None.
Also returns None for empty Series.

Examples
--------

When no null value in a Series, returns first index.

>>> s = Series([3, 4, 5])
>>> s.first_valid_index()
0

When first element is null, returns second index.

>>> s = Series([None, 4, 5])
>>> s.first_valid_index()
1

When all elements are null, returns None

>>> s = Series([None, None, None])
>>> s.first_valid_index()


Returns None for empty Series

>>> s = Series()
>>> s.first_valid_index()

"""
if len(self) == 0:
return None
Expand All @@ -2854,7 +2883,36 @@ def first_valid_index(self):

def last_valid_index(self):
"""
Return label for last non-NA/null value
Return index for last non-NA/null value
If all elements are non-NA/null, returns None.
Also returns None for empty Series.

Examples
--------

When no null value in a Series, returns last index.

>>> s = Series([3, 4, 5])
>>> s.last_valid_index()
2

When last element is null, returns second from the last index.

>>> s = Series([3, 4, None])
>>> s.last_valid_index()
1

When all elements are null, returns None

>>> s = Series([None, None, None])
>>> s.last_valid_index()


Returns None for empty Series

>>> s = Series()
>>> s.last_valid_index()

"""
if len(self) == 0:
return None
Expand Down
5 changes: 5 additions & 0 deletions pandas/tests/frame/test_timeseries.py
Original file line number Diff line number Diff line change
Expand Up @@ -440,6 +440,11 @@ def test_first_last_valid(self):
assert empty.last_valid_index() is None
assert empty.first_valid_index() is None

# GH17400
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a comment (no valid entries)

frame[:] = nan
assert frame.last_valid_index() is None
assert frame.first_valid_index() is None

def test_at_time_frame(self):
rng = date_range('1/1/2000', '1/5/2000', freq='5min')
ts = DataFrame(np.random.randn(len(rng), 2), index=rng)
Expand Down