Skip to content

Support hard-masked numpy arrays #24581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jan 4, 2019
Merged

Support hard-masked numpy arrays #24581

merged 3 commits into from
Jan 4, 2019

Conversation

karldw
Copy link
Contributor

@karldw karldw commented Jan 3, 2019 •
edited
Loading

For the whatsnew entry, is this reasonable?

- :func:`DataFrame` and :func:`Series` now properly handle 
numpy masked arrays with hardened masks. Previously, constructing 
a DataFrame or Series from a masked array with a hard mask would 
create a pandas object containing the underlying value, rather than the
expected NaN. (:issue:`24574`)

@pep8speaks
Copy link

pep8speaks commented Jan 3, 2019 •
edited
Loading

Hello @karldw! Thanks for updating the PR.

Line 767:13: E123 closing bracket does not match indentation of opening bracket's line
Line 778:13: E123 closing bracket does not match indentation of opening bracket's line

Comment last updated on January 03, 2019 at 15:45 Hours UTC

jreback
jreback requested changes Jan 3, 2019
View reviewed changes
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whatsnew looks fine

pandas/tests/frame/test_constructors.py Outdated
@@ -757,6 +757,20 @@ def test_constructor_maskedarray_nonfloat(self):
assert frame['A'][1] is True
assert frame['C'][2] is False

# Check hardened masks
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you make a new tests & add the issue number as a comment

pandas/tests/frame/test_constructors.py Outdated
@@ -757,6 +757,20 @@ def test_constructor_maskedarray_nonfloat(self):
assert frame['A'][1] is True
assert frame['C'][2] is False

# Check hardened masks
mat_hard = ma.masked_all((2, 3), dtype=float).harden_mask()
frame = DataFrame(mat_hard, columns=['A', 'B', 'C'], index=[1, 2])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use the format

result = 
expected = pd.DataFrame(....)
tm.assert_frame_equal(result, expected)

construct the expected exactly

pandas/tests/series/test_constructors.py Outdated
@@ -386,6 +386,11 @@ def test_constructor_maskedarray(self):
expected = Series([nan, nan, nan])
assert_series_equal(result, expected)

data_hard = ma.copy(data).harden_mask()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a separate test

@jreback jreback added Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Jan 3, 2019
@codecov
Copy link

codecov bot commented Jan 3, 2019

Codecov Report

Merging #24581 into master will increase coverage by <.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #24581      +/-   ##
==========================================
+ Coverage   92.36%   92.36%   +<.01%     
==========================================
  Files         166      166              
  Lines       52494    52496       +2     
==========================================
+ Hits        48485    48487       +2     
  Misses       4009     4009
Flag Coverage Δ
#multiple 90.79% <100%> (ø) ⬆️
#single 43.03% <0%> (-0.01%) ⬇️
Impacted Files Coverage Δ
pandas/core/frame.py 96.92% <100%> (ø) ⬆️
pandas/core/internals/construction.py 96.67% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b91140...b20389d. Read the comment docs.

@codecov
Copy link

codecov bot commented Jan 3, 2019 •
edited
Loading

Codecov Report

Merging #24581 into master will increase coverage by 0.02%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #24581      +/-   ##
==========================================
+ Coverage   92.36%   92.38%   +0.02%     
==========================================
  Files         166      166              
  Lines       52494    52490       -4     
==========================================
+ Hits        48485    48493       +8     
+ Misses       4009     3997      -12
Flag Coverage Δ
#multiple 90.81% <100%> (+0.02%) ⬆️
#single 43.05% <0%> (+0.01%) ⬆️
Impacted Files Coverage Δ
pandas/core/frame.py 96.92% <100%> (ø) ⬆️
pandas/core/internals/construction.py 96.67% <100%> (ø) ⬆️
pandas/core/nanops.py 94.54% <0%> (-0.37%) ⬇️
pandas/core/indexes/accessors.py 91% <0%> (ø) ⬆️
pandas/core/indexes/timedeltas.py 90.22% <0%> (ø) ⬆️
pandas/core/arrays/datetimelike.py 97.85% <0%> (ø) ⬆️
pandas/core/generic.py 96.62% <0%> (ø) ⬆️
pandas/core/arrays/datetimes.py 98.01% <0%> (ø) ⬆️
pandas/core/arrays/period.py 98.53% <0%> (+0.01%) ⬆️
pandas/core/internals/managers.py 95.96% <0%> (+0.02%) ⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b91140...ad01a21. Read the comment docs.

@jreback
Copy link
Contributor

jreback commented Jan 3, 2019

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=6404 some lint issues as well

@jreback jreback added this to the 0.24.0 milestone Jan 3, 2019
@jreback
Copy link
Contributor

jreback commented Jan 3, 2019 •
edited
Loading

lgtm. ping on green.

@jreback
Copy link
Contributor

jreback commented Jan 3, 2019

some lint issues

@jreback jreback merged commit 6a0e276 into pandas-dev:master Jan 4, 2019
@jreback
Copy link
Contributor

jreback commented Jan 4, 2019

thanks @karldw

@karldw
Copy link
Contributor Author

karldw commented Jan 4, 2019

Thank you!

@karldw karldw deleted the support-hard-masked-arrays branch January 4, 2019 00:39
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@karldw @Pingviinituutti
Support hard-masked numpy arrays (pandas-dev#24581)
cba304d
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@karldw @Pingviinituutti
Support hard-masked numpy arrays (pandas-dev#24581)
3f8f857
@arw2019 arw2019 mentioned this pull request Nov 21, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jreback jreback jreback approved these changes

Assignees
No one assigned
Labels
Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Milestone
0.24.0
Development

Successfully merging this pull request may close these issues.

Masked arrays with hard masks don't read as missing
3 participants
@karldw @pep8speaks @jreback