Skip to content

BUG: hash_pandas_object fails on array containing tuple #28969 #30508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Dec 31, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions pandas/core/util/hashing.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,11 +85,12 @@ def hash_pandas_object(
if isinstance(obj, ABCMultiIndex):
return Series(hash_tuples(obj, encoding, hash_key), dtype="uint64", copy=False)

if isinstance(obj, ABCIndexClass):
elif isinstance(obj, ABCIndexClass):
h = hash_array(obj.values, encoding, hash_key, categorize).astype(
"uint64", copy=False
)
h = Series(h, index=obj, dtype="uint64", copy=False)

elif isinstance(obj, ABCSeries):
h = hash_array(obj.values, encoding, hash_key, categorize).astype(
"uint64", copy=False
Expand Down Expand Up @@ -306,8 +307,15 @@ def hash_array(
vals = hashing.hash_object_array(vals, hash_key, encoding)
except TypeError:
# we have mixed types
try:
str_vals = vals.astype(str)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would handle this in hash_object_array as a case

except ValueError:
# GH#28969 if we contain tuples, astype fails here,
# apparently related to github.com/numpy/numpy/issues/9441
str_vals = np.array([str(x) for x in vals], dtype=object)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm not sure about this - wouldn't this really make everything hashable at this point, including non-hashable items?

>>> vals = np.array([("a",), ["c", "d"]])
>>> str_vals = np.array([str(x) for x in vals], dtype=object)
array(["('a',)", "['c', 'd']"], dtype=object)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point


vals = hashing.hash_object_array(
vals.astype(str).astype(object), hash_key, encoding
str_vals.astype(object), hash_key, encoding
)

# Then, redistribute these 64-bit ints within the space of 64-bit ints
Expand Down
13 changes: 13 additions & 0 deletions pandas/tests/util/test_hashing.py
Original file line number Diff line number Diff line change
Expand Up @@ -353,3 +353,16 @@ def test_hash_collisions():

result = hash_array(np.asarray(hashes, dtype=object), "utf8")
tm.assert_numpy_array_equal(result, np.concatenate([expected1, expected2], axis=0))


def test_hash_with_tuple():
# GH#28969 array containing a tuple raises on call to arr.astype(str)
# apparently a numpy bug github.com/numpy/numpy/issues/9441

df = pd.DataFrame({"data": [tuple("1"), tuple("2")]})

hash_pandas_object(df)

df2 = pd.DataFrame({"data": [tuple([1]), tuple([2])]})

hash_pandas_object(df2)