Skip to content

COMPAT: hashtable vectors depend on refcount semantics which do not work on PyPy #15854

Closed
@mattip

Description

@mattip

PyPy 5.7 can build and pip install pandas. Most of the tests pass, instructions to reproduce are here However this code

import pandas; 
df = pandas.Series(['b','b','b','a','a','b']); 
print df.unique()

does not work on PyPy:

Traceback (most recent call last):
  File "<module>", line 1, in <module>
  File "pypy_stuff/pypy-latest/site-packages/pandas/core/series.py", line 1241, in unique
    result = super(Series, self).unique()
  File "pypy_stuff/pypy-latest/site-packages/pandas/core/base.py", line 973, in unique
    result = unique1d(values)
  File "pypy_stuff/pypy-latest/site-packages/pandas/core/nanops.py", line 811, in unique1d
    uniques = table.unique(_ensure_object(values))
  File "pandas/src/hashtable_class_helper.pxi", line 826, in pandas.hashtable.PyObjectHashTable.unique (pandas/hashtable.c:14521)
ValueError: cannot resize an array with refcheck=True on PyPy.
Use the resize function or refcheck=False

Calls to array.resize(..., refcheck=True) (note that refcheck=True is the default) check that data reallocation is needed, and if so checks that no other object is dependent on array by testing the array object's refcount. This check is unreliable on PyPy, since we do not use a reference counting garbage collector.

I started a branch to simply expose the refcheck keword through the various places needed, the commit can be seen on my fork of pandas here.

The caller in this patch can know with certainty that there are no other users of the object, and so refcheck=False is safe. The changes are a bit intrusive, so I am opening this as an issue first hoping it generates some discussion before I issue a pull request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Compatpandas objects compatability with Numpy or Python functionsInternalsRelated to non-user accessible pandas implementation

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions