BUG: (PyPy) sometimes find_stack_level() will stop on functools.partial

[mattip]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

I am not sure the bug exists in the latest version and main branch, this is tested against 2.0.1. Although I have no reason to think someone fixed it. Searching issues for find_stack_level does not work, github seems to find unrelated issues. In any case, this reproduces running the test_datetime64 tests with PyPy:

pypy -m pytest lib/pypy3.9/site-packages/pandas/tests/arithmetic/test_datetime64.py -k test_dt64arr_sub_dt64object_array -sv

Issue Description

Working through the failures in conda-forge/pandas-feedstock#162, it seems there is a difference between CPython and PyPy when emitting warnings on array_ops. On PyPy, some of these are functools.partial functions, so when climbing the stack to figure out at what level to emit the warning, the algorithm stops inside functools. I instrumented the function with print statements to figure out what is going on (and added a change to make the tests pass):

diff --git a/pandas/util/_exceptions.py b/pandas/util/_exceptions.py
index f300f2c52f..7c5003bae2 100644
--- a/pandas/util/_exceptions.py
+++ b/pandas/util/_exceptions.py
@@ -41,9 +41,12 @@ def find_stack_level() -> int:
     # https://stackoverflow.com/questions/17407119/python-inspect-stack-is-slow
     frame = inspect.currentframe()
     n = 0
+    print("")
     while frame:
         fname = inspect.getfile(frame)
-        if fname.startswith(pkg_dir) and not fname.startswith(test_dir):
+        print("    " + fname)
+        if fname.endswith("functools.py") or (
+                fname.startswith(pkg_dir) and not fname.startswith(test_dir)):
             frame = frame.f_back
             n += 1
         else:

This prints out, for some tests

lib/pypy3.9/site-packages/pandas/tests/arithmetic/test_datetime64.py::TestDatetime64Arithmetic::test_dt64arr_sub_dt64object_array[python-array-ZoneInfo(key='UTC')]
     <path>/lib/pypy3.9/site-packages/pandas/util/_exceptions.py
    <path>/lib/pypy3.9/site-packages/pandas/core/arrays/datetimelike.py
    <path>/lib/pypy3.9/site-packages/pandas/core/arrays/datetimelike.py
    <path>/lib/pypy3.9/site-packages/pandas/core/ops/common.py
    <path>/lib/pypy3.9/site-packages/pandas/tests/arithmetic/test_datetime64.py
PASSED

and for others (note the functools in the middle)

lib/pypy3.9/site-packages/pandas/tests/arithmetic/test_datetime64.py::TestDatetime64Arithmetic::test_dt64arr_sub_dt64object_array[python-DataFrame-ZoneInfo(key='UTC')]     
    <path>/lib/pypy3.9/site-packages/pandas/util/_exceptions.py
    <path>/lib/pypy3.9/site-packages/pandas/core/arrays/datetimelike.py
    <path>/lib/pypy3.9/site-packages/pandas/core/arrays/datetimelike.py
    <path>/lib/pypy3.9/site-packages/pandas/core/ops/common.py
    <path>/lib/pypy3.9/site-packages/pandas/core/ops/array_ops.py
    <path>/lib/pypy3.9/functools.py
    <path>/lib/pypy3.9/site-packages/pandas/core/internals/ops.py
    <path>/lib/pypy3.9/site-packages/pandas/core/internals/managers.py
    <path>/lib/pypy3.9/site-packages/pandas/core/frame.py
    <path>/lib/pypy3.9/site-packages/pandas/core/frame.py
    <path>/lib/pypy3.9/site-packages/pandas/core/arraylike.py
    <path>/lib/pypy3.9/site-packages/pandas/core/ops/common.py
    <path>/lib/pypy3.9/site-packages/pandas/tests/arithmetic/test_datetime64.py

Expected Behavior

Without the fix, tests fail (and the warning has the wrong source file position). With the fix they pass. The fix I proposed seems rather hacky.

Installed Versions

INSTALLED VERSIONS

commit : 0f43794
python : 3.9.16.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.0-78-generic
Version : #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.0.3
numpy : 1.25.2
pytz : 2023.3
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.2.1
Cython : None
pytest : 7.4.0
hypothesis : 6.82.2
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli :
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[lithomas1]

Do you think this is a bug in PyPy?
(If it's limited to just functools/PyPy I think I would be fine taking the patch)

Testing this is an issue, though, if it's going to be flaky like that.

[mattip]

The use of partial seems to come from get_array_op. The difference is that on PyPy, functools.partial is part of the frame stack, where on CPython it is not. See https://foss.heptapod.net/pypy/pypy/-/issues/3988. So for PyPy currently this is required. I don't think it is flaky as much as it depends on the particular array_op used.

[mattip]

CPython has a c-extension _functools.partial which does not show up in the stack.

[mattip]

It turns out PyPy has a __pypy__.hidden_applevel decorator. This will be fixed with a new PyPy release.