-
-
Notifications
You must be signed in to change notification settings - Fork 18.5k
BUG: fix read_json ignoring the dtype with the pyarrow engine #60997
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 30 commits
835edf6
30d21de
074b3cb
c3f9e7f
28fa332
5a8158b
73f18a4
72675c9
bf830f5
2828ccb
46369f2
ef069f5
025fb30
e1d202d
1d71fe6
3954c84
eb6b283
2572a32
0d85bfe
00f2085
04ff6da
18f69c5
de97266
abc2418
e87097f
a855a59
a4b7f95
6406840
4626ad7
7d7171b
883b84b
80881ae
8df8914
5c581fc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,9 @@ | ||
import datetime | ||
from datetime import timedelta | ||
from io import StringIO | ||
from io import ( | ||
BytesIO, | ||
StringIO, | ||
) | ||
import json | ||
import os | ||
import sys | ||
|
@@ -2183,6 +2186,38 @@ def test_read_json_dtype_backend( | |
# string_storage setting -> ignore that for checking the result | ||
tm.assert_frame_equal(result, expected, check_column_type=False) | ||
|
||
@td.skip_if_no("pyarrow") | ||
@pytest.mark.filterwarnings("ignore::DeprecationWarning") | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you add this There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I couldn't find a way to do that with pytest filterwarnings There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. def test_foo(..., request):
if pa_version_under16p0:
request.applymarker(pytest.mark.filterwarnings(...)) |
||
def test_read_json_pyarrow_with_dtype(self, request): | ||
pa = pytest.importorskip("pyarrow") | ||
version_tuple = tuple(map(int, pa.__version__.split("."))) | ||
|
||
if version_tuple[0] < 16: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You can use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. thanks! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looks like as of 5c581fc, this filter got removed? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yep sorry forgot to leave a comment. I couldn't get this method to work, So I have gone back to using the decorator. I have filtered it to just ignore the BlockManager deprecation warning, is this specific enough? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What errors did you encounter? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I didn't encounter any errors but it wasn't properly filtering the warning so the test kept failing |
||
request.applymarker( | ||
pytest.mark.filterwarnings("ignore::DeprecationWarning") | ||
) | ||
|
||
dtype = {"a": "int32[pyarrow]", "b": "int64[pyarrow]"} | ||
json = b'{"a": 1, "b": 2}\n' | ||
|
||
df = read_json( | ||
BytesIO(json), | ||
dtype=dtype, | ||
lines=True, | ||
engine="pyarrow", | ||
will-larkin marked this conversation as resolved.
Show resolved
Hide resolved
|
||
dtype_backend="pyarrow", | ||
) | ||
|
||
result = df.dtypes | ||
expected = Series( | ||
will-larkin marked this conversation as resolved.
Show resolved
Hide resolved
|
||
data=[ | ||
pd.ArrowDtype.construct_from_string("int32[pyarrow]"), | ||
pd.ArrowDtype.construct_from_string("int64[pyarrow]"), | ||
], | ||
index=["a", "b"], | ||
) | ||
tm.assert_series_equal(result, expected) | ||
|
||
@pytest.mark.parametrize("orient", ["split", "records", "index"]) | ||
def test_read_json_nullable_series(self, string_storage, dtype_backend, orient): | ||
# GH#50750 | ||
|
Uh oh!
There was an error while loading. Please reload this page.