Skip to content

BUG: Constructing a df using 2D numpy arrays raises misleading error #42463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
loodvn opened this issue Jul 9, 2021 · 2 comments · Fixed by #42646
Closed
2 of 3 tasks

BUG: Constructing a df using 2D numpy arrays raises misleading error #42463

loodvn opened this issue Jul 9, 2021 · 2 comments · Fixed by #42646
Assignees
Labels
Bug Constructors Series/DataFrame/Index/pd.array Constructors Error Reporting Incorrect or improved errors from pandas
Milestone

Comments

@loodvn
Copy link

loodvn commented Jul 9, 2021

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

>>> import pandas as pd
>>> import numpy as np
>>> a = [[1], [2]]
>>> pd.DataFrame({'a': np.array(a), 'b': np.array(a)}) 
ValueError: If using all scalar values, you must pass an index

Problem description

Hey all! Thanks for the library, I use pandas on a daily basis.

When attempting to create a DataFrame with a 2-dimensional numpy array as input, a misleading ValueError: If using all scalar values, you must pass an index is raised instead of raising an expected ValueError: Data must be 1-dimensional.

This behaviour changes slightly if there is more than one array:

>>> import pandas as pd
>>> import numpy as np
>>> a = [[1], [2]]
>>> pd.DataFrame({'a': a, 'b': a})   # No error, both nested lists pass validation and index is extracted
     a    b
0  [1]  [1]
1  [2]  [2]
>>> pd.DataFrame({'a': np.array(a), 'b': a})   # Expected behaviour, the nested list passes the check at pd.core.internals.construction._extract_index
ValueError: Data must be 1-dimensional
>>> pd.DataFrame({'a': np.array(a), 'b': np.array(a)}) 
ValueError: If using all scalar values, you must pass an index

I suspect this may have something to do with the check at

elif is_list_like(val) and getattr(val, "ndim", 1) == 1:

not being exhaustive and/or that the shape validation only happens after this index creation step.

Expected Output

>>> pd.DataFrame({'a': np.array(a), 'b': np.array(a)}) 
ValueError: Data must be 1-dimensional

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f00ed8f
python : 3.7.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.104+
Version : #1 SMP Sat Jun 5 09:50:34 PDT 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.0
numpy : 1.19.5
pytz : 2018.9
dateutil : 2.8.1
pip : 19.3.1
setuptools : 57.0.0
Cython : 0.29.23
pytest : 3.6.4
hypothesis : None
sphinx : 1.8.5
blosc : None
feather : 0.4.1
xlsxwriter : None
lxml.etree : 4.2.6
html5lib : 1.0.1
pymysql : None
psycopg2 : 2.7.6.1 (dt dec pq3 ext lo64)
jinja2 : 2.11.3
IPython : 5.5.0
pandas_datareader: 0.9.0
bs4 : 4.6.3
bottleneck : 1.3.2
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.2.2
numexpr : 2.7.3
odfpy : None
openpyxl : 2.5.9
pandas_gbq : 0.13.3
pyarrow : 3.0.0
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : 1.4.20
tables : 3.4.4
tabulate : 0.8.9
xarray : 0.18.2
xlrd : 1.1.0
xlwt : 1.3.0
numba : 0.51.2

@loodvn loodvn added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jul 9, 2021
@jbrockmendel
Copy link
Member

can you add an informative title to the issue

@jbrockmendel jbrockmendel added Constructors Series/DataFrame/Index/pd.array Constructors Error Reporting Incorrect or improved errors from pandas and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jul 9, 2021
@loodvn loodvn changed the title BUG: BUG: Constructing a df using 2D numpy arrays raises misleading error Jul 12, 2021
@debnathshoham
Copy link
Member

take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Constructors Series/DataFrame/Index/pd.array Constructors Error Reporting Incorrect or improved errors from pandas
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants