Skip to content

BUG GH31355 fixed by adding more relevant error message #36169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 84 commits into from

Conversation

TAJD
Copy link
Contributor

@TAJD TAJD commented Sep 6, 2020

Added a test to trigger the reported error using code from the issue. An if statement was included to return a more helpful error message - borrowing the solution from the issue.

meeseeksmachine and others added 30 commits July 29, 2020 15:35
…umn selection no longer raises ValueError (pandas-dev#35651)

Co-authored-by: Matthew Roeschke <[email protected]>
…DataFrame with MI and datatime level (pandas-dev#35716)

Co-authored-by: Simon Hawkins <[email protected]>
@pep8speaks
Copy link

pep8speaks commented Sep 6, 2020

Hello @TAJD! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 2900:3: E227 missing whitespace around bitwise or shift operator
Line 2900:7: E225 missing whitespace around operator
Line 2902:1: E112 expected an indented block
Line 2902:1: E305 expected 2 blank lines after class or function definition, found 0
Line 2902:3: E225 missing whitespace around operator
Line 2902:5: E225 missing whitespace around operator
Line 2902:7: E225 missing whitespace around operator
Line 2904:1: E112 expected an indented block
Line 2904:1: E305 expected 2 blank lines after class or function definition, found 0
Line 2904:7: E225 missing whitespace around operator

Line 53:1: E112 expected an indented block
Line 53:3: E227 missing whitespace around bitwise or shift operator
Line 53:7: E225 missing whitespace around operator
Line 54:5: E113 unexpected indentation
Line 57:3: E225 missing whitespace around operator
Line 57:5: E225 missing whitespace around operator
Line 57:7: E225 missing whitespace around operator
Line 59:7: E225 missing whitespace around operator

Line 398:1: E122 continuation line missing indentation or outdented
Line 398:3: E225 missing whitespace around operator
Line 398:5: E227 missing whitespace around bitwise or shift operator
Line 400:1: E122 continuation line missing indentation or outdented
Line 400:3: E225 missing whitespace around operator
Line 400:5: E225 missing whitespace around operator
Line 400:8: E251 unexpected spaces around keyword / parameter equals
Line 402:1: E122 continuation line missing indentation or outdented
Line 402:7: E225 missing whitespace around operator

Line 3:3: E227 missing whitespace around bitwise or shift operator
Line 3:7: E225 missing whitespace around operator
Line 7:1: E122 continuation line missing indentation or outdented
Line 7:3: E225 missing whitespace around operator
Line 7:5: E225 missing whitespace around operator
Line 7:8: E251 unexpected spaces around keyword / parameter equals
Line 10:1: E122 continuation line missing indentation or outdented
Line 10:7: E225 missing whitespace around operator
Line 15:1: E122 continuation line missing indentation or outdented
Line 15:3: E225 missing whitespace around operator
Line 15:5: E227 missing whitespace around bitwise or shift operator
Line 17:1: E122 continuation line missing indentation or outdented
Line 17:3: E225 missing whitespace around operator
Line 17:5: E225 missing whitespace around operator
Line 17:8: E251 unexpected spaces around keyword / parameter equals
Line 18:1: E122 continuation line missing indentation or outdented
Line 18:7: E225 missing whitespace around operator
Line 1884:1: E122 continuation line missing indentation or outdented
Line 1884:3: E225 missing whitespace around operator
Line 1884:5: E227 missing whitespace around bitwise or shift operator
Line 1885:1: E122 continuation line missing indentation or outdented
Line 1888:16: E251 unexpected spaces around keyword / parameter equals
Line 1888:18: E251 unexpected spaces around keyword / parameter equals
Line 1889:30: E251 unexpected spaces around keyword / parameter equals
Line 1889:32: E251 unexpected spaces around keyword / parameter equals
Line 1911:5: E123 closing bracket does not match indentation of opening bracket's line
Line 1917:23: E251 unexpected spaces around keyword / parameter equals
Line 1917:25: E251 unexpected spaces around keyword / parameter equals
Line 1920:30: E251 unexpected spaces around keyword / parameter equals
Line 1920:32: E251 unexpected spaces around keyword / parameter equals
Line 1924:13: E123 closing bracket does not match indentation of opening bracket's line
Line 1927:11: E251 unexpected spaces around keyword / parameter equals
Line 1927:13: E251 unexpected spaces around keyword / parameter equals
Line 1929:11: E251 unexpected spaces around keyword / parameter equals
Line 1929:13: E251 unexpected spaces around keyword / parameter equals
Line 1933:9: E123 closing bracket does not match indentation of opening bracket's line
Line 1937:13: E251 unexpected spaces around keyword / parameter equals
Line 1937:15: E251 unexpected spaces around keyword / parameter equals
Line 1939:10: E251 unexpected spaces around keyword / parameter equals
Line 1939:12: E251 unexpected spaces around keyword / parameter equals
Line 1946:11: E251 unexpected spaces around keyword / parameter equals
Line 1946:13: E251 unexpected spaces around keyword / parameter equals
Line 1951:12: E251 unexpected spaces around keyword / parameter equals
Line 1951:14: E251 unexpected spaces around keyword / parameter equals
Line 1952:18: E251 unexpected spaces around keyword / parameter equals
Line 1952:20: E251 unexpected spaces around keyword / parameter equals
Line 1953:15: E251 unexpected spaces around keyword / parameter equals
Line 1953:17: E251 unexpected spaces around keyword / parameter equals
Line 1959:1: E122 continuation line missing indentation or outdented
Line 1959:3: E225 missing whitespace around operator
Line 1959:5: E225 missing whitespace around operator
Line 1959:8: E251 unexpected spaces around keyword / parameter equals
Line 1960:1: E122 continuation line missing indentation or outdented
Line 1960:7: E225 missing whitespace around operator

Line 2214:1: E305 expected 2 blank lines after class or function definition, found 0
Line 2214:3: E227 missing whitespace around bitwise or shift operator
Line 2214:7: E225 missing whitespace around operator
Line 2215:9: E113 unexpected indentation
Line 2216:3: E225 missing whitespace around operator
Line 2216:5: E225 missing whitespace around operator
Line 2216:7: E225 missing whitespace around operator
Line 2218:7: E225 missing whitespace around operator

Line 1147:3: E227 missing whitespace around bitwise or shift operator
Line 1147:7: E225 missing whitespace around operator
Line 1148:5: E113 unexpected indentation
Line 1155:3: E225 missing whitespace around operator
Line 1155:5: E225 missing whitespace around operator
Line 1155:7: E225 missing whitespace around operator
Line 1157:1: E112 expected an indented block
Line 1157:7: E225 missing whitespace around operator

Line 27:1: E112 expected an indented block
Line 27:3: E227 missing whitespace around bitwise or shift operator
Line 27:7: E225 missing whitespace around operator
Line 28:5: E113 unexpected indentation
Line 29:3: E225 missing whitespace around operator
Line 29:5: E225 missing whitespace around operator
Line 29:7: E225 missing whitespace around operator
Line 31:7: E225 missing whitespace around operator

Line 37:1: E305 expected 2 blank lines after class or function definition, found 1
Line 37:3: E227 missing whitespace around bitwise or shift operator
Line 37:7: E225 missing whitespace around operator
Line 38:5: E113 unexpected indentation
Line 39:3: E225 missing whitespace around operator
Line 39:5: E225 missing whitespace around operator
Line 39:7: E225 missing whitespace around operator
Line 41:7: E225 missing whitespace around operator

Line 631:3: E227 missing whitespace around bitwise or shift operator
Line 631:7: E225 missing whitespace around operator
Line 632:3: E225 missing whitespace around operator
Line 632:5: E225 missing whitespace around operator
Line 632:7: E225 missing whitespace around operator
Line 651:7: E225 missing whitespace around operator

Line 1066:3: E227 missing whitespace around bitwise or shift operator
Line 1066:7: E225 missing whitespace around operator
Line 1067:3: E225 missing whitespace around operator
Line 1067:5: E225 missing whitespace around operator
Line 1067:7: E225 missing whitespace around operator
Line 1147:7: E225 missing whitespace around operator

Line 33:3: E227 missing whitespace around bitwise or shift operator
Line 33:7: E225 missing whitespace around operator
Line 34:1: E302 expected 2 blank lines, found 0
Line 36:1: E305 expected 2 blank lines after class or function definition, found 0
Line 36:3: E225 missing whitespace around operator
Line 36:5: E225 missing whitespace around operator
Line 36:7: E225 missing whitespace around operator
Line 39:1: E305 expected 2 blank lines after class or function definition, found 0
Line 39:7: E225 missing whitespace around operator
Line 60:3: E227 missing whitespace around bitwise or shift operator
Line 60:7: E225 missing whitespace around operator
Line 61:1: E302 expected 2 blank lines, found 0
Line 63:1: E305 expected 2 blank lines after class or function definition, found 0
Line 63:3: E225 missing whitespace around operator
Line 63:5: E225 missing whitespace around operator
Line 63:7: E225 missing whitespace around operator
Line 66:1: E305 expected 2 blank lines after class or function definition, found 0
Line 66:7: E225 missing whitespace around operator

Line 16:3: E227 missing whitespace around bitwise or shift operator
Line 16:7: E225 missing whitespace around operator
Line 21:3: E225 missing whitespace around operator
Line 21:5: E225 missing whitespace around operator
Line 21:7: E225 missing whitespace around operator
Line 22:7: E225 missing whitespace around operator

Line 404:1: E305 expected 2 blank lines after class or function definition, found 1
Line 404:3: E227 missing whitespace around bitwise or shift operator
Line 404:7: E225 missing whitespace around operator
Line 405:9: E113 unexpected indentation
Line 406:3: E225 missing whitespace around operator
Line 406:5: E225 missing whitespace around operator
Line 406:7: E225 missing whitespace around operator
Line 408:7: E225 missing whitespace around operator

Line 831:1: E305 expected 2 blank lines after class or function definition, found 1
Line 831:3: E227 missing whitespace around bitwise or shift operator
Line 831:7: E225 missing whitespace around operator
Line 832:9: E113 unexpected indentation
Line 833:3: E225 missing whitespace around operator
Line 833:5: E225 missing whitespace around operator
Line 833:7: E225 missing whitespace around operator
Line 835:7: E225 missing whitespace around operator

Line 1467:1: E305 expected 2 blank lines after class or function definition, found 0
Line 1467:3: E227 missing whitespace around bitwise or shift operator
Line 1467:7: E225 missing whitespace around operator
Line 1469:5: E113 unexpected indentation
Line 1478:3: E225 missing whitespace around operator
Line 1478:5: E225 missing whitespace around operator
Line 1478:7: E225 missing whitespace around operator
Line 1479:7: E225 missing whitespace around operator

Line 271:1: E305 expected 2 blank lines after class or function definition, found 0
Line 271:3: E227 missing whitespace around bitwise or shift operator
Line 271:7: E225 missing whitespace around operator
Line 272:3: E225 missing whitespace around operator
Line 272:5: E225 missing whitespace around operator
Line 272:7: E225 missing whitespace around operator
Line 288:1: E305 expected 2 blank lines after class or function definition, found 0
Line 288:7: E225 missing whitespace around operator

Line 307:1: E305 expected 2 blank lines after class or function definition, found 0
Line 307:3: E227 missing whitespace around bitwise or shift operator
Line 307:7: E225 missing whitespace around operator
Line 308:3: E225 missing whitespace around operator
Line 308:5: E225 missing whitespace around operator
Line 308:7: E225 missing whitespace around operator
Line 324:1: E305 expected 2 blank lines after class or function definition, found 0
Line 324:7: E225 missing whitespace around operator

Line 212:1: E305 expected 2 blank lines after class or function definition, found 1
Line 212:3: E227 missing whitespace around bitwise or shift operator
Line 212:7: E225 missing whitespace around operator
Line 213:5: E113 unexpected indentation
Line 214:3: E225 missing whitespace around operator
Line 214:5: E225 missing whitespace around operator
Line 214:7: E225 missing whitespace around operator
Line 215:7: E225 missing whitespace around operator
Line 281:3: E227 missing whitespace around bitwise or shift operator
Line 281:7: E225 missing whitespace around operator
Line 282:5: E113 unexpected indentation
Line 283:3: E225 missing whitespace around operator
Line 283:5: E225 missing whitespace around operator
Line 283:7: E225 missing whitespace around operator
Line 284:7: E225 missing whitespace around operator
Line 310:3: E227 missing whitespace around bitwise or shift operator
Line 310:7: E225 missing whitespace around operator
Line 311:5: E113 unexpected indentation
Line 312:3: E225 missing whitespace around operator
Line 312:5: E225 missing whitespace around operator
Line 312:7: E225 missing whitespace around operator
Line 313:7: E225 missing whitespace around operator
Line 337:3: E227 missing whitespace around bitwise or shift operator
Line 337:7: E225 missing whitespace around operator
Line 338:5: E113 unexpected indentation
Line 339:3: E225 missing whitespace around operator
Line 339:5: E225 missing whitespace around operator
Line 339:7: E225 missing whitespace around operator
Line 340:7: E225 missing whitespace around operator
Line 365:3: E227 missing whitespace around bitwise or shift operator
Line 365:7: E225 missing whitespace around operator
Line 366:5: E113 unexpected indentation
Line 367:3: E225 missing whitespace around operator
Line 367:5: E225 missing whitespace around operator
Line 367:7: E225 missing whitespace around operator
Line 368:7: E225 missing whitespace around operator
Line 395:3: E227 missing whitespace around bitwise or shift operator
Line 395:7: E225 missing whitespace around operator
Line 397:5: E113 unexpected indentation
Line 430:3: E225 missing whitespace around operator
Line 430:5: E225 missing whitespace around operator
Line 430:7: E225 missing whitespace around operator
Line 431:7: E225 missing whitespace around operator

Comment last updated at 2020-09-13 11:52:47 UTC

@TAJD TAJD closed this Sep 6, 2020
@TAJD TAJD reopened this Sep 6, 2020
@TAJD
Copy link
Contributor Author

TAJD commented Sep 7, 2020

Not sure how to fix the test failures - have these been seen before?

meeseeksmachine and others added 11 commits September 7, 2020 15:06
…e does not preserve dataframe index name (pandas-dev#36221)

Co-authored-by: Irv Lustig <[email protected]>
@rhshadrach rhshadrach added Error Reporting Incorrect or improved errors from pandas Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Sep 11, 2020
@rhshadrach rhshadrach added this to the 1.2 milestone Sep 11, 2020
@rhshadrach
Copy link
Member

The failure looks related:

FAILED pandas/tests/test_sorting.py::TestSafeSort::test_codes_out_of_bound[-1]

This test ends up calling cartesian_product with the argument

X = [array([0, 1], dtype=int8), array([0, 1], dtype=int8), array([0, 1, 2], dtype=int8)]

@rhshadrach rhshadrach removed this from the 1.2 milestone Sep 11, 2020
@TAJD
Copy link
Contributor Author

TAJD commented Sep 12, 2020

My working hypothesis is that the worker failure could be due to the size of the array created to test if the error is being raised.

Using tracemalloc it can be seen that the memory allocated during cartesian_product with X as the above value or using the test case for error bounds is around 4 petabytes. This sort of thing breaks the code when the memory is actually created. What I'm wondering is on how to limit the size of the array which is created or whether there is a way to pick a test input which triggers the error without breaking the machine being used.

@jreback
Copy link
Contributor

jreback commented Sep 13, 2020

@TAJD you need to merge master

@TAJD TAJD closed this Sep 13, 2020
@TAJD
Copy link
Contributor Author

TAJD commented Sep 13, 2020

Thanks @jreback I should have read the documentation!

@TAJD TAJD mentioned this pull request Sep 13, 2020
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Error Reporting Incorrect or improved errors from pandas Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: Calling cartesian_product with large combinatorial space overflows int64 gives misleading errors
10 participants