ENH: Add on_bad_lines for pyarrow

[amithkk]

• closes ENH: Pandas 2.0 with pyarrow engine add the argument like 'skip_bad_lines=True' #54480
• All code checks passed.
• Update documentation

This adds the on_bad_lines argument to the pyarrow engine for the from_csv parser that closely follows the behaviour of the python engine. Internally utilizes pyarrow's invalid_row_handler. The built-in callable implementation slightly differs for pyarrow, so the difference is appropriately documented by pointing to pyarrow's documentation.

Usage Example:

example.csv:

a,b,c
acol1,bcol1,ccol1
acol2,ccol2

example.py

df_arrow = pd.read_csv(r"example.csv", engine='pyarrow', dtype_backend='pyarrow', on_bad_lines='warn')
print(df_arrow)

Console output

ParserWarning: Expected 3 columns, but found 2: acol2,ccol2
       a      b      c
0  acol1  bcol1  ccol1

[lithomas1]

Does this fix any tests?

There are some xfailed tests for on_bad_lines I think somewhere (sorry that I don't remember where) for the pyarrow engine.

[amithkk]

@lithomas1 The tests have been refactored to not fail for both the "python" engine and the "pyarrow" engine (see: pandas/tests/io/parser/test_unsupported.py)

I didn't find any other tests that were invalidated (as confirmed by the CI).

[amithkk]

@lithomas1 btw, is there any specific action I should be taking to request a review?

[amithkk]

Rebased from master to pull in compat workaround so that CI Passes

[mroeschke]

Can you add a whatsnew entry to 2.2.0.rst?

[lithomas1]

@lithomas1 The tests have been refactored to not fail for both the "python" engine and the "pyarrow" engine (see: pandas/tests/io/parser/test_unsupported.py)

I didn't find any other tests that were invalidated (as confirmed by the CI).

Try looking in pandas/tests/io/parser/common/test_read_errors.py

Those tests are currently skippped, but on_bad_lines is being used there. You should be able to make at least some of them pass.

(You can try xfailing the tests there by changing the "pyarrow_skip" to a "pyarrow_xfail")

[amithkk]

@lithomas1 I've gotten all the tests within test_read_errors.py applicable for this change to pass.

[amithkk]

@lithomas1 Are there any other changes that you'd like me to look at from the test perspective?

[amithkk]

Any updates on this PR?

[mroeschke]

LGTM cc @lithomas1 if you have any comments

[amithkk]

@lithomas1 Anything else that needs to be covered in this pull request?

[lithomas1]

Thanks for the ping. I left some comments.

[amithkk]

@lithomas1 I've chained the ArrowInvalid exception with ParserError so that the tests are uniform. Does this match what you had in mind?

[amithkk]

@lithomas1 Awaiting your review

[lithomas1]

This looks good to me now. Thanks!

[mroeschke]

Thanks @amithkk