ENH: Pandas 2.0 with pyarrow engine add the argument like 'skip_bad_lines=True'

[sunflyfairy]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

When I use Pandas 2.0 with pyarrow engine to read CSV file, if the file get some bad lines, it will raise a error such as "pyarrow.lib.ArrowInvalid: CSV parse error: Expected 1 columns, got 2: ".

Then I added the argument '', it will raise another error "TypeError: read_csv() got an unexpected keyword argument 'skip_bad_lines'"

So,can you add the argument like 'skip_bad_lines=True' with pyarrow engine.

df_arrow = pd.read_csv(r"VBAP.CSV", engine='pyarrow', dtype_backend='pyarrow')
df_arrow = pd.read_csv(r"VBAP.CSV", engine='pyarrow', dtype_backend='pyarrow', skip_bad_lines=True)

Feature Description

df_arrow = pd.read_csv(r"VBAP.CSV", engine='pyarrow', dtype_backend='pyarrow', skip_bad_lines=True)

Alternative Solutions

df_arrow = pd.read_csv(r"VBAP.CSV", engine='pyarrow', dtype_backend='pyarrow', skip_bad_lines=True)

Additional Context

No response

[lithomas1]

This should be possible in now recent pyarrow versions where invalid_row_handler is present as a ParseOption.

PRs are welcome for this.

[sunflyfairy]

OK, thanks.
If it released, please update this issue, thanks.

[amithkk]

@lithomas1 I've created a pull request that utilizes invalid_row_handler. However, instead of introducing a new argument, I've integrated it into the existing on_bad_lines argument to read_csv when the pyarrow engine is selected. This allows for flexible handling if the invalid line has to put out a warning, be skipped, or have user-defined behavior (much like the standard implementation for the python engine).

To achieve the behavior described in the issue, the syntax would be:
pd.read_csv(r"test.csv", engine='pyarrow', dtype_backend='pyarrow', on_bad_lines='skip')

[amithkk]

Merging/Review is blocked due to #54650

[amithkk]

take