BUG: read_csv not erroring on a bad line with extra columns

[ashja99]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

(this was also posted to Stack Overflow, where I was told this seems like a bug https://stackoverflow.com/questions/66541816/pandas-read-csv-not-erroring-on-a-bad-line-with-extra-columns)

Code Sample, a copy-pastable example

I have the below simple csv

ownername,streetno,streetname
me,320,main st,just,absolute,garbage
you,40,mint ave

The command I'm using to read the file is

import csv
import pandas

file = 'path_to_csv'
data = pandas.read_csv(file,',',header=0, quotechar=None, quoting = csv.QUOTE_NONE, index_col=False)

As long as the extra values (just,absolute,garbage) occur on the first row of data, it will parse the file without errors, giving me the below DataFrame

  ownername  streetno streetname
0        me       320    main st
1       you        40   mint ave

Problem description

This is not erroring on the bad line with extra columns as long as the bad line is the first line of data. Explicitly setting error_bad_lines=True has no effect.

Expected Output

Error tokenizing data. C error: Expected 3 fields in line 2, saw 6, e.g. the same as what happens when the extra columns are only on the 2nd line of data.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f2c8480
python : 3.7.2.final.0
python-bits : 32
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : None.None
pandas : 1.2.3
numpy : 1.20.1
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 54.1.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : 3.0.6
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 1.3.23
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : None
numba : None

[rhshadrach]

Thanks for the report! I've confirmed this on master, further investigation and PR to fix would be very welcome.

[njriasan]

I've also had this issue with error_bad_lines=False. This seems to suggest the issue occurs when the first row has too many columns. With the above csv and pd.read_csv("simple.csv", error_bad_lines=False) I get:

                 ownername  streetno streetname
me  320 main st       just  absolute    garbage
you 40  mint ave       NaN       NaN        NaN

but if I flip the csv line order:

ownername,streetno,streetname
you,40,mint ave
me,320,main st,just,absolute,garbage

I get:

 ownername  streetno streetname
0       you        40   mint ave

and without error_bad_lines=False I get ParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 6.

[njriasan]

I did a little more exploring and the error appears to be related to assuming an implicit index. The Python engine infers the index type from the first line, which produces this issue. Since this is documented I believe it would probably require major changes to fix this directly without index_col=False. Now that also appears to have a bug and gives the faulty line, just only with its first 3 entries. I'll try see if I can better understand that issue.

[njriasan]

For the Python parser the source of the issue appears to be the line if max_len > col_len and self.index_col is not False and self.usecols is None: in python_parser.py. The self.index_col is not False check skips the error checking for rows, which I don't understand.

This seems like the bug as removing fixes the issue. @rhshadrach Do you have any insight as to why checks for malformed lines are disabled with index_col is False. I assume the C parser will have something similar and I'm happy to submit a PR assuming this is actually the bug.

[njriasan]

In the C engine case, the issue seems to arise from:

if (!(self->lines <= self->header_end + 1) &&
        (self->expected_fields < 0 && fields > ex_fields) && !(self->usecols)) {

in tokenizer.c. More specifically self->lines <= self->header_end + 1 ensures that an error is never triggered on row 1. This seems to result from assuming index_col=False is never true, so that +1 should probably be logic equivalent to index_col==False.

[Gaerdy]

I'm having the same issue on a project in a slightly different way.

I try to read a csv file without named columns like this :

a;b;c   
d;e;f;f;f;f;f;f

import pandas

fields = [1, 2, 3]

df = pandas.read_csv('file_path', sep=';', names=fields, error_bad_lines=False)

I would expect to have a dataframe of one row and a warning for the second line of my file, like it does without the names option. (Running on Python 3.8.5 and Pandas 1.2.4)

[njriasan]

I'm having the same issue on a project in a slightly different way.

I try to read a csv file without named columns like this :

a;b;c   
d;e;f;f;f;f;f;f

import pandas

fields = [1, 2, 3]

df = pandas.read_csv('file_path', sep=';', names=fields, error_bad_lines=False)

I would expect to have a dataframe of one row and a warning for the second line of my file, like it does without the names option. (Running on Python 3.8.5 and Pandas 1.2.4)

Hi @Gaerdy. I think the bug you are encountering is this issue: #22144. I have an open PR, that resolves both that issue and this issue and it works on your example.

[Gaerdy]

Hi @Gaerdy. I think the bug you are encountering is this issue: #22144. I have an open PR, that resolves both that issue and this issue and it works on your example.

Hi @njriasan, thanks for your reply. I've thought for a second it was working well with the engine='python' but... there is a but.
It is working fine except if the "to many columns" line is the first or the second one. Does your PR fixing this too ?

[phofl]

We raise a ParserWarning in this case since 1.3, see #21768