BUG: Fixed incorrect stream size check (#14125)

[jeffcarey]

• closes BUG: read_csv() crashes with engine='c' #14125
• tests added / passed: test_large_difference_in_columns in c_parser_only.py, Other tests passed.
• passes git diff upstream/master | flake8 --diff
• whatsnew entry: Added to 0.20 whatsnew

Issue summary: Previously, self->stream_cap was copied into a local variable called maxstreamsize each time tokenize_bytes ran, and then this was checked in the PUSH_CHAR macro. However, there is one other place in the file where function make_stream_space() is called (in end_line()), and when this happens self->stream_cap is increased but maxstreamsize is not updated, making the check incorrect. In rare circumstances (see original issue or test case) this could cause a crash. The resolution is to just check self->stream_cap directly.

[jreback]

can you add the test from the issue?

[jeffcarey]

I can, but a few things to consider first:
The test case from the issue does not follow a simple pattern, and it contains a row that is ~1000 characters long. So putting it into the test file is possible but would look kind of ugly. The test I had added instead causes the same issue, is a little easier to understand since it follows a pattern, and fits nicely into the test suite (in terms of formatting).

So with that said, do you still want the original test? If so, in an external file or directly in the script?

[jreback]

@jeffcarey oh if it replicates that is fine then. thanks!

[jreback]

can you add a whatsnew note? ping when you push (as this is already validated).

[codecov-io]

Current coverage is 85.97% (diff: 100%)

No coverage report found for master at 9309eba.

Powered by Codecov. Last update 9309eba...38c0cce

[jeffcarey]

@jreback whatsnew note has been added.

[jreback]

thanks @jeffcarey !

keep em coming!