BUG: GH11344 in pandas.json when file to read is big

[kawochen]

closes #11344
taken from ultrajson/ultrajson#145

test code

import json
import pandas
from pandas.compat import zip, range
SIZE = 5*10**7
FILENAME = 'generated.json'
with open(FILENAME, 'w') as fileh:
    json.dump(dict(zip(range(SIZE), range(SIZE))), fileh)
with open(FILENAME) as fileh:
    pandas.json.load(fileh)

[jreback]

note that Travis is borked ATM

[jreback]

@kawochen repush, Travis is fixed.

easy way to put in a test for this? (you can mark it slow, but still shouldn't take forever, or blow up memory, not sure if that's possible as Travis has a limited amount)

[kawochen]

sorry about the noise - I had pushed again precisely when Travis was throwing another tantrum.
I can't think of a good way to write a test, but locally, you could change SIZE_MAX to like 10 and check that even if that path is taken, there will be no seg fault. That matters for 32-bit environments way more than for 64-bit.
But that path shouldn't have been taken in OP's case.

[jreback]

@kawochen yeh, let me give this a try on windows.

can you put the test case generation code here for refernce

[kawochen]

added to top

[jreback]

ok dind't crash windows (though didn't finish as don't have a lot of memory and was swapping)...oh well

[jreback]

thanks for the fix @kawochen