Fix mem leak in read_csv

[WillAyd]

Pending validation but I believe this fixes #57039

Initially I misread this function was branching based off a return value that was always 0, but the actual value lies in the side effects

[WillAyd]

@akrherz do you have a quick way of checking this?

[akrherz]

@WillAyd I am sorry, I have never built pandas before, but will try it now. I certainly appreciate the help here.

[akrherz]

I was able to build pandas, whew.
main branch memray screenshot (showing RSS) with the reproducing leak script

running @WillAyd/fix-mem-leak branch

LGTM!

[mroeschke]

@WillAyd could you add a peakmem ASV benchmark for this case?

[WillAyd]

I've added, but in this case it unfortunately would not have made a difference because the extension uses the system free call instead of PyMem_Free, so the Python interpreter does not track this.

Maybe we should use PyMem_Free, although the downside to that would be that it doesn't play nicely with LSAN

[akrherz]

I don't wish to waste dev's time here, but I am curious why the introduction of np.nan in my code causes this memory leak to trigger? A terse response of "its complicated" is more than sufficient :) An automated test would not have caught this memory leak, perhaps...

[mroeschke]

Thanks @WillAyd

[WillAyd]

I don't wish to waste dev's time here, but I am curious why the introduction of np.nan in my code causes this memory leak to trigger? A terse response of "its complicated" is more than sufficient :) An automated test would not have caught this memory leak, perhaps...

I think np.nan is a red herring - it is definitely a mistake in the code to get rid of those free blocks and LSAN detects the leak either way

Thanks again for the quick report!

[akrherz]

I think np.nan is a red herring

Thank you, but the memory leak does not happen with that line removed. Regardless, I am thrilled this is fixed :)