-
-
Notifications
You must be signed in to change notification settings - Fork 32.1k
CSV Writer can not write strings with nulls when no escapechar is specified #97503
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
ran into this today and i can confirm. it's pretty hard to see why it happens, though if |
This appears to be working on the tip of main |
Yep, works with Python 3.11, it looks like. Closing. |
this is not fixed in 3.10.7, and i feel that it should probably be fixed in that branch as it is a clear regression. is 3.10 going into security-only maintenance with this bug still extant? |
If this is a clear bug (behaviour contradicts documentation) and you can post a reproducer, it could be fixed unless that would require too many changes or adding new parameters. |
tip of main: Line 490 in a309ad9
tip of 3.10 (through 3.10.8): Line 484 in e7781fd
the fallback value should be |
From the way the github PR was titled («Add support for») and the discussion on the bugs.python.org ticket (mention of backward compat issue with docutils), my understanding is that this change was deemed a new feature, not a bug fix, and as such will not be backported. |
Commit messages notwithstanding, this is a clear regression, as was observed in the original post of this issue:
All the prior versions of python3 I can get my hands on at the moment write |
What is the result of writing strings like |
with a script like import csv
import io
b = io.StringIO()
csv.writer(b).writerow(["\0", "\"\0", "\n\0", "\0\r\n"])
print(repr(b.getvalue())) the results are:
|
After looking in the code I agree that it is a consequence of #56387. But while Python 3.9 and earlier allowed to write strings with the NUL character, the result could not be read back by the So I would not call it a clear regression. It was rather a fix of a bug. Now you get an error earlier, when write CSV, not when read it. |
Bitten by this regression too.
This "fix" would have made sense if the exception had a sensible error message, such as As it is, I applaud @mumbleskates for his patience, in the face of the responses here. |
thanks; i do think that this is (was?) still a regression, if pedantically so: while 3.9 could not successfully read nul characters, it could still write them... and there are other csv readers in the wild that one might be interested in sending the encoded data to, which is what i was doing. for posterity, in my case i ended up being able to use JSONL with ujson, which writes much faster than CSV anyway even though it is larger (and the encoding step was my bottleneck). (this also uncovered a horrible memory leak in orjson which i have yet to chase down) |
Bug report
This code fails in Python 3.10 with the
need to escape, but no escapechar set
error:However, this worked on Python 3.9 and earlier, and it should work since I shouldn't have to specify an
escapechar
when quoting is available.I believe this is caused by the fix for this issue: #56387
See this comment on how the C snippet added for the fix introduced this bug: #56387 (comment)
The text was updated successfully, but these errors were encountered: