Add support of UnicodeTranslateError in standard error handlers

[serhiy-storchaka]

BPO	23676
Nosy	@malemburg, @doerwalter, @ncoghlan, @vstinner, @vadmium, @serhiy-storchaka
Files	translate_error_handlers.patch translate_error_handlers_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2015-03-15.21:59:08.104>
labels = ['interpreter-core', 'type-feature']
title = 'Add support of UnicodeTranslateError in standard error handlers'
updated_at = <Date 2015-03-26.22:45:41.845>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2015-03-26.22:45:41.845>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2015-03-15.21:59:08.104>
creator = 'serhiy.storchaka'
dependencies = []
files = ['38502', '38504']
hgrepos = []
issue_num = 23676
keywords = ['patch']
message_count = 8.0
messages = ['238163', '238180', '238973', '239018', '239353', '239355', '239357', '239358']
nosy_count = 6.0
nosy_names = ['lemburg', 'doerwalter', 'ncoghlan', 'vstinner', 'martin.panter', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23676'
versions = ['Python 3.5']

Linked PRs

• gh-67864: Add support of UnicodeTranslateError in standard error handlers #121212

[serhiy-storchaka]

Proposed patch adds support of UnicodeTranslateError in standard error handlers "xmlcharrefreplace", "namereplace" and "surrogatepass". Support in "backslashreplace" was added in bpo-22286, support in "strict", "ignore" and "replace" was always, support in "surrogateescape" is unlikely possible.

This can be used with bpo-18814.

[serhiy-storchaka]

Fixed a bug in "surrogatepass" with translating and added the versionchanged directive.

[vadmium]

I think I saw your patch for bpo-18814 proposes to use UnicodeTranslateError. Is there any other case where it is used, either currently or in the past? All I know of it is the documentation, which says it is raised “during translating”.

Experimenting with the constructor reveals that the “object” attribute is only allowed to be a text string (not bytes). So perhaps “translating” actually means converting from text strings to text strings, like “rot-13”. It would be nice if this were documented somewhere, rather than just saying translating is now supported.

[serhiy-storchaka]

No, currently UnicodeTranslateError is not used in the stdlib in 3.x. But it is documented and supported by some error handlers. I think it should be wider used in text-to-text translations similar to proposed in bpo-18814.

[vstinner]

I'm sorry, I don't understand this issue. Could you please elaborate the use case? Why do you want to support more error handlers? str.translate() calls _PyUnicode_TranslateCharmap() with errors="ignore", it's not possible to choose the error handler.

Many codecs are implemented in Python and some of them are implemented with "charmap". Does this issue enhance the codecs implemented with "charmap"?

"a\udc80".encode("latin9", "surrogatepass") raises UnicodeEncodeError with and without the patch, b"\x81".decode("cp1252", "surrogatepass") raises UnicodeDecodeError with and without the patch.

Hum, I'm not sure that codecs.charmap_build() is related str.translate().

[serhiy-storchaka]

str.encode, bytes.decode and str.translate are unrelated to UnicodeTranslateError. But str.transform could be.

[vstinner]

Serhiy Storchaka added the comment:

str.encode, bytes.decode and str.translate are unrelated to UnicodeTranslateError. But str.transform could be.

Can you please give an example of Python code to show your change?

[serhiy-storchaka]

bpo-18814