PyUnicode_FSConverter() has confusing reference semantics

[Yhg1s]

BPO	46083
Nosy	@Yhg1s

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-12-15.14:28:25.028>
labels = ['expert-C-API', '3.8', '3.9', '3.10', '3.11']
title = 'PyUnicode_FSConverter() has confusing reference semantics'
updated_at = <Date 2021-12-15.14:28:25.028>
user = 'https://github.com/Yhg1s'

bugs.python.org fields:

activity = <Date 2021-12-15.14:28:25.028>
actor = 'twouters'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2021-12-15.14:28:25.028>
creator = 'twouters'
dependencies = []
files = []
hgrepos = []
issue_num = 46083
keywords = []
message_count = 1.0
messages = ['408604']
nosy_count = 1.0
nosy_names = ['twouters']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue46083'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-90241: Clarify documentation for PyUnicode_FSConverter and PyUnicode_FSDecoder #128451
• [3.13] gh-90241: Clarify documentation for PyUnicode_FSConverter and PyUnicode_FSDecoder (GH-128451) #128542
• [3.12] gh-90241: Clarify documentation for PyUnicode_FSConverter and PyUnicode_FSDecoder (GH-128451) #128543

[Yhg1s]

The PyUnicode_FSConverter function has confusing reference semantics, and confusing documentation.

https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_FSConverter says the output argument "must be a PyBytesObject* which must be released when it is no longer used." That seems to suggest one must pass a PyBytesObject to it, and indeed one of the error paths assumes an object was passed (https://github.com/python/cpython/blob/main/Objects/unicodeobject.c#L4116-- 'addr' is called 'result' in the docs). Not passing a valid object would result in trying to DECREF NULL, or garbage. However, the function doesn't actually use the object, and later in the function overwrites the value *without* DECREFing it, so passing a valid object would in fact cause a leak.

I understand the function signature is the way it is so it can be used with PyArg_ParseTuple's O& format, but there are reasons to call it directly (e.g. with METH_O functions), and it would be nice if the semantics were more clear.

[encukou]

Another gotcha, if you're used to “regular” C API functions, are the return values. Should we also document these? And perhaps explicitly point out that it's different than most C API functions, in case readers skim past the numbers? e.g.:

   On failure, return 0 with an exception set.
   On success, return ``Py_CLEANUP_SUPPORTED`` is *obj* was set, or 1 if *obj* is ``NULL``.
   (Note that the values are different from most C API functions.)