Add PyUnicodeWriter API

[vstinner]

I propose adding a new public PyUnicodeWriter API:

typedef struct PyUnicodeWriter PyUnicodeWriter; // members are private

PyAPI_FUNC(PyUnicodeWriter*) PyUnicodeWriter_Create(Py_ssize_t length);

// Destroy the writer
PyAPI_FUNC(void) PyUnicodeWriter_Discard(PyUnicodeWriter *writer);

// Create the final string and destroy the writer
PyAPI_FUNC(PyObject*) PyUnicodeWriter_Finish(PyUnicodeWriter *writer);

// Write a single character
PyAPI_FUNC(int) PyUnicodeWriter_WriteChar(
    PyUnicodeWriter *writer,
    Py_UCS4 ch);

// Decode a string from UTF-8
PyAPI_FUNC(int) PyUnicodeWriter_WriteUTF8(
    PyUnicodeWriter *writer,
    const char *str,  // decoded from UTF-8
    Py_ssize_t len);  // use strlen() if len < 0

// Similar to PyUnicode_FromFormat(format, ...)
PyAPI_FUNC(int) PyUnicodeWriter_Format(
    PyUnicodeWriter *writer,
    const char *format,
    ...);

// Write str(obj)
PyAPI_FUNC(int) PyUnicodeWriter_WriteStr(
    PyUnicodeWriter *writer,
    PyObject *obj);

// Write repr(obj)
PyAPI_FUNC(int) PyUnicodeWriter_WriteRepr(
    PyUnicodeWriter *writer,
    PyObject *obj);

// Write str[start:end]
PyAPI_FUNC(int) PyUnicodeWriter_WriteSubstring(
    PyUnicodeWriter *writer,
    PyObject *str,
    Py_ssize_t start,
    Py_ssize_t end);

This API is based on the existing private _PyUnicodeWriter API which exists for 12 years in Python. The private API is used by 5 projects on PyPI top 7,500 projects:

• Cython (3.0.9)
• asyncpg (0.29.0)
• catboost (1.2.3)
• frozendict (2.4.0)
• immutables (0.20)

Example of usage:

static PyObject *
union_repr(PyObject *self)
{
    unionobject *alias = (unionobject *)self;
    Py_ssize_t len = PyTuple_GET_SIZE(alias->args);

    PyUnicodeWriter *writer = PyUnicodeWriter_Create();
    if (writer == NULL) {
        return NULL;
    }

    for (Py_ssize_t i = 0; i < len; i++) {
        if (i > 0 && PyUnicodeWriter_WriteUTF8(writer, " | ", 3) < 0) {
            goto error;
        }
        PyObject *p = PyTuple_GET_ITEM(alias->args, i);
        if (PyUnicodeWriter_WriteRepr(writer, p) < 0) {
            goto error;
        }
    }
    return PyUnicodeWriter_Finish(writer);

error:
    PyUnicodeWriter_Discard(writer);
    return NULL;
}

• Issue: [C API] Add an efficient public PyUnicodeWriter API python/cpython#119182
• PR: gh-119182: Add PyUnicodeWriter C API python/cpython#119184

[vstinner]

The API is open for bikeshedding :-) See also the issue for different proposed names. I'm not sure about PyUnicodeWriter_Finish() name.

@encukou proposed PyUnicodeWriter_WriteUCS4Char() for PyUnicodeWriter_WriteChar(). I prefer the shorter name, there is no plan to add other WriteChar() functions, since Py_UCS4 covers the whole Unicode Character Set.

[vstinner]

Oh, an important note, this new API leaves the existing private API unchanged. Once a decision will be taken on this public API, I plan to open a new issue to discuss removing the private API. I prefer to discuss it separately.

[vstinner]

cc @gvanrossum @zooba

[zooba]

I like this proposal in general, there are a few simplifications to the API we should probably make, but overall I think this is a good one for us to expose.

• I think all functions should return a status code (i.e. no voids)
• agreed with calling it WriteChar() and always taking UCS4 (just like the C library functions that take int)
• Finish() is fine to me - something like GetString() would suggest you still have to destroy it afterwards
• how important is SetOverallocate? What happens if we just always overallocate? What happens if you don't set it? Is this really just setting the capacity? Or it is a flag that affects overallocation behaviour?
• I would personally like a WriteUCS2(wchar_t *s, ...) as well, but would survive calling Format("%w", s) (or whatever the code is)
• I like this for a Limited API, but we'd need a varargs alternative - how would that look, and should we design it now?

[davidhewitt]

I saw this and have a passing thought it's interesting to me because it might be a way to build Unicode objects using Rust formatting machinery while avoiding an allocation on the Rust side.

But I don't know if the performance of all the FFI calls would be worse overhead than the allocation (probably depends on use case).

So while Rust/PyO3 might be a consumer of this API, my intuition is that the most efficient thing we could ever do would be to allocate a utf8 buffer on the Rust side and transfer ownership of that into a Unicode objects via a pointer plus a function pointer for the destructor (but I know that would require a lot of work inside the interpreter to support such a thing).

All to say this caught my eye and I'm not sure yet how much it might be used by Rust 😁

[vstinner]

I think all functions should return a status code (i.e. no voids)

PyUnicodeWriter_SetOverallocate() only sets an internal flag, why should it be able to report an error?

how important is SetOverallocate? What happens if we just always overallocate? What happens if you don't set it? Is this
really just setting the capacity? Or it is a flag that affects overallocation behaviour?

Controlling overallocation is a key for performance: you should disable overallocation before the last write.

Example (pseudo-code without error checking):

writer = PyUnicodeWriter_Create();
PyUnicodeWriter_WriteChar(writer, "x");
PyUnicodeWriter_WriteUTF8(writer, <very long string>);
return PyUnicodeWriter_Finish(writer);

The WriteUTF8() call is inefficient since the very long string will be overallocated, and then truncated by Finish(). It's better to disable overallocation:

writer = PyUnicodeWriter_Create();
PyUnicodeWriter_WriteChar(writer, "x");
PyUnicodeWriter_SetOverallocate(writer, 0);
PyUnicodeWriter_WriteUTF8(writer, <very long string>);
return PyUnicodeWriter_Finish(writer);

I would personally like a WriteUCS2(wchar_t *s, ...) as well, but would survive calling Format("%w", s) (or whatever the code is)

We can extend the API later, but I propose a minimalist API to make it more likely to be accepted :-)

I like this for a Limited API, but we'd need a varargs alternative - how would that look, and should we design it now?

PyUnicodeWriter_Format() is mostly a convenient API, it's possible to avoid it with more code.

[zooba]

We can extend the API later, but I propose a minimalist API to make it more likely to be accepted :-)

There's a very fine line between "minimalist" and "incomplete". Given we have wide-char APIs for Unicode already, this gap feels incomplete, and the overallocation flag API feels non-minimalist (to me, at least).

What do you think about having _Create() take an estimated size? Allocate the internal buffer to that size, and then no reallocations needed until the end (unless it's exceeded). Usually a realloc to make the buffer smaller is going to be cheap, right? It's only when you grow the buffer that it may need to be copied.

For really performance intensive situations I'd do my own write into a preallocated array (reused for each loop iteration, because if there's no loop then it's not "intensive"), and then just use PyUnicode_FromString at the end. But we're aiming for a combination of convenience and performance here, not too extreme in either direction. (The scenario I'm thinking of is when I want to convert an array of native strings into a single char-separated Python string. I think most recently I allocated an array myself for this, but I've definitely made a list and called join before. This proposed API would be better.)

PyUnicodeWriter_Format() is mostly a convenient API, it's possible to avoid it with more code.

Yes, but that other code likely involves other varargs APIs, or worse, APIs that rely on C's macros for varargs handling ;-) Perhaps there's a nice API we could have here to make something like _WriteStr() that passes through a NULL or consumes a reference so that _Write???(PyLong_FromLong(value)) only needs one error check? Otherwise code length is going to multiply out very quickly (but perhaps the non-C scenarios I'm thinking of here need to use all that code anyway and nothing would be saved?)

[vstinner]

For really performance intensive situations I'd do my own write into a preallocated array (reused for each loop iteration, because if there's no loop then it's not "intensive"), and then just use PyUnicode_FromString at the end. But we're aiming for a combination of convenience and performance here, not too extreme in either direction. (The scenario I'm thinking of is when I want to convert an array of native strings into a single char-separated Python string. I think most recently I allocated an array myself for this, but I've definitely made a list and called join before. This proposed API would be better.)

Performance still remains a mystery to me :-) Recently, I tried to micro-optimize PyUnicode_FromFormat("%R", str_obj) (for Python str object) using a private _PyUnicodeWriter internally. It was 10 to 30% slower. I didn't make sense to me, but well, I trust benchmark numbers :-)

In some cases, creating a temporary string and then write the string into _PyUnicodeWriter is faster than writing directly into _PyUnicodeWriter buffer. These operations are around 100 ns, it's very fast, and any abstraction layer (_PyUnicodeWriter) can make the code slower.

Proposed public PyUnicodeWriter API is not only about performance. It's designed for the limited C API which has no API to write characters to build a Python str object. It only has functions to decode strings from another format. See also python/cpython#119609 : there is no API in the limited API to create a string from a UCS-4 buffer for example (@serhiy-storchaka suggests using PyUnicode_DecodeUTF32()).

[vstinner]

I would personally like a WriteUCS2(wchar_t *s, ...) as well, but would survive calling Format("%w", s) (or whatever the code is)

UCS-2 is different than Windows "UTF-16" wchar_t strings. UCS-2 is for strings limited to the BMP range: [U+0000; U+ffff]. Whereas Windows "UTF-16" wchar_t strings use surrogate pairs.

For Windows "UTF-16" wchar_t strings, you can reuse PyUnicode_FromWideChar(). That's what I mean by: "you can reuse the existing API" when PyUnicodeWriter_Format() is not available. Internally, PyUnicodeWriter_Format() is likely doing the same: create a temporary string and then write it into the writer.

[vstinner]

cc @encukou @erlend-aasland

[erlend-aasland]

I like Steve's suggestion about _Create taking an estimated size. The _SetOverallocate API feels unergonomic.

[vstinner]

I like Steve's suggestion about _Create taking an estimated size. The _SetOverallocate API feels unergonomic.

Ok, I updated the API:

• I added an initial length parameter to PyUnicodeWriter_Create().
• I removed the PyUnicodeWriter_SetOverallocate() function.

[zooba]

UCS-2 is different than Windows "UTF-16" wchar_t strings. UCS-2 is for strings limited to the BMP range: [U+0000; U+ffff]. Whereas Windows "UTF-16" wchar_t strings use surrogate pairs.

Trust me, I'm well aware of this 😉 I used UCS-2 specifically because it means we can ignore surrogate pairs and pass them through (which is what Windows does, despite claiming "UTF-16"), but if you prefer to properly decode then that's fine by me.

I just know that the vast majority of my cases will involve wchar_t * strings, so selfishly, I'd like to not have to write the conversion myself or have to use _Format(...) in all cases. But it's a selfish ask, so if you don't like it, don't do it and I'll live.

[serhiy-storchaka]

Here is how memory management works in many performant codecs:

• We estimate the upper size of the output data and allocate the str or bytes object of the specified size.
• Normally, bytes/characters are translated without checking for buffer overflow.
• If a UnicodeError occurs, the error handler is called, which can return a replacement string. The replacement string can need resizing the output buffer. The new size is calculated taking into account the current positions in the input string and the output buffer. If previously translated characters consumed less than expected, the new size can be less than the old size, so no resize is needed.
• Finally the output buffer is resized to the actual size.

Therefore, we need the ability not only to specify the initial size of the buffer, but also to change it during the writing process. It is more convenient to specify a minimum reserved size in addition to already written characters than an absolute buffer size.