Reference handling

[encukou]

Most of this is already in the devguide.

Rules for new API:

Function arguments

Functions must not steal references to their arguments.
IOW, after a function call, the caller still owns the arguments it passed in.

The caller must guarantee that references stay valid for the entire call.

Return values and "output parameters" (*result)

Functions returning references must return a new reference.
IOW, after a function call, the caller owns (a reference to) the return value.

Same rules apply to filling an output parameter (*result, #13).

Borrowed references

We're likely to have exceptions to these rules (#4) which create borrowed references.
These must be documented, and the documentation must explicitly state what the reference is borrowed "from" and how long it is valid for. (Even if it's like None can be borrowed from the interpreter so it's valid until interpreter shutdown.)

[gvanrossum]

Functions must not steal references to their arguments.
IOW, after a function call, the caller still owns the arguments it passed in.

Exception: sometimes an API exists for the sole purpose of transferring ownership. (Canonical example is DECREF, but I'm sure there are some others.)

[gvanrossum]

@encoku The heading "Borrowed references" is followed by a section that is not about borrowed references. Is some text missing, or did you mean to type a different heading, or what?

[encukou]

It shouldn't have been under a heading. I moved it to the top.

[zooba]

Functions must not steal references to their arguments.

With nogil in the mix, we probably also need to specify that callers must guarantee the reference is not lost while the function is executing. There are almost certainly cases today where a native function can borrow all its references and then potentially have them all deallocated during execution. e.g.:

PyObject *do_something(PyObject *mod, PyObject *, PyObject *)
{
    PyObject *mod_dict = PyModule_GetDict(mod); // borrowed
    PyObject *value = PyDict_GetItemString(mod, "key"); // borrowed
    // meanwhile, another thread replaces module.key and value is DECREF'd to zero
    return SomeOtherApi(value);
}

Solving this is a more complicated problem that doesn't fully belong in this issue, but we'll either need to define that the caller is responsible for keeping the reference alive, or the callee is, and we can choose that now.

I propose choosing the caller, at least until we prove that it makes it too hard for people to migrate to nogil and we decide to make it Just Work (usually).