Make static methods created by @staticmethod callable

[vstinner]

BPO	43682
Nosy	@rhettinger, @mdickinson, @vstinner, @methane, @markshannon, @serhiy-storchaka
PRs	bpo-43682: Make staticmethod objects callable #25117 bpo-43682: @staticmethod inherits attributes #25268

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-12.08:07:43.223>
created_at = <Date 2021-03-31.14:31:21.903>
labels = ['library', '3.10']
title = 'Make static methods created by @staticmethod callable'
updated_at = <Date 2021-04-14.00:48:59.564>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-04-14.00:48:59.564>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-12.08:07:43.223>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2021-03-31.14:31:21.903>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 43682
keywords = ['patch']
message_count = 22.0
messages = ['389905', '389906', '389907', '389909', '389910', '389914', '389916', '389917', '389963', '390496', '390525', '390594', '390607', '390639', '390704', '390738', '390802', '390823', '390829', '390841', '390848', '391024']
nosy_count = 6.0
nosy_names = ['rhettinger', 'mark.dickinson', 'vstinner', 'methane', 'Mark.Shannon', 'serhiy.storchaka']
pr_nums = ['25117', '25268']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43682'
versions = ['Python 3.10']

[vstinner]

Currently, static methods created by the @staticmethod decorator are not callable as regular function. Example:

@staticmethod
def func():
    print("my func")

class MyClass:
    method = func

func() # A: regular function
MyClass.method() # B: class method
MyClass().method() # C: instance method

The func() call raises TypeError('staticmethod' object is not callable) exception.

I propose to make staticmethod objects callable to get a similar behavior than built-in functions:

func = len

class MyClass:
    method = func

func("abc") # A: regular function
MyClass.method("abc") # B: class method
MyClass().method("abc") # C: instance method

The 3 variants (A, B, C) to call the built-in len() function work just as expected.

If static method objects become callable, the 3 variants (A, B, C) will just work.

It would avoid the hack like _pyio.Wrapper:

class DocDescriptor:
    """Helper for builtins.open.__doc__
    """
    def __get__(self, obj, typ=None):
        return (
            "open(file, mode='r', buffering=-1, encoding=None, "
                 "errors=None, newline=None, closefd=True)\n\n" +
            open.__doc__)

class OpenWrapper:
    """Wrapper for builtins.open


    Trick so that open won't become a bound method when stored
    as a class variable (as dbm.dumb does).

    See initstdio() in [Python/pylifecycle.c](https://github.com/python/cpython/blob/main/Python/pylifecycle.c).
    """
    __doc__ = DocDescriptor()

    def __new__(cls, *args, **kwargs):
        return open(*args, **kwargs)

Currently, it's not possible possible to use directly _pyio.open as a method:

class MyClass:
    method = _pyio.open

whereas "method = io.open" just works because io.open() is a built-in function.

See also bpo-43680 "Remove undocumented io.OpenWrapper and _pyio.OpenWrapper" and my thread on python-dev:

"Weird io.OpenWrapper hack to use a function as method"
https://mail.python.org/archives/list/python-dev@python.org/thread/QZ7SFW3IW3S2C5RMRJZOOUFSHHUINNME/

[mdickinson]

Seems like a duplicate of bpo-20309.

[vstinner]

Seems like a duplicate of bpo-20309.

My usecase is to avoid any behavior difference between io.open and _pyio.open functions: PEP-399 "Pure Python/C Accelerator Module Compatibility Requirements". Currently, this is a very subtle difference when it's used to define a method.

I dislike the current _pyio.OpenWrapper "hack". I would prefer that _pyio.open would be directly usable to define a method. I propose to use @staticmethod, but I am open to other ideas. It could be a new decorator: @staticmethod_or_function.

Is it worth it to introduce a new @staticmethod_or_function decorator just to leave @staticmethod unchanged?

Note: The PEP-570 "Python Positional-Only Parameters" (implemented in Python 3.8) removed another subtle difference between functions implemented in C and functions implemented in Python. Now functions implemented in Python can only have positional only parameters.

[markshannon]

I don't understand what the problem is. _pyio.open is a function not a static method.

>>> import _pyio
>>> _pyio.open
<function open at 0x7f184cf33a10>

[vstinner]

I don't understand what the problem is. _pyio.open is a function not a static method.

The problem is that _pyio.open doesn't behave exactly as io.open when it's used to define a method:

#from io import open
from _pyio import open

class MyClass:
   my_open = open

MyClass().my_open("document.txt", "w")

This code currently fails with a TypeError, whereas it works with io.open.

The problem is that I failed to find a way to create a function in Python which behaves exactly as built-in functions like len() or io.open().

[markshannon]

Isn't the problem that Python functions are (non-overriding) descriptors, but builtin-functions are not descriptors?
Changing static methods is not going to fix that.

How about adding wrappers to make Python functions behave like builtin functions and vice versa?

[vstinner]

Changing static methods is not going to fix that.

My plan for the _pyio module is:

(1) Make static methods callable
(2) Decorate _pyio.open() with @staticmethod

That would only fix the very specific case of _pyio.open(). But open() use case seems to be common enough to became the example in the @staticmethod documentation!
https://docs.python.org/dev/library/functions.html#staticmethod

Example added in bpo-31567 "Inconsistent documentation around decorators" by: commit 03b9537

commit 03b9537dc515d10528f83c920d38910b95755aff
Author: Éric Araujo \<merwok@users.noreply.github.com\>
Date:   Thu Oct 12 12:28:55 2017 -0400

    bpo-31567: more decorator markup fixes in docs (GH-3959) (bpo-3966)

[vstinner]

Isn't the problem that Python functions are (non-overriding) descriptors, but builtin-functions are not descriptors?
Changing static methods is not going to fix that.
How about adding wrappers to make Python functions behave like builtin functions and vice versa?

I would love consistency, but is that possible without breaking almost all Python projects?

Honestly, I'm annoying by the having to use staticmethod(), or at least the fact that built-in functions and functions implemented in Python don't behave the same. It's hard to remind if a stdlib function requires staticmethod() or not. Moreover, maybe staticmethod() is not needed today, but it would become required tomorrow if the built-in function becomes a Python function somehow.

So yeah, I would prefer consistency. But backward compatibility may enter into the game as usual. PR 25117 tries to minimize the risk of backward compatibility issues.

For example, if we add __get__() to built-in methods and a bound method is created on the following example, it means that all code relying on the current behavior of built-in functions (don't use staticmethod) would break :-(

class MyClass:
    # built-in function currently converted to a method
    # magically without having to use staticmethod()
    method = len

Would it be possible to remove __get__() from FunctionType to allow using a Python function as a method? How much code would it break? :-) What would create the bound method on a method call?

def func():
    ...

class MyClass:
    method = func

# magic happens here!
bound_method = MyClass().method

[serhiy-storchaka]

If make staticmethod a calllable and always wrap open, we need to change also its repr and add the __doc__ attribute (and perhaps other attributes to make it more interchangeable with the original function).

Alternate option: make staticmethod(func) returning func if it is not a descriptor.