Using ctypes.CField for annotations of fields within ctypes.Structure-like

[junkmd]

Currently, attributes of ctypes.Structure, such as ctypes.wintypes.POINT, are annotated with names and types that specified in _fields_.

typeshed/stdlib/ctypes/wintypes.pyi

Lines 126 to 128 in df08fce

	class POINT(Structure):
	x: LONG
	y: LONG

The problem with this approach is that at runtime, the attributes return types like int or bytes, while type checkers interpret them as returning c_int or c_char.

Below is the simplest example of a structure created for explanation purposes.
At runtime, the types of these fields are ctypes.CField. In the example below, x is a data descriptor that returns int as the getter and accepts both c_int and int as the setter.

>>> import ctypes
>>>      
>>> class Foo(ctypes.Structure):
...     pass
... 
>>> Foo._fields_ = [('x', ctypes.c_int)]
>>> Foo.x
<Field type=c_long, ofs=0, size=4>
>>> type(Foo.x) 
<class '_ctypes.CField'>
>>> foo = Foo()
>>> foo
<__main__.Foo object at 0x0000023CDB80B8C0>
>>> foo.x
0
>>> foo.x = 3
>>> foo.x
3
>>> foo.x = ctypes.c_int(2)
>>> foo.x
2

Having stubs with types that deviate from the runtime is not an ideal situation.
Therefore, I propose modifying CField as shown below for use in annotating fields.

_T = TypeVar("_T")
_CT = TypeVar("_CT", bound=_CData)


class CField(Generic[_T, _CT]):
    offset: int
    size: int
    @overload
    def __get__(self, instance: None, owner: type[Any]) -> Self: ...
    @overload
    def __get__(self, instance: Any, owner: type[Any] | None) -> _T: ...
    def __set__(self, instance: Any, value: _T | _CT) -> None: ...


class Foo(Structure):
    x: ClassVar[CField[int, c_int]]


# Foo._fields_ = [('x', c_int)]  # required in runtime


a = Foo.x  # CField[int, c_int]
foo = Foo()
b = foo.x  # int
foo.x = 3  # OK
foo.x = c_int(2)  # OK
foo.x = 3.14  # NG
foo.x = c_double(3.14)  # NG

another idea

I thought it elegant to specify only subclasses of _SimpleCData[_T] as type parameters, given the potential for inferring _T, as shown below.

class Bar(Structure):
    x: ClassVar[CField[c_int]]  # returns `int`, can take `int` or `c_int`

However, current static type system cannot that.

Moreover, considering that there exist not only subclasses of _SimpleCData defined within ctypes, but also third-party developers who define _SimpleCData subclasses within their own projects.

There is the redundancy, but I believe that utilizing the two type parameters would enhance flexibility.

[junkmd]

I know python/cpython#104533. It might be necessary to communicate with stakeholders in https://discuss.python.org/t/annotation-based-sugar-for-ctypes/26579/ .

[srittau]

I've only looked briefly at your proposal, but using the descriptor protocol looks indeed to be the right solution in this case. An exploratory PR to judge the effects of such a change would be welcome!

[junkmd]

While working, I noticed something.
In fields like Array[c_wchar], we cannot assign an instance of the Array[c_wchar] type; we can only assign types like str.

>>> import ctypes
>>> class Foo(ctypes.Structure):
...     pass
... 
>>> Foo._fields_ = [('x', (ctypes.c_wchar * 14))] 
>>> Foo.x
<Field type=c_wchar_Array_14, ofs=0, size=28>
>>> foo = Foo()
>>> foo.x
''
>>> foo.x = 'spam'
>>> foo.x
'spam'
>>> y = (ctypes.c_wchar * 14)()      
>>> y.value = 'ham' 
>>> foo.x = y
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unicode string expected instead of c_wchar_Array_14 instance

Therefore, I believe it would be more appropriate to have a descriptor protocol with below three type parameters;

• The type that configured to the field (the type passed to _fields_)
• The type that the field returns
• The type that the field accepts

They are such as CField[_CT, _RetT, _SetT].

[junkmd]

In the initial PR, I intend to make changes only to _ctypes, ctypes, and wintypes.
For other third-party libraries, as I am not well-versed in their specifications, I would prefer to leave that topic to other maintainers.

[junkmd]

I PRed #10595.

[junkmd]

#10595 is merged.

[einarwar]

Sorry to comment on a closed issue. I posted the following issue on pyright: microsoft/pyright#5932 (comment) which redirected me back here.

In the following example, both using from_buffer_copy() inside something() and calling it directly on the MyStruct object gives a pyright error

import ctypes
from typing import ClassVar


class MyStruct(ctypes.Structure):
    _fields_ = [("a", ctypes.c_uint32)]


class MyClass:
    _struct: ClassVar = MyStruct

    @classmethod
    def something(cls, encoded: bytes):
        return cls._struct.from_buffer_copy(encoded)


MyStruct.from_buffer_copy(b"1234")

I have not used the ctypes library a lot, am i doing something wrong by calling a classmethod (e.g from_buffer_copy() on a Structure object directly?

[junkmd]

The curious thing is that Structure is a subclass of _ctypes._CData, which has a from_buffer_copy class method, yet it ended up in this situation.

_StructUnionMeta has __getattr__, and when we access attributes not statically defined in _StructUnionBase, it returns _CField.

However, _StructUnionBase is a subclass of _CData, so from_buffer_copy is already defined.

Before I changed the type stubs, _CField has not been Callable.

Is there some interaction between the descriptor protocol and metaclasses that I'm not aware of?

[junkmd]

I have the understanding that the issue has been resolved, as mentioned below.

#10777
#10795
microsoft/pyright#5932 (comment), microsoft/pyright#5932 (comment)
https://github.com/microsoft/pyright/releases/tag/1.1.330