Python 3.5/3.6 Functions with Type Hints are lost

[achapkowski]

Let's say I have a simple function:

import cloudpickle
import typing, inspect


def add(x: int=1,y: int=2 ) -> int:
    return x+y

It runs as expected:

>>> add()
3
>>> print(typing.get_type_hints(add))
{'x': <class 'int'>, 'return': <class 'int'>, 'y': <class 'int'>}

Now if I pickle it, then unpickle the function, I lose the annotation for the typing.

>>> f = cloudpickle.loads(cloudpickle.dumps(add))
>>> print(typing.get_type_hints(f))
{}
>>> f()
3

So when a method is pickled/unpickled the typing is dropped. Is there anyway to include typing in the pickling process? It helps with introspection of a user's method when you send it someplace else.

[pierreglaser]

I think this can be closed?

[achapkowski]

@pierreglaser no this still doesn't work even on later versions of python. Cloudpickle drops typing for the methods.

[pierreglaser]

If I run your example, using Python 3.7.0 and cloudpickle 1.1.1:

In [1]: import cloudpickle 
   ...: import typing, inspect 
   ...:  
   ...:  
   ...: def add(x: int=1,y: int=2 ) -> int: 
   ...:     return x+y 
   ...:                                                                                                                                                                                                            

In [2]: print(typing.get_type_hints(add))                                                                                                                                                                          
{'x': <class 'int'>, 'y': <class 'int'>, 'return': <class 'int'>}

In [3]: f = cloudpickle.loads(cloudpickle.dumps(add))                                                                                                                                                              

In [4]: f                                                                                                                                                                                                          
Out[4]: <function __main__.add(x: int = 1, y: int = 2) -> int>

In [5]: print(typing.get_type_hints(f))                                                                                                                                                                            
{'x': <class 'int'>, 'y': <class 'int'>, 'return': <class 'int'>}  

It seems to work. Did you have something else in mind?

[achapkowski]

So the fix out be in this line:

if hasattr(func, '__annotations__') and sys.version_info >= (3, 7):

Annotations have been around since PEP 3107. So it should this line needs to be changed to:

if hasattr(func, '__annotations__') and sys.version_info >= (3, 0):

That way it will work with all Python 3 versions.

[achapkowski]

@pierreglaser Added a PR to fix the issue.

[pierreglaser]

This ought to be closed now that #276 was merged.

[pierreglaser]

@achapkowski, letting you know that unfortunately, it is not possible to pickle complex annotations for Python 3.4-3.6, thus we are forced to revert #276.

[achapkowski]

@pierreglaser how so? Can you provide code to show this?

[pierreglaser]

Here are the relevant issues:
#299 (my PR to revert #276)
#298 (the issue stating that dropping annotations for python3.4-3.6 was intentional)
#193 (the issue explaining why annotations in python3.4-3.6 is a problem)

[achapkowski]

@pierreglaser instead of reverting it, how about we state that only simple annotations are support at 3.4-3.6

[cicdw]

@achapkowski The problem with that is it prevents users with complex type annotations from using cloudpickle (and consequently other tools like dask). I suppose there could be a try / except of some kind that only removes annotations if they fail to serialize but the inconsistency in behavior there seems like it might create more complications than it solves.