Cannot pickle nested function that refers to itself

[mehrdadn]

The following code does not work, and may be a potential bug:

>>> def f():
    def g(): return g
    return g
>>> import cloudpickle; cloudpickle.dumps(f())
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    import cloudpickle; cloudpickle.dumps(f())
  File "cloudpickle.py", line 629, in dumps
    cp.dump(obj)
  File "cloudpickle.py", line 111, in dump
    raise pickle.PicklingError(msg)
PicklingError: Could not pickle object as excessively deep recursion required.

Is it possible to fix this, or is this a fundamental limitation of closures in Python that cannot be worked around?

[rgbkrk]

dill can certainly pickle this, though it needs to be loaded by dill as well.

In [1]: def f():
   ...:     def g():
   ...:         return g
   ...:     return g
   ...:

In [2]: import dill

In [3]: dill.dumps(f)
Out[3]: b'\x80\x03cdill.dill\n_create_function\nq\x00(cdill.dill\n_load_type\nq\x01X\x08\x00\x00\x00CodeTypeq\x02\x85q\x03Rq\x04(K\x00K\x00K\x00K\x03K\x03C\x16\x87\x00\x00f\x01\x00d\x01\x00d\x02\x00\x86\x00\x00\x89\x00\x00\x88\x00\x00Sq\x05Nh\x04(K\x00K\x00K\x00K\x01K\x13C\x04\x88\x00\x00Sq\x06N\x85q\x07))X\x1e\x00\x00\x00<ipython-input-1-8b97fd33a37c>q\x08X\x01\x00\x00\x00gq\tK\x02C\x02\x00\x01q\nh\t\x85q\x0b)tq\x0cRq\rX\x0c\x00\x00\x00f.<locals>.gq\x0e\x87q\x0f))h\x08X\x01\x00\x00\x00fq\x10K\x01C\x04\x00\x01\x12\x02q\x11)h\t\x85q\x12tq\x13Rq\x14c__builtin__\n__main__\nh\x10NN}q\x15tq\x16Rq\x17.'

In [4]: import pickle

In [5]: pickle.loads(dill.dumps(f))
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-5-99e7ef07cfcd> in <module>()
----> 1 pickle.loads(dill.dumps(f))

ImportError: No module named '__builtin__'

In [6]: dill.loads(dill.dumps(f))
Out[6]: <function __main__.f>

cloudpickle can only pickle that which can be depickled by the builtin pickle module.

[mehrdadn]

@rgbkrk I think you're pickling f, not f()? Try the latter.

[rgbkrk]

Oh whoops, I see that now. Let me try.

[rgbkrk]

Ah, k. That won't work in dill either.

¯_(ツ)_/¯

[mehrdadn]

The only practical solution I can think of is to use PyCell_Set in the Python C APIs, because the fundamental problem is that a reconstructed function's closure can't refer to itself.

[emlynoregan]

I have a full solution to this problem, which involves rewriting the function to remove recursions from the cell graph, and avoids the problem that a reconstructed function's closure can't refer to itself.

I've forked the repo here:
https://github.com/emlynoregan/yccloudpickle

In save_function_tuple(), here
https://github.com/emlynoregan/yccloudpickle/blob/master/yccloudpickle/yccloudpickle.py

I've added a call to the Combinator() function, to modify the function if necessary:
func = Combinator(func)

and I've put the code for Combinator() at the bottom.

The solution is non obvious! More information in this article:
Automatically Serialising Recursive Inner Functions in Python using the Y Combinator

I've also added a couple of tests that would throw the "excessively deep recursion" error in cloudpickle, but which yccloudpickle fixes.

I'd love someone to take a look; are you interested in incorporating this into cloudpickle proper?

Anyone who wants to experiment with it:

pip install yccloudpickle
...
import yccloudpickle
yccloudpickle.dumps(f)

etc. I've changed all the names in the package so you can have cloudpickle and yccloudpickle installed side-by-side.

[emlynoregan]

eg:

    def test_complex_recursive_inner_function(self):
        def makefactouter():
            def fact(n):
                return fact(n-1) * n if n > 0 else 1
            def fact2(n):
                return fact2(n-1) * n if n > 0 else 1
            def makefact():
                fact2(4)
                return fact
            
            return makefact

        fact = pickle_depickle(makefactouter())
        self.assertEqual(fact()(10), 3628800)

[llllllllll]

This is fixed in the latest release.

[emlynoregan]

Very cool!