AttributeError: Can't pickle local object 'augment_visit.<locals>.augment_func'

[prokher]

Dear Colleagues,

enabling this plugin on the DjangoChannelsGraphqlWs gives me the traceback below. It is worth mentioning that it has worked fine some months ago. Unfortunately I could not track what particular change made this happen, probably Python version upgrade or PyLInt version upgrade, I am not sure. Anyway, currently I cannot make it running and had to disable pylint-django otherwise PyLint fails.

Traceback (most recent call last):
  File "/Volumes/Dev/DjangoChannelsGraphqlWs/.venv/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/Volumes/Dev/DjangoChannelsGraphqlWs/.venv/lib/python3.8/site-packages/pylint/__init__.py", line 22, in run_pylint
    PylintRun(sys.argv[1:])
  File "/Volumes/Dev/DjangoChannelsGraphqlWs/.venv/lib/python3.8/site-packages/pylint/lint/run.py", line 344, in __init__
    linter.check(args)
  File "/Volumes/Dev/DjangoChannelsGraphqlWs/.venv/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 874, in check
    check_parallel(
  File "/Volumes/Dev/DjangoChannelsGraphqlWs/.venv/lib/python3.8/site-packages/pylint/lint/check_parallel.py", line 92, in check_parallel
    with multiprocessing.Pool(jobs, initializer=initializer, initargs=[linter]) as pool:
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'augment_visit.<locals>.augment_func'

Here is pip freeze output:

pip freeze 
WARNING: Could not find setup.py for directory /Volumes/Dev/DjangoChannelsGraphqlWs (tried all parent directories)
aiohttp==3.6.2
aioredis==1.3.1
aniso8601==7.0.0
apipkg==1.5
appdirs==1.4.4
appnope==0.1.0
asgiref==3.2.10
astroid==2.4.2
async-timeout==3.0.1
attrs==19.3.0
autobahn==20.7.1
Automat==20.2.0
backcall==0.2.0
black==19.10b0
cffi==1.14.0
cfgv==3.0.0
channels==2.4.0
channels-redis==3.0.1
chardet==3.0.4
click==7.1.2
constantly==15.1.0
coverage==5.2.1
cryptography==3.0
daphne==2.5.0
decorator==4.4.2
distlib==0.3.1
Django==3.0.8
-e git+https://gitlab.sd.datadvance.net/sd/pseven/DjangoChannelsGraphqlWs.git@ce8d5936a4dda3489c3d8951380f24745264106a#egg=django_channels_graphql_ws
execnet==1.7.1
filelock==3.0.12
graphene==2.1.8
graphene-django==2.12.1
graphql-core==2.3.2
graphql-relay==2.0.1
hiredis==1.1.0
hyperlink==19.0.0
identify==1.4.25
idna==2.10
incremental==17.5.0
ipython==7.16.1
ipython-genutils==0.2.0
isort==4.3.21
jedi==0.17.2
lazy-object-proxy==1.4.3
mccabe==0.6.1
more-itertools==8.4.0
msgpack==1.0.0
multidict==4.7.6
mypy==0.782
mypy-extensions==0.4.3
nodeenv==1.4.0
packaging==20.4
parso==0.7.1
pathspec==0.8.0
pexpect==4.8.0
pickleshare==0.7.5
pluggy==0.13.1
plumbum==1.6.9
pre-commit==2.1.1
promise==2.3
prompt-toolkit==3.0.3
ptyprocess==0.6.0
py==1.9.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pydocstyle==5.0.2
Pygments==2.6.1
PyHamcrest==2.0.2
pylint==2.5.3
pylint-django==2.2.0
pylint-plugin-utils==0.6
pylint-quotes==0.2.1
pyOpenSSL==19.1.0
pyparsing==2.4.7
pytest==5.4.3
pytest-asyncio==0.14.0
pytest-cov==2.10.0
pytest-django==3.9.0
pytest-forked==1.2.0
pytest-pythonpath==0.7.3
pytest-xdist==1.33.0
pytz==2020.1
PyYAML==5.3.1
regex==2020.7.14
Rx==1.6.1
service-identity==18.1.0
singledispatch==3.4.0.3
six==1.15.0
snowballstemmer==2.0.0
sqlparse==0.3.1
toml==0.10.1
tox==3.18.0
traitlets==4.3.3
Twisted==20.3.0
txaio==20.4.1
typed-ast==1.4.1
typing-extensions==3.7.4.2
Unidecode==1.1.1
virtualenv==20.0.28
wcwidth==0.2.5
wrapt==1.12.1
yarl==1.4.2
zope.interface==5.1.0

[ivasic]

Setting jobs=1 in .pylintrc seems to help, as a temporary workaround.

[tizbecool]

Looks like it's coming from https://github.com/PyCQA/pylint-plugin-utils/blob/master/pylint_plugin_utils/__init__.py#L52

the inner function augment_func() is not pickable and with python3.8 on macOS, spawn mode is now enable by default instead of fork previously for multiprocessing
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

[dineshtrivedi]

Hi, @tizbecool any suggestions on how to go forward with this fix?
I wouldn't mind trying it, but I am not sure yet how to start

[dineshtrivedi]

I started working on this PR - pylint-dev/pylint-plugin-utils#21
I would really appreciate any feedback

[carlio]

pylint-plugin-utils version 0.7 just got pushed to PyPI which has the fix that @dineshtrivedi made in it - so that should hopefully fix this issue.

I'll close it because I am an optimist, but please re-open if it's not fixed or if there are other side-effects that turn up :-)

[dineshtrivedi]

Thank you @carlio
It seems that the release fixed the issue partially. There are two other problems

The first:

Traceback (most recent call last):
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/__init__.py", line 24, in run_pylint
    PylintRun(sys.argv[1:])
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/run.py", line 398, in __init__
    linter.check(args)
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1000, in check
    check_parallel(
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/parallel.py", line 113, in check_parallel
    pool = multiprocessing.Pool(  # pylint: disable=consider-using-with
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'generic_is_view_attribute.<locals>.is_attribute'

After fixing it, there is a second one:

Traceback (most recent call last):
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/__init__.py", line 24, in run_pylint
    PylintRun(sys.argv[1:])
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/run.py", line 398, in __init__
    linter.check(args)
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1000, in check
    check_parallel(
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/parallel.py", line 113, in check_parallel
    pool = multiprocessing.Pool(  # pylint: disable=consider-using-with
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'is_class.<locals>.<lambda>'

[Pierre-Sassoulas]

In pylint we had to add a dependency to dill to be able to pickle in threads, see pylint-dev/pylint#5609

[dineshtrivedi]

Hello @Pierre-Sassoulas

Thank you for your response. So in your point of view, there is no value in accepting the PR I made as it will be fixed once pylint 2.13.0 is released? I am fine with it if that is the case.

[Pierre-Sassoulas]

So in your point of view, there is no value in accepting the PR I made as it will be fixed once pylint 2.13.0 is released?

I'm not saying that, just sharing the way we handled a similar problem. Seeing your comment before mine it seemed the problem was only partially resolved. Maybe the link provided in pylint-dev/pylint#5609 would be helpful ?

Also now that you say it, I'm thinking that starting from pylint 2.13 using dill to make your life easier would make sense as you'll have it as a dependency through pylint anyway.

[carlio]

It does suggest that the pylint 2.13 would have fixed this problem already, but I think that the change in the pickling on pylint-django (and pylint-plugin-utils) to use the class-based approach that @dineshtrivedi created is a good idea anyway. For backwards compatability if nothing else.

@dineshtrivedi I'll merge this in because I think it's forwards compatible with planned pylint changes.

@Pierre-Sassoulas dill sounds like it'd solve any other upstream issues but rather than rely on it, for what is a small change, might as well just be a bit more robust.

[dineshtrivedi]

Thank you @carlio

I have made one last change to include a unit test to make sure the plugin remains picklable after future changes. I thought could be a good idea. Maybe once pylint 2.13.0 is out we can consider using dill for the unit test instead of pickle.

The change:
https://github.com/PyCQA/pylint-django/pull/348/files#diff-0100e2ebad96c65eae259f46064a7f7488d6d3ee48ac911c805ed8534710f0a9R117