[Feature Request]: Upgrade or provide ETA on dill

[eddie-scio]

What would you like to happen?

2.41.0 has dill<0.3.2, which is now more than 2 years old. We're trying to upgrade one of our core ML libraries that we run on Beam, but we are blocked on a dill>=0.3.4 requirement there. I do see the note about dill incompat in setup.py, and I do see a rollback PR for the upgrade -- I'm wondering if there is any ETA or communication with dill that can be shared.

Issue Priority

Priority: 2

Issue Component

Component: sdk-py-core

[tvalentyn]

@ryanthompson591 could you provide your input here please?

[eddie-scio]

Hey @ryanthompson591 , just a friendly ping!

[ryanthompson591]

There are two efforts underway. The first was to upgrade our dill version, which got blocked by some failed tests.

See:
uqfoundation/dill#517
python/cpython#94245

In dill's defense it's a python error - that gets resolved in 3.10.

The second is to vendor dill, which there is some agreement to do.

Either would solve this problem.

As to the question of when, I am not yet sure. Can you give some context as to why allen NLP requires this version of dill? Maybe it can help us prioritize.

[tvalentyn]

In dill's defense it's a python error - that gets resolved in 3.10.

Is this fixed on dill master?

[ryanthompson591]

I just downloaded dill master, and tried this code out and it is fixed.

Dill added a workaround for the python error here:
https://github.com/uqfoundation/dill/blob/25a7e450ed76c7a0820834a3a91134476b1b8253/dill/_dill.py#L746d.

[eddie-scio]

That's awesome. So we just need dill to release 0.3.6 (https://github.com/uqfoundation/dill/milestones), and then release a new version of Beam that changes the dependency constraint (e.g.dill>=0.3.6)?

[eddie-scio]

Good news, dill just released 0.3.6 (https://pypi.org/project/dill/0.3.6/) two days ago, so in 2.42.1 we should be able to take care of this?

[ryanthompson591]

Thanks for this eddie. I will be looking into doing this import and testing it throughout our code bases. I was able to validate that the new version of dill did fix this problem.

[eddie-scio]

Thanks so much, Ryan! Really excited to get this working.