PEP 750: TemplateIter is not pickleable

[sobolevn]

Bug report

Right now you can't pickle string.templatelib.TemplateIter objects. I missed it during my pickle PR.

Demo:

>>> import pickle
>>> pickle.dumps(iter('abc'))
b'\x80\x05\x95"\x00\x00\x00\x00\x00\x00\x00\x8c\x08builtins\x94\x8c\x04iter\x94\x93\x94\x8c\x03abc\x94\x85\x94R\x94K\x00b.'

>>> pickle.dumps(iter(t'abc'))
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    pickle.dumps(iter(t'abc'))
    ~~~~~~~~~~~~^^^^^^^^^^^^^^
TypeError: cannot pickle 'string.templatelib.TemplateIter' object

I think that it should be fixed. I will send a PR with code and tests soon.

[JelleZijlstra]

Do we need this? Compare #101588 where we deprecated support for pickling itertools iterators because it was inefficient and bug-prone.

[sobolevn]

In my opinion, yes, because __iter__ is an essential part of Template API. And str_iterator is also pickelable.

But, I am open to other opinions.

[rhettinger]

My opinion is that unless a strong use case emerges we should say "no".

Generators are the easiest and most common way to make iterators, but they cannot be pickled. That creates a strong presumption that iterators are not picklable in general. (That is why this is remarked as a possible feature rather than a bug).

We do have some holdovers like str_iterator from a time where we tried to pickle everything. It is still there because there is no point in creating pain for someone who did rely on it. But it is mostly unknown and hasn't proven itself to be essential. And since strings are immutable and self-contained, there isn't much that can go wrong, so it is mostly harmless.

One other thought. We should be careful with "lever arguments" in the form of "you did it once in tool A, so now you have to do it tool B". Sometimes the cost/benefit for A was that it was nearly zero cost (an easy case) and we speculated that there might be a benefit. Later when B comes along, it may be a more complex case (more complex state, mutable inputs, non-picklable inputs, etc) and have already learned that A never really paid-off.

Also, we should have a principle of parsimony and be reluctant to implement possible YAGNI features.

While __iter__ may be an important part of Template, that doesn't imply that the iterator needs to be pickeable. Consider the file and socket APIs where iteration is an important part of the API, but the iterators cannot be (and should not be) picklable.

I will stop short of saying that pickling iterators is always a bad idea. But it usually is. Mostly, nothing good with come from it. If it were really necessary, we would have long ago made generators pickeable (saving the gi_frame's locals and lineno).