File tree 1 file changed +4
-10
lines changed
1 file changed +4
-10
lines changed Original file line number Diff line number Diff line change @@ -318,16 +318,10 @@ def untokenize(iterable):
318
318
with at least two elements, a token number and token value. If
319
319
only two tokens are passed, the resulting output is poor.
320
320
321
- Round-trip invariant for full input:
322
- Untokenized source will match input source exactly
323
-
324
- Round-trip invariant for limited input:
325
- # Output bytes will tokenize back to the input
326
- t1 = [tok[:2] for tok in tokenize(f.readline)]
327
- newcode = untokenize(t1)
328
- readline = BytesIO(newcode).readline
329
- t2 = [tok[:2] for tok in tokenize(readline)]
330
- assert t1 == t2
321
+ The result is guaranteed to tokenize back to match the input so
322
+ that the conversion is lossless and round-trips are assured.
323
+ The guarantee applies only to the token type and token string as
324
+ the spacing between tokens (column positions) may change.
331
325
"""
332
326
ut = Untokenizer ()
333
327
out = ut .untokenize (iterable )
You can’t perform that action at this time.
0 commit comments