[Parser] Tuples are not parsed by the new parser

[akshanshbhatt]

Issues #507 and #620 give insights on how nested tuples and function arguments with tuple type are not parsed correctly. While working on these issues, I realized that even the basic tuples are not parsed by the new parser. At present, we can only parse assignment operation of a tuple type to a variable (example a = (1, 2)). All the basic tuple code examples mentioned below fail to get get parsed -

(1, 2)

(1, 2, 3,)

1, 2

1, 2,

This according to me is because of the fact that tuple is not a component of expr node, which ideally it should be. Including it in among the expr syntax rules would close both of these issues. For instance, I can generate ast for an arbitrary nested list because it's an expr member -

❯ cat examples/expr2.py
[[[1, 2], 3], [1, 2], [1, ], 1, ]
❯ lpy --show-ast --new-parser examples/expr2.py  # New Parser
(Module [(Expr (List [(List [(List [(ConstantInt 1 ()) (ConstantInt 2 ())] Load) (ConstantInt 3 ())] Load) (List [(ConstantInt 1 ()) (ConstantInt 2 ())] Load) (List [(ConstantInt 1 ())] Load) (ConstantInt 1 ())] Load))] [])
❯ lpy --show-ast examples/expr2.py  # Old Parser
(Module [(Expr (List [(List [(List [(ConstantInt 1 ()) (ConstantInt 2 ())] Load) (ConstantInt 3 ())] Load) (List [(ConstantInt 1 ()) (ConstantInt 2 ())] Load) (List [(ConstantInt 1 ())] Load) (ConstantInt 1 ())] Load))] [])

[akshanshbhatt]

Another reason for making tuple an extension of the expr type is that arithmetic operations would be valid.

print("%s%s%s" % (a, b, c))
(1, 2) * 3

❯ lpyast examples/expr2.py
(Module [(Expr (Call (Name print Load) [(BinOp (ConstantStr "%s%s%s" ()) Mod (Tuple [(Name a Load) (Name b Load) (Name c Load)] Load))] [])) (Expr (BinOp (Tuple [(ConstantInt 1 ()) (ConstantInt 2 ())] Load) Mult (ConstantInt 3 ())))] [])

[certik]

I remember @Thirumalai-Shaktivel and I fixing some of these in the new parser, and I guess there are more use cases. I know there are all kinds of issues. We just have to tackle these.

If needed, we can get some help from the tokenizer by making it stronger.

[Thirumalai-Shaktivel]

Most of the tuple syntaxes are parsed. I think the only left-out part is the tuple without parentheses.
Example:

1, 2, 
1, 2, 3

[certik]

Of these:

e = 1, 2
e = (1, 2)
(1, 2)
1, 2 = e
(1, 2) = e
1, 2

Everything works except the last line. So I think the last case can be handled with some special rule, that there can be not just an expression on a line but also an expression list.

[Thirumalai-Shaktivel]

Other examples:

yield x, y
assert_(np.isnan(div)), 'dt: %s, rem: %s' % (dt, rem)
obj = obj,
(left + size + right for size, (left, right) in zip(array.shape, pad_width))
for like_function in np.zeros_like, np.ones_like, np.empty_like:
    pass
for a, in t:
    pass

[Thirumalai-Shaktivel]

The last tuple issue known is:

with x, y, z as a, b:
    pass

which has been reported as a separate issue: #919