Discrepancy between context and inferred type for list display in context containing complicated unions

[LivInTheLookingGlass]

I have a rather large type alias to figure out what the msgpack library can serialize.

from typing import (cast, Any, Callable, Dict, Iterable, Iterator, List, NamedTuple, Sequence, Tuple, Union)

_MsgPackable_ = Union[None, bool, int, float, str, bytes]
_MsgPackable = Union[_MsgPackable_, List[_MsgPackable_], Tuple[_MsgPackable_, ...], Dict[str, _MsgPackable_]]
MsgPackable = Union[_MsgPackable, List[_MsgPackable], Tuple[_MsgPackable, ...], Dict[str, _MsgPackable]]


def test_method():
    #type: () -> Iterator[MsgPackable]
    return ((x, x) for x in range(30))

However, the above generates an error from your library. Yet your documentation suggests that something like this should work.

I get that this is a fairly complex usage, but I'm unsure how else to do this sort of thing. Can you help?

[LivInTheLookingGlass]

I would note that this only seems to happen when I feed it to List or Iterator.

So the following yields no error:

from typing import (cast, Any, Callable, Dict, Iterable, Iterator, List, NamedTuple, Sequence, Tuple, Union)

_MsgPackable_ = Union[None, bool, int, float, str, bytes]
_MsgPackable = Union[_MsgPackable_, List[_MsgPackable_], Tuple[_MsgPackable_, ...], Dict[str, _MsgPackable_]]
MsgPackable = Union[_MsgPackable, List[_MsgPackable], Tuple[_MsgPackable, ...], Dict[str, _MsgPackable]]


def test_method():
    #type: () -> MsgPackable
    return 30

[ilevkivskyi]

I don't think this is a problem with type aliases (I expanded all aliases and still get the same error), this is rather a problem with type inference. In the error message

 Incompatible return value type (got Iterator[union type (8 items)], expected Iterator[union type (4 items)])

The union type (8 items) looks very strange. Outside of the function, the type of the value in return statement is (correctly) inferred as Iterator[Tuple[builtins.int*, builtins.int*]]

[ilevkivskyi]

My first guess is that it is some weird interaction between simplified and non-simplified unions.

[gvanrossum]

Hm, when I paste in the example from the initial comment I get a different error:

__tmp__.py:10: error: Incompatible return value type (got Iterator[Tuple[int, int]], expected union type (4 items))

That seems reasonable (due to invariance of List/Dict, proably). What mypy versions are you using?

[ilevkivskyi]

I just checked out from master and installed and I get this

mypy tstbug.py
tstbug.py:10: error: Incompatible return value type (got Iterator[union type (8 items)], expected Iterator[union type (4 items)])

and this

mypy --py2 tstbug.py
tstbug.py:10: error: Incompatible return value type (got Iterator[union type (7 items)], expected Iterator[union type (4 items)])

For the code in original comment.
EDIT: I have got some old typeshed for Python 2, here is the output after updating it.

[ilevkivskyi]

Something very strange is going on...

[gvanrossum]

Sorry, this was an operator error on my part. I get the same error as Ivan. With some limits bumped 10x in messages.py (in format(), 150 -> 1500 and 400 -> 4000) I get this long error: ``` tstbug.py:10: error: Incompatible return value type (got Iterator[Union[float, str, bytes, List[Union[float, str, bytes]], Dict[str, Union[float, str, bytes]], List[Union[float, str, bytes, List[Union[float, str, bytes]], Tuple[Union[float, str, bytes], ...], Dict[str, Union[float, str, bytes]]]], Tuple[Union[float, str, bytes, List[Union[float, str, bytes]], Tuple[Union[float, str, bytes], ...], Dict[str, Union[float, str, bytes]]], ...], Dict[str, Union[float, str, bytes, List[Union[float, str, bytes]], Tuple[Union[float, str, bytes], ...], Dict[str, Union[float, str, bytes]]]]]], expected Iterator[Union[Union[Union[bool, int, float, str, bytes], List[Union[bool, int, float, str, bytes]], Tuple[Union[bool, int, float, str, bytes], ...], Dict[str, Union[bool, int, float, str, bytes]]], List[Union[Union[bool, int, float, str, bytes], List[Union[bool, int, float, str, bytes]], Tuple[Union[bool, int, float, str, bytes], ...], Dict[str, Union[bool, int, float, str, bytes]]]], Tuple[Union[Union[bool, int, float, str, bytes], List[Union[bool, int, float, str, bytes]], Tuple[Union[bool, int, float, str, bytes], ...], Dict[str, Union[bool, int, float, str, bytes]]], ...], Dict[str, Union[Union[bool, int, float, str, bytes], List[Union[bool, int, float, str, bytes]], Tuple[Union[bool, int, float, str, bytes], ...], Dict[str, Union[bool, int, float, str, bytes]]]]]]) ```

[LivInTheLookingGlass]

@gvanrossum I was using what I thought was the latest on pypi. I didn't realize you'd change the name to mypy (not mypy-lang). It looks like @ilevkivskyi is confirming this is on master as well, though, and pulling the latest from pypi I'm getting similar results.

test.py:10: error: Incompatible return value type (got Iterator[union type (8 items)], expected Iterator[union type (4 items)])

And for python 2 test

test.py:10: error: Incompatible return value type (got Iterator[union type (7 items)], expected Iterator[union type (4 items)])

[gvanrossum]

One of you should pairwise compare the two types from the error message I pasted.

[LivInTheLookingGlass]

This is it formatted nicer:

Iterator[
    Union[
        float,
        str,
        bytes,
        List[Union[float, str, bytes]],
        Dict[str, Union[float, str, bytes]],
        List[
            Union[
                float,
                str,
                bytes,
                List[Union[float, str, bytes]],
                Tuple[Union[float, str, bytes], ...],
                Dict[str, Union[float, str, bytes]]
            ]
        ],
        Tuple[
            Union[
                float,
                str,
                bytes,
                List[Union[float, str, bytes]],
                Tuple[Union[float, str, bytes], ...],
                Dict[str, Union[float, str, bytes]]
            ],
            ...
        ],
        Dict[
            str,
            Union[
                float,
                str,
                bytes,
                List[Union[float, str, bytes]],
                Tuple[Union[float, str, bytes], ...],
                Dict[str, Union[float, str, bytes]]
            ]
        ]
    ]
]


Iterator[
    Union[
        Union[
            Union[bool, int, float, str, bytes],
            List[Union[bool, int, float, str, bytes]],
            Tuple[Union[bool, int, float, str, bytes], ...],
            Dict[str, Union[bool, int, float, str, bytes]]
        ], 
        List[
            Union[
                Union[bool, int, float, str, bytes],
                List[Union[bool, int, float, str, bytes]],
                Tuple[Union[bool, int, float, str, bytes], ...],
                Dict[str, Union[bool, int, float, str, bytes]]
            ]
        ],
        Tuple[
            Union[
                Union[bool, int, float, str, bytes],
                List[Union[bool, int, float, str, bytes]],
                Tuple[Union[bool, int, float, str, bytes], ...],
                Dict[str, Union[bool, int, float, str, bytes]]
            ],
            ...
        ],
        Dict[
            str, 
            Union[
                Union[bool, int, float, str, bytes],
                List[Union[bool, int, float, str, bytes]],
                Tuple[Union[bool, int, float, str, bytes], ...],
                Dict[str, Union[bool, int, float, str, bytes]]
            ]
        ]
    ]
]

The second one looks like it's closer to what I was after.

The first looks to me like it was expanded improperly. Several places where it should be recursively defined are not. It also seems to be missing one of the tuple definitions.

However, even when I replace the recursively-defined type with the second expanded one, it still produces errors:

from typing import (cast, Any, Callable, Dict, Iterable, Iterator, List, NamedTuple, Sequence, Tuple, Union)

MsgPackable = \
Union[
    Union[
        Union[bool, int, float, str, bytes],
        List[Union[bool, int, float, str, bytes]],
        Tuple[Union[bool, int, float, str, bytes], ...],
        Dict[str, Union[bool, int, float, str, bytes]]
    ],
    List[
        Union[
            Union[bool, int, float, str, bytes],
            List[Union[bool, int, float, str, bytes]],
            Tuple[Union[bool, int, float, str, bytes], ...],
            Dict[str, Union[bool, int, float, str, bytes]]
        ]
    ],
    Tuple[
        Union[
            Union[bool, int, float, str, bytes],
            List[Union[bool, int, float, str, bytes]],
            Tuple[Union[bool, int, float, str, bytes], ...],
            Dict[str, Union[bool, int, float, str, bytes]]
        ],
        ...
    ],
    Dict[
        str,
        Union[
            Union[bool, int, float, str, bytes],
            List[Union[bool, int, float, str, bytes]],
            Tuple[Union[bool, int, float, str, bytes], ...],
            Dict[str, Union[bool, int, float, str, bytes]]
        ]
    ]
]

def test_method():
    #type: () -> Iterator[MsgPackable]
    return ((x, x) for x in range(30))

Mutating it in various ways doesn't seem to help. For instance, I tried flattening it a bit, but nothing really changed. Hope that's helpful.

[gvanrossum]

Hm, you're trying to approximate a recursive alias. That doesn't work. There's obviously a real bug here, but it's going to be tough to track it down. One thing that looks suspicious is that there are two different ways to map the value `(x, x)` (assuming the intended type is `Tuple[int, ...]`) into the expanded alias -- it matches the Tuple alternative in the first sub-union, but also the Tuple alternative at the outer level. The intention is that `Union[Union[X, Y], Z]` is equivalent to `Union[X, Y, Z]`, but it appears that mypy doesn't always simplify unions completely (as Ivan mentioned early in the thread). Maybe we can simplify the test case by removing the Dict and List branches from the unions? And probably some of the primitive types too (maybe just keeping int and str).

[LivInTheLookingGlass]

Approximate is a key word there. I figured it was a pretty safe assumption that nobody would use the library more than a few layers deep.

In any case, that ends up with:

from typing import (Iterator, Tuple, Union)

_MsgPackable_ = Union[int, str]
_MsgPackable = Union[_MsgPackable_, Tuple[_MsgPackable_, ...]]
MsgPackable = Union[_MsgPackable, Tuple[_MsgPackable, ...]]


def test_method():
    #type: () -> Iterator[MsgPackable]
    return ((x, x) for x in range(30))

Which yields no errors, on my machine at least. Adding back the other primitives also seems to yield no errors.

This is the smallest case I found which yields an error:

from typing import (Iterator, List, Tuple, Union)

_MsgPackable_ = Union[int, float, bool]
_MsgPackable = Union[_MsgPackable_, Tuple[_MsgPackable_, ...]]
MsgPackable = Union[_MsgPackable, List[_MsgPackable], Tuple[_MsgPackable, ...]]


def test_method():
    #type: () -> Iterator[MsgPackable]
    return ((x, x) for x in range(30))

[gvanrossum]

Thanks! I boiled it down some more, to these two lines:

from typing import List, Union
a = ['']  # type: List[Union[str, List[Union[int, float, bool]]]]

We now get (I've reflowed the message a bit):

tst.py:2: error: Incompatible types in assignment
(expression has type List[Union[str, List[float]]],
variable has type List[Union[str, List[Union[int, float, bool]]]])

What seems to be happening here is that the expression [''] is analyzed in the context of the expected type, and the analysis infers the type List[Union[str, List[float]]]. But that's not compatible with the expected type: invariance means that the item types of the two lists should be identical, whereas here the inferred type contains a union with only float, and the expected type has a union of int, float and bool.

The reason for this must be that the inference handles type promotions (esp. from int to float) differently than type comparisons handles them. And it seems that the presence of bool contributes.

That's as far as I expect to get tonight. Interesting problem!

[JukkaL]

It looks like that at least part of of the problem is that mypy doesn't recognize that Union[int, float, bool] is semantically equivalent to float.

[ilevkivskyi]

OK, it looks like this was fixed by #3086