Infer optional type for list with None values

[JukkaL]

This is unexpected:

x = [1, None]
reveal_type(x)   # Revealed type is 'builtins.list[builtins.object*]'

I'd expect the list item type to to be Optional[int]. A join of a non-optional type with None should probably produce an optional type.

[gvanrossum]

Maybe, but in most cases mypy just sees Optional[x] as Union[x, None], and unification is loath to create unions. So we'd have to make an exception for unification with NoneType. (That's fine, I'm just not surprised that it doesn't already work this way.)

[JukkaL]

Yeah, I think it's reasonable to make an exception here.

[ddfisher]

Instead, I think we should be less loath to infer unions. If someone has [1, "foo", 3] I think they're more likely to expect it to have List[Union[int, str]] than List[object].

[JukkaL]

Inferring a union type for something like [1, 'foo'] is not very controversial, but it gets less clear if we consider all the possible cases where this might happen. Some examples below.

First, sometimes mypy can already infer a non-object join. It's not clear if this should take precedence over a union or not:

class Employee: pass
class Manager(Employee): pass
class Engineer(Employee): pass
class Designer(Employee): pass
workers = [Manager(), Engineer()]  # List[Employee] or List[Union[Manager, Engineer]]?
workers.append(Designer())  # Should this be okay?

Second, if some of the types involved aren't simple instance types, the union may become tricky to work with, and it can be difficult to predict what the union actually looks like. Some examples below:

def f(y: MyClass  = None) -> int: return 0
def g(x: int, z: int) -> int: return x + z

a = [f, g]   # Should the item type be a union of callables?
b = [f, int]  # Callable[[], int] or Union[something]?
c = [(1, 2), [1, 2]]  # List[Sequence[int]] or List[Union[Tuple[int, int], List[int]]]?
d = [MyClass, MyClass()]   # List[object] or Union[MyClass, Type[MyClass]]?
e = [ClassA, ClassB]  # Union[Type[ClassA], Type[ClassB]], Union[Type[ClassA, ClassB]], 
                # some callable, or 'type'?
x = [1]
y = ['']
f = [x, y]  # List[Union[List[int], List[str]]?

Third, it's not uncommon to have complex data structures involving nested lists, dictionaries, tuples and types/callables initialized at the top level of modules. If we'd infer unions aggressively, it's likely that for some of these we'd infer something silly looking like Dict[str, List[Union[Dict[str, Union[....]...]. This might even pass type checking, but when there is an error, mypy would print out this monster type and it would be difficult to figure out what it actually means. Complex error messages are one of the more common complaints about languages like Haskell and C++, and I wouldn't like to mypy to join this group :-)

Finally, an additional reason why mypy likes to fall back to List[object] etc. is that these are pretty 'safe' fallback types, even if they aren't what the programmer expects, since almost any operation on object will be rejected, so the bad types won't propagate very far and will be easily caught. If we are very aggressive about type inference, we can infer types that are technically correct but don't make much sense intuitively.

One idea would be to only infer "simple" union types such as Union[int, str] or Optional[int] but not union types involving callables, tuple types or generic types that only differ in values of type arguments. Also, perhaps instead of always falling back to List[object], require a type annotation as it's usually not the desired type (if we are inferring a variable definition). The exact rules would have to be much more involved, since there are various edge cases to consider. Some examples:

a = [1, '']   # List[Union[int, str]]
a = [1, None]  # List[Optional[int]]
b = [int, str]   # Require annotation, as this is tricky
x = [1]
y = ['']
d = [x, y]  # Require annotation

[ddfisher]

Those are good points. I don't think the problems are necessarily insurmountable, but they would definitely require more thought.

For now, this is fixed by #1999, which only affects lists containing None.