Correct return type of sum() builtin

[teepark]

sum([]) always returns the integer 0.

[JelleZijlstra]

Not sure about this; we tend to avoid union return types.

[teepark]

Some context on why I'm changing this: I had code that would sum a list of decimal.Decimal, and then I attempted to .quantize() the returned value, which resulted in a production bug when the list was empty for the first time.

mypy is in a position to know that the return value could possibly be an integer. Can you clarify why union return types are avoided? It's certainly more accurate in this case.

[JukkaL]

One problem with union in this context is that some code may be careful to never pass an empty iterable to sum() and thus it will never return an int. In this case a union return type could generate false positives. I don't have a strong opinion either way, though.

[teepark]

Thanks for the response. That makes sense and I imagine this change would be more painful for people being careful to never pass an empty list.

But ultimately that's a dependency only on runtime data, which mypy isn't able to take into consideration. It's not really any different from a function that returns different types in the branches of an if statement -- mypy can only accurately model it as a union, even though code using the function might have more detailed knowledge based on how it crafted the arguments.

[gvanrossum]

It's similar to pow(), where usually pow(x, y) with int arguments is expected to return an int, but if y is negative it will return a float. We solved that with a plugin because usually y is a compile-time constant expression. But before it just returned Any.

Maybe Any is a better return type for sum()? I'm not sure -- it seems there's no good answer here, and a plugin won't help because a plugin can only tell whether the list is empty at compile time, which is no use for a typical sum() call.

One obvious improvement is to create two overloads, sum(x) and sum(x, y). The latter does not return an int, it returns y.

[matthiaskramm]

For pytype, we would write

@overload
def sum(iterable: Iterable[nothing]) -> int: ...
@overload
def sum(iterable: Iterable[_T], start: _T = ...) -> _T: ...

(See e.g. the pytype definition of "reduce", https://github.com/google/pytype/blob/master/pytype/pytd/builtins/__builtin__.pytd#L92)

However, neither mypy nor PEP 484 has a concept of "nothing" so this doesn't work here.
(It also doesn't work because mypy doesn't stop, after the first matching @overload)

[gvanrossum]

But how would you know whether a particular variable containing a list of integers can be empty or not? Otherwise this won't be very useful -- unlike pow(), sum() is rarely called with an empty list literal -- it's called with a variable that may or may not be guaranteed to be non-empty.

[teepark]

@gvanrossum I agree the overload is a definite improvement.

As a starting point, here's what I believe we'd need to accurately model all behavior of sum():

@overload
def mysum(iterable: Iterable[_T]) -> Union[_T, int]: ...

@overload
def mysum(iterable: Iterable[_T], start:_S) -> Union[_T, _S]: ...

In the two argument version we need a second type variable and the union because there's nothing outright requiring that the second argument be the same type as the items in the iterable, and promotion can happen either way:

sum([1, 2, 3], 4.5)  # float (start)
sum([1.0, 2.0, 3.5], 4)  # float (iterable)

Do we want to model all that behavior? I would have found it helpful for mypy to notify me of the problem with my blind sum(my_decimals).quantize(precision), but if we want to optimize for the (probably?) common case where the iterable can't possibly be empty that makes sense too.

[gvanrossum]

Yes, this sounds right. We do this for e.g. dict.get() as well.