use-a-generator hint in sum method

[TiesWestendorp]

Current problem

Running pylint on the following snippet, I get a R1729: Use a generator hint for the all expression, but not for the sum. I suspect that there might be more occurrences where this happens.

all([True for i in range(1)])
sum([True for i in range(1)])

Desired solution

I would expect the same hint for the second line, with the sum method.

Additional context

No response

[Pierre-Sassoulas]

I think the original idea was performance improvement. sum need to use all the element all the time so the performance improvement of using the generator is less pronounced or even negative if the sequence is small. Could be a a consider-using-generator ? (See #3309)

[TiesWestendorp]

Ah, that actually makes a lot of sense. Hadn't given it much thought in hindsight. Still, I would expect there to be some overhead in the form of actually constructing the list, right? I'm not sure how it could be slower though, even for small sequences. That's a bit tangential though, I agree that it might be better as a consider-using-generator.

[Pierre-Sassoulas]

I would expect there to be some overhead in the form of actually constructing the list, right?

Yes I was not expecting that either, see this comment

[TiesWestendorp]

Huh, the given benchmark actually performs worse the for generator version for sum as well:

python -m timeit "some_list = list(range(1000));b = sum([x % 7 == 0 for x in some_list]);"
# 5000 loops, best of 5: 65.5 usec per loop
python -m timeit "some_list = list(range(1000));b = sum(x % 7 == 0 for x in some_list);"
# 5000 loops, best of 5: 75.6 usec per loop

[TiesWestendorp]

Given the benchmark result, I guess this can be closed.

[Pierre-Sassoulas]

This is not a one size fit all benchmark, it's just a datapoint showing that it's not always a performance increase (we can't use use-a-generator directly). There's probably a threshold where generators are better again (for really large list, or low ram ?). Digging into the issue to understand what will make generator better would be great but I think there's a lot of factor that we don't control so maybe it's best to just let pylint's user consider if they should do it or not (Does this code need optimization, does generator makes it better, on what hardwares, etc ?). So imo we could warn with consider-using-generator.

[Pierre-Sassoulas]

@ikraduya do you have an opinion on the subject ? :)

[Pierre-Sassoulas]

Closing as this was done in #6595