python3 futures.as_completed timeout broken if future contains undefined reference

[dallasmarlow]

BPO	41694
Nosy	@MojoVampire, @dallasmarlow

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-09-02.23:33:27.392>
created_at = <Date 2020-09-02.16:15:41.278>
labels = ['interpreter-core', 'invalid', '3.7', '3.8']
title = 'python3 futures.as_completed timeout broken if future contains undefined reference'
updated_at = <Date 2020-09-02.23:33:27.386>
user = 'https://github.com/dallasmarlow'

bugs.python.org fields:

activity = <Date 2020-09-02.23:33:27.386>
actor = 'josh.r'
assignee = 'none'
closed = True
closed_date = <Date 2020-09-02.23:33:27.392>
closer = 'josh.r'
components = ['Interpreter Core']
creation = <Date 2020-09-02.16:15:41.278>
creator = 'dallasmarlow'
dependencies = []
files = []
hgrepos = []
issue_num = 41694
keywords = []
message_count = 5.0
messages = ['376250', '376252', '376254', '376257', '376267']
nosy_count = 2.0
nosy_names = ['josh.r', 'dallasmarlow']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue41694'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[dallasmarlow]

I recently noticed that timeouts do not work with futures.as_completed if a future contains an undefined variable or method. The following code runs for 30 seconds which I believe is a bug as I would expect the invalid print future to throw an exception immediately or the as_completed method throw an exception instead.

#########################

import concurrent.futures

with concurrent.futures.ThreadPoolExecutor() as ex:
	futures = [
		ex.submit(time.sleep, 30),
		ex.submit(print, a),
	]
	for future in concurrent.futures.as_completed(futures, timeout=1):
		_ = future.result()

[dallasmarlow]

After closer inspection it seems as though timeouts are not working as I would expect even when there are no undefined references. The following code runs for 30s when I would expect a timeout exception thrown after 1 second. as_completed eventually does throw a timeout exception, but only after the sleep future completes.

###############################

import concurrent.futures
import time

with concurrent.futures.ThreadPoolExecutor() as ex:
	futures = [
		ex.submit(time.sleep, 30),
		ex.submit(print, 'test'),
	]
	for future in concurrent.futures.as_completed(futures, timeout=1):
		_ = future.result()

[dallasmarlow]

I also verified that this issue persists with blocking futures other than sleep calls:

import concurrent.futures
import subprocess

with concurrent.futures.ThreadPoolExecutor() as ex:
	futures = [
		ex.submit(subprocess.run, ['sleep', '30']),
		ex.submit(print, 'test'),
	]
	for future in concurrent.futures.as_completed(futures, timeout=1):
		_ = future.result()

[dallasmarlow]

I apologize for all the messages, but the more I look into this issue the stranger it seems. The following code does the following:

• print the time
• execute 3 futures (2 w/ 30s sleeps)
• call as_completed
• catch the timeout exception
• print the time again
• then raise the timeout exception, but only after all futures complete.

the second time print executes after the as_completed timeout duration, but the exception was only raised after all futures completed. the exception message incorrectly states that it left 2/3 futures unfinished.

#########################

import concurrent.futures
import time

with concurrent.futures.ThreadPoolExecutor() as ex:
	print(time.time())
	futures = [
		ex.submit(time.sleep, 30),
		ex.submit(time.sleep, 30),
		ex.submit(print, 'test'),
	]
	try:
		for future in concurrent.futures.as_completed(
			futures, timeout=1):
			_ = future.result()
	except Exception as e:
		print(time.time())
		raise e

[MojoVampire]

The problem is a lot simpler than you're making it:

1. You submit a time.sleep(30) task. This begins running immediately
2. You try to submit another task, but a NameError is raised, bypassing the rest of the code (you never call as_completed, with or without a timeout)
3. The ThreadPoolExecutor's __exit__ is invoked, which implicitly invokes shutdown(wait=True). This does not return until the successfully submitted task (time.sleep(30)) finished.
4. At that point, the exception that was interrupted by with statement cleanup resumes bubbling

All of this is behaving exactly as documented, no bug is occurring.