Improve IDLE's "subprocess didn't make connection" message

[roseman]

BPO	25514
Nosy	@terryjreedy, @kbkaiser, @serwy, @roseman, @willingc
PRs	bpo-25514: Improve IDLE's connection refused message #2177 bpo-25514: Improve IDLE's connection refused message (#2177) #2178

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2015-10-29.20:47:20.690>
labels = ['easy', 'expert-IDLE', 'type-feature', '3.7', 'docs']
title = 'Improve IDLE\'s  "subprocess didn\'t make connection" message'
updated_at = <Date 2018-09-07.21:42:46.760>
user = 'https://github.com/roseman'

bugs.python.org fields:

activity = <Date 2018-09-07.21:42:46.760>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'IDLE']
creation = <Date 2015-10-29.20:47:20.690>
creator = 'markroseman'
dependencies = []
files = []
hgrepos = []
issue_num = 25514
keywords = ['easy']
message_count = 17.0
messages = ['253693', '253694', '253695', '253704', '253705', '253706', '253745', '253833', '255578', '258498', '295369', '295937', '295950', '295972', '295975', '295977', '324800']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'kbk', 'roger.serwy', 'markroseman', 'docs@python', 'willingc', 'lac']
pr_nums = ['2177', '2178']
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue25514'
versions = ['Python 3.6', 'Python 3.7']

[roseman]

When we create e.g. string.py that shadows a stdlib module needed by IDLE, it would be nice if a better error message could be shown, pointing to that cause.

Original message:

lac at smartwheels:~/junk$ echo "print ('hello there')" >string.py
lac at smartwheels:~/junk$ idle-python3.5
hello there
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.5/idlelib/run.py", line 10, in <module>
    from idlelib import CallTips
  File "/usr/lib/python3.5/idlelib/CallTips.py", line 16, in <module>
    from idlelib.HyperParser import HyperParser
  File "/usr/lib/python3.5/idlelib/HyperParser.py", line 14, in <module>
    _ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits + "_")
AttributeError: module 'string' has no attribute 'ascii_letters'

IDLE then produces a popup that says:

IDLE's subprocess didn't make connection. Either IDLE can't stat a subprocess por personal firewall software is blocking the connection. <ok>

--------

I think that life would be a whole lot easier for people if instead we got
a message:

Warning: local file /u/lac/junk/string.py shadows module named string in the
Standard Library

I think that it is python exec that would have to do this -- though of
course the popup could also warn about shadowing in general, instead of
sending people on wild goose chases over their firewalls.

----------------------

Laura, I think what you want should actually be more-or-less doable in IDLE.

The main routine that starts IDLE should be able to detect if it starts correctly (something unlikely to happen if a significant stdlib module is shadowed), watch for an attribute error of that form and try to determine if shadowing is the cause, and if so, reissue a saner error message.

The subprocess/firewall error is occurring because the ‘string’ problem in your example isn’t being hit right away so a few startup things already are happening. The point where we’re showing that error (as a result of a timeout) should be able to check as per the above that IDLE was able to start alright, and if not, change or ignore the timeout error.

There’ll probably be some cases (depending on exactly what gets shadowed) that may be difficult to get to work, but it should be able to handle most things.

See full thread starting at https://mail.python.org/pipermail/python-dev/2015-October/142061.html

[lac]

Note that the full path name of the file that is doing the shadow
is of important interest to teachers. We get machines will all sorts of garbage written on them -- the amount of hell caused by a misnamed turtle.py is hard to measure -- and anything as helps in the finding of the file as needs removing or renaming will be eternally welcome.

[lac]

s/machines will/machines with/

(I was tired.)

[terryjreedy]

Laura, I did not understand from your python-list post that you were complaining about shadowing disabling IDLE itself, as opposed to user code. That is partly because you followed up on Peter Otten complaining because IDLE tries to run user code the same way that python does.

The solution for IDLE itself is for IDLE to do what python does on startup, which is to ensure that the stdlib modules it needs are indeed imported from the stdlib. I presume that python only prepends '' to sys.path *after* it does its imports. Example:

>>> import sys; list(sys.modules)
[..., abc, ...]
C:\Users\Terry>echo "print('abc')" >abc  # verified with dir
C:\Users\Terry>python
>>> >>> import sys; sys.modules['abc'].__file__
'C:\\Programs\\Python35\\lib\\abc.py'

So IDLE should remove sys.path[0] from sys.path and, in the user process, restore it only after normal imports. Ignore -n single process mode for now.

Aside from this, the popup message needs to be changed, but better that it occur less often (by the fix above) or never (by switching to pipes?).

As for user code: I have in mind that there should be a doc how-to about tracebacks and exception messages, with a sections giving possible explanations and remedies for specific messages, such as the above. This I would make messages like the above a link to the how-to entry. In the meanwhile, we could consider a special case search of error message lines.

-I just noticed that this issue is the result of a pydev thread. Off to read that.

[terryjreedy]

"I think that removing '' from sys.path, so that IDLE can run, is better than a nicer warning that it cannot run. This, of course, requires that sys not be shadowed, so that sys.path can be accessed." So after importing sys, we should check sys.__file__, or do the manipulation in try: except: <post error message>. This will be part of a much more careful startup sequence.

[terryjreedy]

Either tkinter and tkinter.messagebox or subprocess must be imported to display a message when IDLE is started from an icon, and there is no console. With subprocess, '''python -i -c "print('Cannot start IDLE (or whatever)')"''' should work. If nothing else, this would be a way to display a message when import tkinter fails.

[lac]

Re: misunderstanding

Sorry Terry. From my end the great 4 day "weekend of mail.python.org needs to be rebuilt from scratch" happened precisely as I was about
to reply to your post. The reply to Peter Otten showed up later
as part of my mailer agent trying to send the thing every 4 hours for
a week. So confusion reigns.

[lac]

can I send a file

[lac]

webmaster got another one of these today.

[terryjreedy]

I have seen another report that is likely due to masking. I don't know why this has suddenly become seemingly much more common, but I have made fixing the problem a priority. See new issue bpo-26143.

I would like to narrow this issue to the user process, and broaden it to reporting all currently plausible causes of non-connection. I revised the title to say this.

I am looking at https://stackoverflow.com/questions/15888186/cant-run-python-via-idle-from-explorer-2013-idles-subprocess-didnt-make-c to see what causes people have found or think they have found.

1. Antivirus/firewall (4 people, linux iptables, Windows firewall, antivisus , so not a myth ;-)
2. Shadow files, as described above, and in the python directory (about as common, tkinter,py, subprocess.py).
3. Registry problem (some people can fix, but most should re-install after complete removal).
4. Zombie pythonw process (kill with Windows Task manager).
5. Permissions (run as admin, though should not be needed with proper install)
   ?. Multiple installed versions clash due to misconfiguration?

https://stackoverflow.com/questions/874757/python-idle-subprocess-error
Add:
?. example program that reads and writes (test)
6. Running after stopping run with ^C (temporary, I believe Restart Shell will fix) Call this a timing issue.
<<Note: I have experienced temporary connection failures myself, but have not written what I did>>
7. Modem rebot (rather obsolete answer for most of us).

There are plenty of miscellaneous possibilities to add even if masking, the most common reason, is fixed.

[terryjreedy]

bpo-26413 is about fixing shadowing problem.

Current thought for this issue: add a section to IDLE doc listing known possible causes (message above). Reference section in message, giving web link.

[willingc]

Based on Terry's latest message and recommendation, I've marked this as a Documentation issue and suitable for new contributors.

[terryjreedy]

Thanks for the reminder to get busy on this ;-). My short notes are insufficient for anyone else to write this.

[terryjreedy]

I tested (and tweaked) the visible box with this:
from idlelib.run import show_socket_error
show_socket_error(OSError(61, 'connection refused'), ('port', 8888))
URL was tested with cut-and-paste from run.py.

This should be turned into an htest (human view test). I did not do so now because I wanted get this into 3.6.2, and there is no indication of when the grace-period delay will end.

Message boxes do not allow copy to clipboard. A nice refinement would be a [View URL] button. I believe the code to do so is somewhere in IDLE. Or the F1 Python Docs code, which opens the Windows help version, should be adjustable to open to a specific topic.

[terryjreedy]

New changeset 188aedf by terryjreedy in branch 'master':
bpo-25514: Improve IDLE's connection refused message (bpo-2177)
188aedf

[terryjreedy]

New changeset 22d909f by terryjreedy in branch '3.6':
[3.6]bpo-25514: Improve IDLE's connection refused message (bpo-2177) (bpo-2178)
22d909f

[terryjreedy]

Module shadowing can affect either the gui process or the run process. bpo-34584 was the latter: debug_obj imports rpc imports socket imports selectors imports math, which was shadowed.

Pyshell starts run. If run starts OK but cannot connect to pyshell, it displays a 'subprocess cannot connect' message, which now refers to the new doc section, and exits. If pyshell does does get a connection, it currently displays

"Subprocess Startup Error",
"IDLE's subprocess didn't make connection.  Either IDLE can't "
"start a subprocess or personal firewall software is blocking "
"the connection.",

This also needs revision, with a reference to the doc section. And add 'math.py' to shadowed modules.

[StanFromIreland]

What exactly is there left to do? The error message was updated in your PR, no?

[terryjreedy]

The Sept 7, 2018 message was written after the merge and specifies a bit more to do. If you understand, try a PR with the code fix in pyshell.py, but the doc fix should wait until I fix help.py for PR #129727. Otherwise, leave it to me.

[StanFromIreland]

The 2018 message states that

This also needs revision, with a reference to the doc section.

However you updated the error message in 2019 in 8fac122 ?