SSH target connection issue

[douglas-raillard-arm]

Upon connecting to an SSH target, this exception is raised:

  File "testtarget.py", line 16, in <module>
    target = Target(kind='linux', host='myip', username='myuser', password='mypassword'),
  File "/home/dourai01/Data/Git/lisa/repo/lisa/target.py", line 270, in __init__
    self.target = self._init_target(
  File "/home/dourai01/Data/Git/lisa/repo/lisa/target.py", line 637, in _init_target
    target.connect(check_boot_completed=wait_boot, timeout=wait_boot_timeout)
  File "/home/dourai01/Data/Git/lisa/repo/external/devlib/devlib/target.py", line 267, in connect
    self._resolve_paths()
  File "/home/dourai01/Data/Git/lisa/repo/external/devlib/devlib/target.py", line 1074, in _resolve_paths
    self.working_directory = self.path.join(self.execute("pwd").strip(), 'devlib-target')
  File "/home/dourai01/Data/Git/lisa/repo/external/devlib/devlib/target.py", line 399, in execute
    return self.conn.execute(command, timeout=timeout,
  File "/home/dourai01/Data/Git/lisa/repo/external/devlib/devlib/utils/ssh.py", line 223, in execute
    raise TargetStableError(
devlib.exception.TargetStableError: cannot split reply (target misconfiguration?):

It seems that the output of any command is just \r\r rather than being the actual output, but the command is still being executed (checked with touch foobar).

Versions:
OpenSSH_8.1p1, OpenSSL 1.1.1d 10 Sep 2019
devlib commit: c212ef2

[marcbonnici]

What kind of target are you attempting to use? And are you able to SSH into the device directly?

[douglas-raillard-arm]

That's my host computer so x86-64. I am both able to SSH into my machine using ssh client, and using devlib as well since target.execute('touch /home/myuser/foobar') did work. The problem seems to be related to stdout capture in one way or another, since echoing something does not work.

[marcbonnici]

Right ok. And the output of \r\r is that what you get as the value of full_output? (https://github.com/ARM-software/devlib/blob/master/devlib/utils/ssh.py#L237)

[douglas-raillard-arm]

Yes, sorry if that was confusing

[marcbonnici]

Ok just double checking.
Have you seen this when connecting to other machine or just your host machine?

[douglas-raillard-arm]

I've not seen anywhere else apart from another host with same distribution (Archlinux), but OTH OpenSSH 8.1 was released on 2019-10-09 so only bleeding edge setups would be affected, assuming that's related to ssh client or server version. If you are interested, I could send you the full_output content so you can easily mock my case.

[marcbonnici]

Ah that's interesting and yes please send that over.
Also which version of pexpect do you have installed?

[douglas-raillard-arm]

Ok, so here it is:

Inside _execute_and_wait_for_prompt():

command == '(export LC_ALL=C && pwd); __devlib_ec=$?; echo; echo $__devlib_ec'

# execute the command blabla
self._sendline(command)

self.conn.before == b'\x1b[K\r\r\x1b[m\x1b[m\x1b[m\x1b[J'

output = process_backspaces(output)
output == '\r\r'

This makes the parent function (Target.execute()) choke with:

TargetStableError: cannot split reply (target misconfiguration?):
'

'

[douglas-raillard-arm]

Pexpect version:

>>> import pexpect
>>> pexpect.__version__
'4.7.0'

[marcbonnici]

Thanks for the information, I have created a VM with the latest arch live stick, OpenSSH 8.1p1-2 and pexpect 4.7 and I'm still not able to recreate the problem.
Are there any notable similarities between the 2 machines you're having this problem with which could be causing that response?

[douglas-raillard-arm]

that's strange, what was your Target() invocation ?
The only common point between them is that I'm administrating both, but they are otherwise not sharing anything except my emacs configuration ...

Do you end up with the same self.conn.before content when executing the dummy echo command ?

[marcbonnici]

This is my reproducer:

from devlib import LinuxTarget
conn_settings= {'host': host, 'username': username, 'password': password}
t = LinuxTarget(connection_settings=conn_settings, working_directory='/tmp')
t.execute('echo')
ipdb> self.conn.before                                                                                                                                                                                                                                                         
b'\r\n\r\n0\r\n'
out: '\r\n'

t.execute('touch /tmp/test')
ipdb> self.conn.before  
b'\r\n0\r\n'
out: ''

So the content I get in self.conn.before is what I would expect including the echoed return code. Could this be some sort of locale mismatch causing this?

[douglas-raillard-arm]

According to that page, my self.conn.before output contains some "erase line" sequences:
https://notes.burke.libbey.me/ansi-escape-codes/

\x1b[  => Control Sequence Introducer
K => Erase in Line
\r\r => carriage return x2
\x1b[  => Control Sequence Introducer
m => Select Graphics Rendition
\x1b[  => Control Sequence Introducer
m => Select Graphics Rendition
\x1b[  => Control Sequence Introducer
m => Select Graphics Rendition
\x1b[  => Control Sequence Introducer
J = Erase in Display

This looks like something that is resetting a terminal or something like that. That is not the output I get from reset or clear though.

[marcbonnici]

That would make sense however I'm not sure what would be causing that to happen. Would it be possible to place a break point before the send_line and step through the code to get a better picture of what's going on?

[douglas-raillard-arm]

I'll give that a go. I also searched for similar issues, and ended up in that issue:
DFE/MONK#108

As a maintainer of pexpect, I highly recommend using paramiko instead of pexpect for ssh :) pexpect's ssh support is from long ago where no native-ssh option existed, it only remains for compatibility.
With paramiko, remote programs will not detect you as a terminal and will not likely write terminal sequences.

That's a comment from 2014, so I assume that paramiko is nowadays in a relatively good shape, so even more relevant. If I find the time to implement it, would that sound like a good idea to you to either move SshConnection to paramiko, or introduce a new connection class ? This should eliminate issues like this one, and also the terminal size issue hanging over our head that will surely bite us again, since we allow unbounded amount of output from commands (and we even use that to dump files content with grep).

EDIT: it already took more time trying to understand the issue that it took to move SshConnection to paramiko (by far), so I propose we do ourselves the favor of closing this issue and spending time on polishing #450 instead :-)