push: userless ssh url doesn't work

[c-meier]

Bug Report

Description

When pushing to a ssh remote which url doesn't have a user (the user is set in ~/.ssh/config) I get the following error:

ERROR: unexpected error - No existing session

When checking with dvc push --verbose I get the following debug message:

DEBUG: Establishing ssh connection with 'myhost.example' through port '22' as user 'None'

Reproduce

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc remote add --default ssh-remote ssh://myhost.example/path/to/folder
5. Configure ~/.ssh/config with User and IdentityFile for Host myhost

Host myhost.example
    HostName myhost.example
    User myuser
    IdentityFile /path/to/my/private/key

6. dvc push

Expected

My data is pushed to the ssh remote

Environment information

• My computer username is different from the one used with ssh remote

Output of dvc doctor:

$ dvc doctor
DVC version: 2.4.1 (rpm)
---------------------------------
Platform: Python 3.8.10 on Linux-5.12.12-arch1-1-x86_64-with-glibc2.7
Supports: azure, gdrive, gs, webhdfs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/VolGroup00-lvolhome
Caches: local
Remotes: ssh, ssh
Workspace directory: ext4 on /dev/mapper/VolGroup00-lvolhome
Repo: dvc, git

Additional Information (if any):

$ dvc push --verbose
2021-06-25 15:16:19,871 DEBUG: Check for update is enabled.
2021-06-25 15:16:20,289 DEBUG: Preparing to upload data to 'ssh://myhost.example/path/to/folder'
2021-06-25 15:16:20,289 DEBUG: Preparing to collect status from ssh://myhost.example/path/to/folder
2021-06-25 15:16:20,289 DEBUG: Collecting information from local cache...
2021-06-25 15:16:20,290 DEBUG: Collecting information from remote cache...                          
2021-06-25 15:16:20,291 DEBUG: Matched '0' indexed hashes
2021-06-25 15:16:20,293 DEBUG: Establishing ssh connection with 'myhost.example' through port '22' as user 'None'
2021-06-25 15:16:20,937 ERROR: unexpected error - No existing session                               
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc/fs/pool.py", line 51, in get_connection
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dvc/main.py", line 55, in main
  File "dvc/command/base.py", line 50, in do_run
  File "dvc/command/data_sync.py", line 57, in run
  File "dvc/repo/__init__.py", line 51, in wrapper
  File "dvc/repo/push.py", line 44, in push
  File "dvc/data_cloud.py", line 79, in push
  File "dvc/remote/base.py", line 57, in wrapper
  File "dvc/remote/base.py", line 494, in push
  File "dvc/remote/base.py", line 351, in _process
  File "dvc/remote/base.py", line 195, in _status
  File "dvc/remote/base.py", line 145, in hashes_exist
  File "dvc/objects/db/ssh.py", line 73, in hashes_exist
  File "concurrent/futures/_base.py", line 619, in result_iterator
  File "concurrent/futures/_base.py", line 444, in result
  File "concurrent/futures/_base.py", line 389, in __get_result
  File "concurrent/futures/thread.py", line 57, in run
  File "dvc/objects/db/ssh.py", line 64, in exists_with_progress
  File "dvc/objects/db/ssh.py", line 31, in batch_exists
  File "contextlib.py", line 113, in __enter__
  File "dvc/fs/pool.py", line 11, in get_connection
  File "dvc/fs/pool.py", line 53, in get_connection
  File "dvc/fs/ssh/connection.py", line 59, in __init__
  File "paramiko/client.py", line 435, in connect
  File "paramiko/client.py", line 764, in _auth
  File "paramiko/client.py", line 740, in _auth
  File "paramiko/transport.py", line 1570, in auth_publickey
paramiko.ssh_exception.SSHException: No existing session
------------------------------------------------------------
2021-06-25 15:16:23,055 DEBUG: Version info for developers:
DVC version: 2.4.1 (rpm)
---------------------------------
Platform: Python 3.8.10 on Linux-5.12.12-arch1-1-x86_64-with-glibc2.7
Supports: azure, gdrive, gs, webhdfs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/VolGroup00-lvolhome
Caches: local
Remotes: ssh, ssh
Workspace directory: ext4 on /dev/mapper/VolGroup00-lvolhome
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-06-25 15:16:23,058 DEBUG: Analytics is enabled.
2021-06-25 15:16:23,297 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpr0kchyzn']'
2021-06-25 15:16:23,300 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpr0kchyzn']'

Workaround:

Overriding url in local config by adding the user (e.g. ssh://myuser@myhost.example/path/to/folder)

Possible lead:

One possible fix to the issue is:

diff --git a/dvc/fs/ssh/__init__.py b/dvc/fs/ssh/__init__.py
index 192e3ef3..622954f6 100644
--- a/dvc/fs/ssh/__init__.py
+++ b/dvc/fs/ssh/__init__.py
@@ -126,9 +126,9 @@ class SSHFileSystem(BaseFileSystem):  # pylint:disable=abstract-method
 
         return get_connection(
             SSHConnection,
-            path_info.host,
-            username=path_info.user,
-            port=path_info.port,
+            self.host,
+            username=self.user,
+            port=self.port,
             key_filename=self.keyfile,
             timeout=self.timeout,
             password=self.password,

However I don't know if it breaks other actions.

[pmrowla]

The SSH backend used in DVC (paramiko) won't read per-host User and IdentityFile from your OpenSSH (~/.ssh) config.

As you noted, the solution here is to just set the username when you configure your remote. You can either set it in the URL, or using in the user config field for the DVC remote (see the Click for SSH section in: https://dvc.org/doc/command-reference/remote/modify#available-parameters-per-storage-type)

We are in the process of switching SSH backends to asyncssh (#6064), which afaik is better about reading OpenSSH configs than paramiko, so it's possible that this use case will also be handled without needing to explicitly set your username after that switch is made.

But either way, for DVC SSH remotes you should just set the appropriate user in the DVC remote config.

[efiop]

@isidentical Btw, does asyncssh handle that? Just wondering.

[c-meier]

@pmrowla I tried setting user with:

$ dvc remote modify --local ssh-remote user myuser

But it has the same issue.

[c-meier]

The SSH backend used in DVC (paramiko) won't read per-host User and IdentityFile from your OpenSSH (~/.ssh) config.

DVC reads the OpenSSH config here. And it works. The issue is that you don't use the username that you retrieved when creating the ssh connection here

[pmrowla]

Looks like it's related to the changes in #6059

Previously, fs.path_info was populated using self.user, but that was lost when path_info was removed from fs/remote into the ODB classes

[Erotemic]

I'm also running into this issue.

I verified in dvc/remote/base.py in Remote.__init__ the **config does indeed contain the "user" field, but in dvc/fs/ssh/connection.py in SSHConnection.__Init__, the **kwargs does not contain "username".

I'm not sure if the difference between "user" and "username" is important or not.

In dvc/fs/ssh/__init__.py would a solution being changing:

    def ssh(self, path_info):
        self.ensure_credentials()

        from .connection import SSHConnection

        return get_connection(
            SSHConnection,
            path_info.host,
            username=path_info.user,
            port=path_info.port,
            key_filename=self.keyfile,
            timeout=self.timeout,
            password=self.password,
            gss_auth=self.gss_auth,
            sock=self.sock,
            allow_agent=self.allow_agent,
        )

to

    def ssh(self, path_info):
        self.ensure_credentials()

        from .connection import SSHConnection

        return get_connection(
            SSHConnection,
            path_info.host,
            username=self.user,
            port=path_info.port,
            key_filename=self.keyfile,
            timeout=self.timeout,
            password=self.password,
            gss_auth=self.gss_auth,
            sock=self.sock,
            allow_agent=self.allow_agent,
        )

Basically username=path_info.user, -> username=self.user,. The self.user variable seems to have the correct information in it.

[efiop]

@pmrowla @Erotemic Thanks for the research, guys! Indeed, it is a regression from #6059 🙁 I'll send a fix shortly while we are waiting for https://github.com/iterative/sshfs , which will remove this code anyway in the future.