Support poetry virtual environments

Motivation

Support for poetry environments is currently our highest upvoted feature request on github. It attempts to solve problems with all predating tools to manage Python packages: virtualenv, pip, pipenv, etc by being an all-in-one tool, and hence is becoming increasingly popular.

Scope

• Discover environments created using Poetry
• Identify if a provided custom interpreter path belongs to a Poetry environment
• Install dev tools using Poetry when related poetry environment is selected

Things we decided we need not address,

• Activate environment using poetry CLI: poetry shell command is similar to pipenv shell, which has its issues #4404. Besides, we don't need to activate these environments via poetry, as it manages venv/virtualenv environments which can directly be activated using the activate scripts.
• Run scripts using poetry CLI: poetry run command can be useful to run scripts if for some reason activating poetry environment doesn't work properly. But that is not the case observed with these environments for now, so we need not use it.

Design

Identifier

Global poetry environments

• Our best guess is to use the environment name pattern which Global poetry environments follow, as evident from the code here:

<sanitized_env_name>-<cwd_hash>-py<major>.<micro>

Implementation details behind <sanitized_env_name> and <cwd_hash> are too much to rely upon, so for our purposes the best we can do is verify if the pattern matches:

<anything>-<anything>-py<number>.<number>

• Should be a virtualenv/venv type environment.

If this doesn't work well, we can follow other alternatives mentioned in the spike.

Local poetry environments

• Environment folder is named .venv
• The directory alongside the .venv folder pyproject.toml which is valid and contains a poetry section.
  • We can directly run poetry check to verify this assumption.
  • If running command for every .venv-named folder turns out to be expensive, directly look for pyproject.toml file alongside the .venv folder. If found, read the file and see if contains the string [tool.poetry].

Discovery

Location for poetry environments is decided by the virtualenvs.path setting. It defaults to {cache-dir}/virtualenvs where {cache-dir} is another setting with its own defaults. Poetry looks at the following entries in order to get a setting,

• Environment variable: Every setting key has an associated environment variable which begins with POETRY_ (more). For eg. for the virtualenvs.path setting the environment variable to look into is POETRY_VIRTUALENVS_PATH.
• Local config: Stored in poetry.toml which is located besides pyproject.toml.
• Global config: Stored in these locations, in a file named config.toml. Parse file using npm package toml to get the setting value.
• Default value

We have two locators to search for all these locations for,

• Global poetry locator:

  • Env variable POETRY_VIRTUALENVS_PATH
  • Global virtualenvs.path setting

  Now we look at {cache-dir}/virtualenvs, possible {cache-dir} to look into:

  • Env variable POETRY_CACHE_DIR
  • Global cache-dir setting
  • Default setting value based on OS

• Workspace poetry locator

  • Local virtualenvs.path setting
  • Local cache-dir setting
  • Looks into .venv folder if it exists

Use identifier, pyvenv.cfg files and other utils to get type, version and other details.

API: Each locator extends FSWatchingLocator for file watching, and uses the existing ILocator interface.

Installer

API: It uses the existing IModuleInstaller interface which exposes two methods:

• isSupported: Returns whether we can install dev tools using poetry for a particular resource. If the selected environment for the resource is the active poetry environment which was created for the resource, return true, otherwise return false. Steps,

  • Get the associated poetry environment for the resource using poetry env info -p command.
  • Verify if it's the same as the selected environment for the resource.

  Running command here is not expensive as we already have the extension activated, and this only happens after user-consent. But we can use other alternatives if for some reason this doesn't work out.

• installModule: Use poetry add --dev <moduleName> to install dev tools. Add --allow-prereleases for packages for which released version is not available yet. (Example, black)

Note that users can specify the poetry executable to use using the python.poetryPath setting.

Special Considerations and possible poetry asks

We're depending on certain internal implementation details of poetry where appropriate poetry CLI is not available:

• The way global environment directories are named: <anything>-<anything>-py<number>.<number>.

  Possible resolution: Pipenv environments have a .project file, using which we can locate the project for which the current environment is created. If we have a similar feature for poetry we need not rely on the internal env naming pattern.

• For every OS, location and name of

  • global config file: config.toml.
  • local config file: poetry.toml.

  Possible resolution: Current poetry config commands only allow us to get the final value of the setting. Having commands to query for local and global settings separately would help here.

• Default location value of cache-dir setting for each OS.

Telemetry and Experimentation

Except for poetry check, we do not run additional commands for discovery. So it should be just like locators and doesn't require specific telemetry. Success of the feature can be measured along with other locators using the pythonDiscoveryModule experiment.

Testing Strategy

For discovery, unit tests uses real files to mock the file system, set the environment variables, and stubs shell execution. The corresponding ILocator APIs are then called to verify if we're getting all the environments. For installation, we have unit tests to stub shell execution, active interpreter and verify the implementation.