[RFC]: Limit the use of envvars in vLLM

[zhuohan123]

Motivation.

vLLM's envvars are on the edge of getting out of control. There are many envvars should instead be configs. Like the attention backend, all2all kernel backend, and even there is a flag to control KV cache layout. envvars are evil because:

1. It is equivalent of using global variables everywhere in the code, which is a bad programming practice.
2. envvars have no advanced structure like, hierarchy, typechecks, etc.

In summary, I think envvars are the kind of thing that is very easy to add so people tend to add envvars as the shortest path to implement their feature, but this can quickly becomes unmanageable and makes the project very hard to use.

Proposed Change.

I think we should:

1. Spend some effort on reviewing the current envvars and move many of them to configs.
2. Have very strict bar on what can be an envvar and question every new envvars in vllm.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Related to #22914

[ProExpertProg]

I agree we should reduce our reliance on environment variables. I see a couple of reasonable uses for them:

• debugging probes and other properties we want to be able to control directly from the outside without changing code or access to the actual vllm command that's running, for example:
  • VLLM_LOGGING_LEVEL
  • VLLM_CACHE_DIR
  • VLLM_DEBUG_DUMP_PATH (in [torch.compile]: Add VLLM_DEBUG_DUMP_PATH environment variable #25651)
• temporary switches for unstable experimental features

For the ones that do stay around, we should improve their handling. I sent this in the #contributors channel a while ago, I'm happy to create a separate RFC.

I'm hoping to get some preliminary feedback on env variable restructuring. Instead of duplicating the type annotations and getter dictionary, I propose the following:

• Use the type-hinted declaration as the single source of truth, including defaults and custom parsing, and move them into envs/_variables.py
• Import those directly if TYPE_CHECKING, otherwise use them as a default store
• Allow defaults
• Standardize argument parsing for str/int/float/bool/Path/list[str] "trivial" types
• Add a pre-commit check to validate _variables.py and that they are never imported directly

Skeleton for `vllm/envs/__init__.py`:

from typing import TYPE_CHECKING, get_type_hints

import envs._variables
from envs._variables import _defaults as _env_defaults
from envs.utils import EnvFactory, parse_list, parse_path, unwrap_optional, is_type_with_args

if TYPE_CHECKING:
    # This way IDEs & type checkers get the declarations directly
    from envs._variables import *


_type_hints = get_type_hints(_variables)

def __getattr__(name):
    if name not in _env_defaults:
        raise AttributeError(f"module {__name__} has no attribute {name}")
    default = _env_defaults[name]

    if (value := os.getenv(name)) is None:
        # Not found, return default value
        if isinstance(default, EnvFactory):
            default = default.default_value

        # Call if lazy-initialized
        return default() if callable(default) else default

    if isinstance(default, EnvFactory):
        # If factory provided, use it to parse string
        return default.parse(value)

    # Automatic parsing of "trivial" data types
    # ...

Skeleton for `vllm/envs/_variables.py`:

VLLM_VAR1: Optional[str] = None
VLLM_VAR2: Optional[bool] = None
VLLM_VAR3: bool = False
VLLM_VAR4: int = 42
VLLM_VAR5: bool = True

VLLM_LOGGING_LEVEL: str = env_factory("INFO", lambda x: x.upper())
"""Logging level for vLLM. Default is "INFO"."""

# TODO might need to use default_factory here, AI tools want to "call" the var
VLLM_CACHE_ROOT: Path = lambda: parse_path(os.getenv("XDG_CACHE_HOME",
													 "~/.cache")) / "vllm")
"""Root directory for vLLM cache files. 
Default is `~/.cache/vllm` unless `XDG_CACHE_HOME` is set."""

VLLM_PLUGINS: Optional[list[str]] = None
"""A list of plugin names to load (comma-separated).
If not set, all plugins will be loaded.
If set to an empty string, no plugins will be loaded."""

# TODO could be improved
_defaults = {k: v for k, v in globals().items() if isinstance(k, str) and k.startswith("VLLM_")}

[zhuohan123]

@ProExpertProg Yeah I think that worth a separate RFC. I think my point here is to reduce the absolute the number of envvars, which is orthogonal on how to manage our current envvars.

[yyzxw]

+1 , I noticed that there is os.environ.get(ENV_NAME) in many codes, but the environment variables are not declared in envs.py and there is no documentation to prompt

[youkaichao]

I don’t think it’s too bad tbh. Like NCCL and cuda have lots of env vars, that’s the main way of turning on/off some knobs.

It can be used when vllm is integrated into other systems (e.g. verl), but users want to change some knobs not exposed.

[ProExpertProg]

I think my point here is to reduce the absolute the number of envvars, which is orthogonal on how to manage our current envvars.

I think if we handle them better (including actual documentation in the CLI) that might decrease the need for reducing their numbers :)

[elizabetht]

Hi @ProExpertProg @youkaichao @zhuohan123 @yyzxw - I am looking for a good first issue to work on. Would any of you provide guidance on what needs to happen for this issue?

[ProExpertProg]

@elizabetht I think inspecting the environemnt variables and seeing which ones could be eliminated, consolidated, or converted to config would be a great start :)

[ardecode]

@ProExpertProg , If no one is actively working on this, I can pick it up.

[ProExpertProg]

Please coordinate with @elizabetht !