[Core] Support configuration parsing plugin

[charlotte12l]

Purpose

In RLHF use cases, internal customized trainer and checkpoint may be used. In order to support these customization, we also need to inject custom vLLM config parser.
This PR makes it more extensible to allow plugin registration for config parsers.

See details in #23009. This PR aims to support
"Custom configuration plugin system" mentioned in proposal.

Note: this is the first step to split out configuration logics, we will continue work on decouple customized config parser with hf

Example:

from transformers import PretrainedConfig

from vllm.transformers_utils.config import (get_config_parser,
                                            register_config_parser)
from vllm.transformers_utils.config_parser_base import ConfigParserBase


@register_config_parser("custom_config_parser")
class CustomConfigParser(ConfigParserBase):

    def parse(self,
              model: Union[str, Path],
              trust_remote_code: bool,
              revision: Optional[str] = None,
              code_revision: Optional[str] = None,
              **kwargs) -> tuple[dict, PretrainedConfig]:
        raise NotImplementedError


type(get_config_parser("custom_config_parser"))

<class 'CustomConfigParser'>

Test Plan

python -m pytest tests/transformers_utils/test_config_parser_registry.py

Test Result

passed

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces an excellent plugin system for configuration parsers, enhancing vLLM's extensibility. The refactoring of existing logic into distinct parser classes is clean and well-executed. My feedback primarily addresses inconsistencies in terminology within the new API, particularly in logging and error messages. Correcting these instances, where 'model loader' appears to be a remnant from copy-pasting and should be 'config parser', will significantly improve the developer experience for this new feature.

[22quinn]

Looks good as a first step

[LysandreJik]

Thanks for the PR! Let us know if there's anything we can do on HF's side to make this work simpler; as it turns out to be an issue, we could simplify the configuration serialization on our side so that a parser isn't required.

Let us know if useful and we'd be happy to help.

[charlotte12l]

Thanks for the PR! Let us know if there's anything we can do on HF's side to make this work simpler; as it turns out to be an issue, we could simplify the configuration serialization on our side so that a parser isn't required.

Let us know if useful and we'd be happy to help.

@LysandreJik Thank you! I think a parser can't be avoided if a model is not using HF format?

[facebook-github-bot]

@charlotte12l has imported this pull request. If you are a Meta employee, you can view this in D82033854.

[DarkLight1337]

Please add this directory to "Async Engine, Inputs, Utils, Worker Test" so it's actually being run in CI

[charlotte12l]

Thanks! Added in #24615