Skip to content

Allow defining a default base model in the lora syncer configuration #340

@ahg-g

Description

@ahg-g

What would you like to be added:

Allow defining a default base model in the lora syncer configuration

Why is this needed:

It is repetitive to list it in each list item in our case since we are mostly assuming a single base model per pool.

so instead of

apiVersion: v1
kind: ConfigMap
metadata:
  name: vllm-llama2-7b-adapters
data:
  configmap.yaml: |
      vLLMLoRAConfig:
        name: vllm-llama2-7b
        port: 8000
        ensureExist:
          models:
          - base-model: meta-llama/Llama-2-7b-hf
            id: tweet-summary-1
            source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm
          - base-model: meta-llama/Llama-2-7b-hf
             id: tweet-summary-2
             source: mahimairaja/tweet-summarization-llama-2-finetuned  

we should be able to define it as

apiVersion: v1
kind: ConfigMap
metadata:
  name: vllm-llama2-7b-adapters
data:
  configmap.yaml: |
      vLLMLoRAConfig:
        name: vllm-llama2-7b
        port: 8000
        defaultBaseModel: meta-llama/Llama-2-7b-hf
        ensureExist:
          models:
          - id: tweet-summary-1
            source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm
          - id: tweet-summary-2
             source: mahimairaja/tweet-summarization-llama-2-finetuned  

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureCategorizes issue or PR as related to a new feature.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions