Lora syncer docs #320
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: vllm-llama2-7b-pool | ||
| spec: | ||
| selector: | ||
| app: vllm-llama2-7b-pool | ||
| ports: | ||
| - protocol: TCP | ||
| port: 8000 | ||
| targetPort: 8000 | ||
| type: ClusterIP | ||
| --- | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: vllm-llama2-7b-pool | ||
| spec: | ||
| replicas: 3 | ||
| selector: | ||
| matchLabels: | ||
| app: vllm-llama2-7b-pool | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: vllm-llama2-7b-pool | ||
| spec: | ||
| containers: | ||
| - name: lora | ||
| image: "vllm/vllm-openai:latest" | ||
| imagePullPolicy: Always | ||
| command: ["python3", "-m", "vllm.entrypoints.openai.api_server"] | ||
| args: | ||
| - "--model" | ||
| - "meta-llama/Llama-2-7b-hf" | ||
| - "--tensor-parallel-size" | ||
| - "1" | ||
| - "--port" | ||
| - "8000" | ||
| - "--enable-lora" | ||
| - "--max-loras" | ||
| - "4" | ||
| - "--max-cpu-loras" | ||
| - "12" | ||
| - "--lora-modules" | ||
| - '{"name": "tweet-summary-0", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}' | ||
| - '{"name": "tweet-summary-1", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}' | ||
| env: | ||
| - name: PORT | ||
| value: "8000" | ||
| - name: HUGGING_FACE_HUB_TOKEN | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: hf-token | ||
| key: token | ||
| - name: VLLM_ALLOW_RUNTIME_LORA_UPDATING | ||
| value: "true" | ||
| ports: | ||
| - containerPort: 8000 | ||
| name: http | ||
| protocol: TCP | ||
| livenessProbe: | ||
| failureThreshold: 240 | ||
| httpGet: | ||
| path: /health | ||
| port: http | ||
| scheme: HTTP | ||
| initialDelaySeconds: 5 | ||
| periodSeconds: 5 | ||
| successThreshold: 1 | ||
| timeoutSeconds: 1 | ||
| readinessProbe: | ||
| failureThreshold: 600 | ||
| httpGet: | ||
| path: /health | ||
| port: http | ||
| scheme: HTTP | ||
| initialDelaySeconds: 5 | ||
| periodSeconds: 5 | ||
| successThreshold: 1 | ||
| timeoutSeconds: 1 | ||
| resources: | ||
| limits: | ||
| nvidia.com/gpu: 1 | ||
| requests: | ||
| nvidia.com/gpu: 1 | ||
| volumeMounts: | ||
| - mountPath: /data | ||
| name: data | ||
| - mountPath: /dev/shm | ||
| name: shm | ||
| - name: adapters | ||
| mountPath: "/adapters" | ||
| initContainers: | ||
| - name: lora-adapter-syncer | ||
| tty: true | ||
| stdin: true | ||
| image: us-central1-docker.pkg.dev/ahg-gke-dev/jobset2/lora-syncer:6dc97be | ||
| restartPolicy: Always | ||
| imagePullPolicy: Always | ||
| env: | ||
| - name: DYNAMIC_LORA_ROLLOUT_CONFIG | ||
| value: "/config/configmap.yaml" | ||
| volumeMounts: # DO NOT USE subPath | ||
| - name: config-volume | ||
| mountPath: /config | ||
| restartPolicy: Always | ||
| schedulerName: default-scheduler | ||
| terminationGracePeriodSeconds: 30 | ||
| volumes: | ||
| - name: data | ||
| emptyDir: {} | ||
| - name: shm | ||
| emptyDir: | ||
| medium: Memory | ||
| - name: adapters | ||
| emptyDir: {} | ||
| - name: config-volume | ||
| configMap: | ||
| name: dynamic-lora-config | ||
|
|
||
| --- | ||
|
|
||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: dynamic-lora-config | ||
| data: | ||
| configmap.yaml: | | ||
| vLLMLoRAConfig: | ||
| name: sql-loras-llama | ||
| port: 8000 | ||
| ensureExist: | ||
| models: | ||
| - base-model: meta-llama/Llama-2-7b-hf | ||
| id: tweet-summary-0 | ||
| source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm | ||
| - base-model: meta-llama/Llama-2-7b-hf | ||
| id: tweet-summary-1 | ||
| source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm | ||
| ensureNotExist: | ||
| models: | ||
| - base-model: meta-llama/Llama-2-7b-hf | ||
| id: tweet-summary-2 | ||
| source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| # Getting started with Gateway API Inference Extension with Dynamic lora updates on vllm | ||
|
|
||
| The goal of this guide is to get a single InferencePool running with vLLM and demonstrate use of dynamic lora updating! | ||
|
|
||
| ### Requirements | ||
| - Envoy Gateway [v1.2.1](https://gateway.envoyproxy.io/docs/install/install-yaml/#install-with-yaml) or higher | ||
| - A cluster with: | ||
| - Support for Services of type `LoadBalancer`. (This can be validated by ensuring your Envoy Gateway is up and running). For example, with Kind, | ||
| you can follow [these steps](https://kind.sigs.k8s.io/docs/user/loadbalancer). | ||
| - 3 GPUs to run the sample model server. Adjust the number of replicas in `./manifests/vllm/deployment.yaml` as needed. | ||
|
|
||
| ### Steps | ||
|
|
||
| 1. **Deploy Sample VLLM Model Server with dynamic lora update enabled and dynamic lora syncer sidecar ** | ||
| [Redeploy the vLLM deployment with Dynamic lora adapter enabled and Lora syncer sidecar and configmap](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/manifests/vllm/dynamic-lora-sidecar/deployment.yaml) | ||
|
|
||
| Rest of the steps are same as [general setup](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/site-src/guides/index.md) | ||
|
|
||
|
|
||
| ### Safely rollout v2 adapter | ||
|
|
||
| 1. Update the LoRA syncer ConfigMap to make the new adapter version available on the model servers. | ||
|
|
||
| ```yaml | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: dynamic-lora-config | ||
| data: | ||
| configmap.yaml: | | ||
| vLLMLoRAConfig: | ||
| name: sql-loras-llama | ||
| port: 8000 | ||
| ensureExist: | ||
| models: | ||
coolkp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - base-model: meta-llama/Llama-2-7b-hf | ||
| id: tweet-summary-0 | ||
| source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm | ||
| - base-model: meta-llama/Llama-2-7b-hf | ||
| id: tweet-summary-1 | ||
| source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm | ||
| - base-model: meta-llama/Llama-2-7b-hf | ||
| id: tweet-summary-2 | ||
| source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm | ||
|
Comment on lines
+24
to
+44
There was a problem hiding this comment. need to close this with ```, I couldn't make a suggestion to fix that. |
||
| 2. Configure a canary rollout with traffic split using LLMService. In this example, 40% of traffic for tweet-summary model will be sent to the ***tweet-summary-2*** adapter . | ||
|
|
||
| ```yaml | ||
| model: | ||
| name: tweet-summary | ||
| targetModels: | ||
| targetModelName: tweet-summary-0 | ||
| weight: 20 | ||
| targetModelName: tweet-summary-1 | ||
| weight: 40 | ||
| targetModelName: tweet-summary-2 | ||
| weight: 40 | ||
coolkp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ``` | ||
|
|
||
| 3. Finish rollout by setting the traffic to the new version 100%. | ||
| ```yaml | ||
| model: | ||
| name: tweet-summary | ||
| targetModels: | ||
| targetModelName: tweet-summary-2 | ||
| weight: 100 | ||
| ``` | ||
|
|
||
| 4. Remove v1 from dynamic lora configmap. | ||
| ```yaml | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: dynamic-lora-config | ||
| data: | ||
| configmap.yaml: | | ||
| vLLMLoRAConfig: | ||
| name: sql-loras-llama | ||
| port: 8000 | ||
| ensureExist: | ||
| models: | ||
| - base-model: meta-llama/Llama-2-7b-hf | ||
| id: tweet-summary-2 | ||
| source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm | ||
| ensureNotExist: | ||
| models: | ||
| - base-model: meta-llama/Llama-2-7b-hf | ||
| id: tweet-summary-1 | ||
| source: gs://[HUGGING FACE PATH] | ||
| - base-model: meta-llama/Llama-2-7b-hf | ||
| id: tweet-summary-0 | ||
| source: gs://[HUGGING FACE PATH] | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.