generated from runpod-workers/worker-template
-
Notifications
You must be signed in to change notification settings - Fork 224
Closed
Description
`
lora_modules = os.getenv('LORA_MODULES', None)
if lora_modules is not None:
try:
lora_modules = json.loads(lora_modules)
lora_modules = [LoRAModulePath(**lora_modules)]
except:
lora_modules = None
self.serving_models = OpenAIServingModels(
engine_client=self.llm,
model_config=self.model_config,
base_model_paths=self.base_model_paths,
lora_modules=None,
prompt_adapters=None,
)
self.chat_engine = OpenAIServingChat(
engine_client=self.llm,
model_config=self.model_config,
models=self.serving_models,
response_role=self.response_role,
request_logger=None,
chat_template=self.tokenizer.tokenizer.chat_template,
chat_template_content_format="auto",
# enable_reasoning=os.getenv('ENABLE_REASONING', 'false').lower() == 'true',
# reasoning_parser=None,
# return_token_as_token_ids=False,
enable_auto_tools=os.getenv('ENABLE_AUTO_TOOL_CHOICE', 'false').lower() == 'true',
tool_parser=os.getenv('TOOL_CALL_PARSER', "") or None,
enable_prompt_tokens_details=False
)
`
lora_module variable is assigned to None in OpenAIServingModels, despite being assigned other values.
Unable to use my fine-tuned model with Serverless vllm template using this version.
Metadata
Metadata
Assignees
Labels
No labels