Two GPU configuration?

[krzysiekpodk]

I tried to modify YAML for deepseek to utilize more vram across two GPUs, any suggestions how to modify it - i.e. which modules could potentially also weight around 24gb?

[Azure-Tang]

We have designed our operators to support different devices, which potentially allowing pipelines to utilize two devices simultaneously. But by default, all weights are loaded onto cuda:0 for convenience, considering that our project primarily serves local llama users. To enable pipeline processing on a single compute node, we need to implement a device map to manage the device allocation for operations. This approach could also potentially extend to multi-node pipelines. I have added this feature to my to-do list and plan to introduce it soon.

[Azure-Tang]

Hi,

We’ve updated our multi-GPU configuration. You can find the tutorial here. Additionally, we’ve provided yaml examples for 2-GPU setups using deepseek, qwen. These examples are available in the following directory: ktransformers/optimize/optimize_rules.

If you encounter any issues with the two-GPU configuration, don’t hesitate to reach out.

[Yongbozzz]

etups using deepseek, qwen. These examples are available in the following directory: ktransformers/optimize/optimize_rules

Hi , thank you so much , I tried the 2 GPUs version and it works, but the performance is not better than 1 GPU? I see the total GPU utils is not changing compared with single GPU(2 GPU: 17%+16% ,1 GPU:30+%)