Plugin in PyCharm and local model in Windows.

[SrVill]

Is it possible to connect the plugin via the oobabooga-webui or koboldcpp API to a locally running model (Refact-1.6b, starcoder, etc.)?
If possible, how? Or is it possible to work with local models only as described here?

[olegklimov]

Yes we want CPU support, and a small inference server code without much dependencies would be great. The current work is in #77

[SrVill]

I had something else in mind. It does not matter on what to run the model locally (GPU or CPU), it is important that the plugin can work with a local model running not only in a docker container in WSL, because - and why do this when there is already oobabooga, where we can locally run models in a variety of formats. Refact is also launched in oobabooga, but it's not clear how to connect the plugin to it via the API.

[olegklimov]

We'll actually solve this! New plugins with a rust binary will use standard API. (HF or OpenAI style)

[hazratisulton]

#95 (comment)