Skip to content

single binary #1888

@sozercan

Description

@sozercan

Is your feature request related to a problem? Please describe.

LocalAI should support a single binary instead of multiple options for avx, avx2, cuda, etc

Describe the solution you'd like

Support for single binary that can check capabilities and fallback when needed. It should start with GPU by checking libraries, then adjust layers if not enough VRAM, and finally fallback to CPU and adjust instruction set depending on the host capabilities.

This will make AIO simpler as logic will be handled automatically inside the binary.

Subtasks:

Describe alternatives you've considered

Additional context

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions