I keep ending up with the same problem when running llama.cpp: multiple models, multiple instances, different GPU allocations, ports, flags, and a growing pile of scripts to keep it all manageable.
LlamaRack is my attempt at making that less annoying.
It manages llama.cpp instances, models, GPU assignment and runtime settings, with an OpenAI-compatible API on top.
It works well enough on my own setup now, but I’d really like to find a few other llama.cpp users willing to try it on different hardware and workflows.
Not looking for promotion or reviews - mostly bugs, rough edges, “why does it work like this?” and things I’ve completely overlooked.
I keep ending up with the same problem when running llama.cpp: multiple models, multiple instances, different GPU allocations, ports, flags, and a growing pile of scripts to keep it all manageable.
LlamaRack is my attempt at making that less annoying.
It manages llama.cpp instances, models, GPU assignment and runtime settings, with an OpenAI-compatible API on top.
It works well enough on my own setup now, but I’d really like to find a few other llama.cpp users willing to try it on different hardware and workflows.
Not looking for promotion or reviews - mostly bugs, rough edges, “why does it work like this?” and things I’ve completely overlooked.
https://github.com/brantje/LlamaRack
I'll give it a go. Currently using TurboLLM, so it'll be interesting to see the difference.