66 points | by Maverick617 10 hours ago ago
10 comments
claude code appends a role:"system" block after the user prompt, so a proxy rewriting the trailing user message is a no-op.
I didn't see any benchmarks against vllm, sglang, exllama, etc
Since this is basically a wrapper around libllama.so, I would assume that the performance is roughly the same as llama.cpp upstream.
Woof. Wonder if the creator knows that
From what I've seen, Vulkan adds a lot of overhead on Intel hardware.
> llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback
I got excited about someone paying attention to intel. Oh well.
What hardware do you have? I’ve been playing with a 258V and OpenVINO has come a longggggg way.
Two arc b60s. The intel vllm build is getting me ~15t/s decode with heavy context using qwen3.8 27b.
llama.cpp sycl and vllm xmx work is pretty incredible right now - you just gotta build it with some extra flags
Llama would be nice for the ggufs. Any specific flags or tutorials I should look at?
claude code appends a role:"system" block after the user prompt, so a proxy rewriting the trailing user message is a no-op.
I didn't see any benchmarks against vllm, sglang, exllama, etc
Since this is basically a wrapper around libllama.so, I would assume that the performance is roughly the same as llama.cpp upstream.
Woof. Wonder if the creator knows that
From what I've seen, Vulkan adds a lot of overhead on Intel hardware.
> llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback
I got excited about someone paying attention to intel. Oh well.
What hardware do you have? I’ve been playing with a 258V and OpenVINO has come a longggggg way.
Two arc b60s. The intel vllm build is getting me ~15t/s decode with heavy context using qwen3.8 27b.
llama.cpp sycl and vllm xmx work is pretty incredible right now - you just gotta build it with some extra flags
Llama would be nice for the ggufs. Any specific flags or tutorials I should look at?