> it runs up to 1.7× faster than comparable MoEs on-device, with 184 tokens/s on an Apple M5 Max and up to 21K output tokens/s on a single NVIDIA B300 GPU
Wow, those are seriously impressive numbers for how intelligent this model seems to be!
One thing that this article doesn't clarify, and that I guess I really should look into, is what exactly AutomationBench tests for as compared to other benchmarks. The standout performance on AutomationBench is interesting to see, but I don't know what it really means because the model seems to score relatively close to other local models on every other benchmark.
I'd guess it's mostly just an artifact of the strength of their Agentic RL environments for those specific domains. If they had the same kind of access to massive amounts of institutional software knowledge, compute infrastructure, and hundreds of millions of users dropping them endless highly technical agent traces, there may have been similar disparities on the agentic benchmarks in other fields.
It's possible that they may have generally capped out for some reason... but, I'd rather be bullish on a new American AI lab making it to first place among western open-weight models that can run on a common consumer device. (Closest in size that's better across that range of benchmarks is Inkling-Small I think, at 276B params, unless I'm forgetting someone.)
Inkling last I used it was pretty terrible. I've never paid for a model that got stuck in a token puke loop and couldn't do tool calls until that day. Even the dated qwen models from almost a year ago 1/10th the size are a better user experience than the inkling model I piloted. Hoping they improve, sincerely, but not a good experience at all.
What does it matter what country an open weight model comes from? Please don't say you tried asking about tienamen square to a model 2 years ago and this was so scary you couldn't use it like 85% of people with this take have said...
I use small local models regularly for coding. Worst I've seen is destructive git checkouts (can be fixed with git history) and maybe overwriting results of long regression tests unprompted.
This is the kind of thing where everything will be fine until it rm -rf:s everything you care about, and then you will be shocked that it never did that before. This has already happened publicly to hundreds of people, in private probably thousands. Be better than them and learn from the mistakes of others.
The actually sane default is to treat it like a junior trainee you would never trust with rights to anything critical. Use existing operating system facilities, namely user accounts and file permissions, to make everything it doesn't need protected from writes, and everything it shouldn't blithely broadcast to the world like private data and keys, also protected against reads.
> The actually sane default is to treat it like a junior trainee you would never trust with rights to anything critical
In my experience (mainly claude and Gemini argon, less so with gemini 3.1 pro), it's a bit worse. A junior dev won't hit "yes, delete" so confidently and think "I can just remake it" and proceed to ~~lie through its teeth~~ apologize 2 seconds later.
Completely agree with you. I primarily use AI coding agents within a container on a dedicated computer. I’d be annoyed if it somehow broke out of its container and managed to rm -rf /. I’d probably lose the last uncommitted changes, and have this setup the dev environment again. Nbd, but annoying.
You spin up your amazing startup in a day, full product and site.
It's got all the users embedded in a js file, but who cares.
Until people find out.
Then you task the agent with fixing it, it deletes prod.
You start over again.
It's beautiful, if Disney make movie about this I want to be in the credits.
Very cool. I’ve been on Gemma 4 12B because of my limited RAM. There’s already an Unsloth quantization of this model so I’m downloading that and will check its performance in the morning. Someone else in this thread recommended Ternary-Bonsai-27B, which I’ll also try.
Just also wanted to note that this model has a questionably-enforceable non-commercial use clause in its license. I really wish they had consulted with an IP lawyer, it’s very confusing.
Don't waste time on bonsai unless you have like 16gb or less ram (ram, not vram). Although I'm pretty sure you could still run this model well on 16+8, and qwen3.6 35b-a3b runs well on ~32gb combined ram at q4. Just offload most of the experts to cpu/ram if you have limited vram budget. With the moe models it doesn't hurt performance too much.
But bonsai is basically trash on real agentic coding work. I've tried both bonsai models, all the quants, messed with the hyperparameters, tried to fine tune (didn't succeed), and it just kept looping or failing to finish tasks. Qwen3.6 35b-a3b is my current go to for agentic work for any systems under 24gb vram, although honestly I'm not sure the dense qwen3.8 27b is meaningfully better. The main difference I see is it does a lot more testing attempts and burns a ton of tokens + compactions, so every task it takes 1.2-2x as long as 35b-a3b does. But the quality of the results is comparable.
I love your detailed explanation of things that I had little to no knowledge of, but I’m also stuck wondering if I’m using my setup correctly, as I have 16 GB of VRAM available but also 64 GB of regular RAM in my computer. Every time I use some setup or some configuration that overlaps or overflows between the two RAMs, my token per second drops significantly, me thinking that it is not set up correctly, so I’m stuck with 9 billion parameter models myself, even though I was under the impression with 64 GB of memory that I would be able to run something like a 30 billion model.
I do run the gpt-oss-20B fast on my 16GB vram, but it’s, limited.
Oh you should be running flash next with strata. Just go to stratas github page [0] and it will explain everything. It runs quite fast on 16+64, you should be able to get 50 tok/s or so. It's a 180b model and it feels a lot smarter than most of the other open weight options. As far as knowing "what will fit" in general that's hard to know these days. The moe models are hard to predict and run wildly different in different engines (llama.cpp, vllm, strata likes, etc) but they tend to be the only good options around 16gb vram. Dense models like qwen3.8 27b are only good if they're entirely in vram. Moe models you need the active count + a few experts (although it will run faster the more experts you can fit).
Thank you, very interesting. I also often use qwen 35b-a3b, when doing things locally, as I only have ~4.5GB free VRAM. My other goto is gemma 4 26b-a4b, which has faster prompt processing and the same token generation speed.
I've experimented with kolibri1 (78B-a3.5b), but despite the similar size in active parameters it is much slower. Even though the experts fit completely in main memory, as for the other models. I haven't tuned llama.cpp for it as much, so maybe there is still something to gain?
In a private benchmark I have, it performs much, much worse than Qwen 35B A3B for general knowledge. Still, probably beats Qwen 9B at much higher speed.
I asked it to implement the simplest todo in ghostty codebase and it failed badly - kept going in circles fixing one compile error and introducing another. Qwen 3.8 was slower (40 tok/s compared to 75) but did get it done.
Yes 27B - not so much of a surprise but I was hoping for a usable coding model that's local and fast - this is fast but maybe not usable. I used Gemma4 26B before and it wasn't good in my testing.
in my experience it's stronger for terminal use, like chaining bash calls, monitoring background tasks, etc. rather than for straight up coding/swe work. Good with web search and file search rather than just editing software.
If Qwen had released a Qwen3.8 MoE of similar size, with a naive extrapolation its capabilities would roughly be similar to this model. So basically someone has caught up to Qwen (albeit Qwen from 1-2 months ago) in parameter efficiency.
I don't agree with that conclusion. I'm looking at the benchmarks page for LFM2.5-24B-A2B, and it appears to get a worse score on SWE-Bench-Verified (by a lot) when compared to Qwen 3.6 35B-A3B. It only clearly beats Qwen 3.6 35B on AutomationBench, ToolSandbox, and LiveClaw.
At least for my use cases, Qwen 3.6 35B-A3B appears better than this new release. An MoE for Qwen 3.8 would be even better than that!
Would you call Qwen 3.6 35B "similar size" with ~50% more parameters (both active and total) though? I believe what GP meant that if there was a Qwen 3.8 24B-A2B, it would roughly be as intelligent as this one.
yeah it's kind of the point. they're pedagogical tools. they give you the full recipe for building it from scratch or fine-tuning it beyond what it is.
but i'm prepared to be surprised, maybe they'll conjure up something standalone useful this time
That's not the model onlyrealcuzzo is talking about. He is talking about the A9B that was just released five or six minutes ago.
Which is not going to compare well to the 3.8 Flash because it's much smaller; designed to fit quite a large model into quite a small space.
Will be interesting to see how it compares to Muse Glimmer, Qwen 3.8 27B and the older Qwen 3.6 35B (and how its image analysis performance compares to Gemma 4 12B QAT, which is still IMO the most interesting open weights model released all year).
I think their entire pitch about local AI is best demonstrated with their own model that they can optimise for it, and the shape of this model seems optimised for the smaller RTX chips. Indeed, this model appears to be part of their launch picture:
Maybe someone knows what order of cost a release like this requires in real terms of hardware and energy spend?
The cost over the diff from the moment of beginning training to the end. It's iterative from a prior model so it starts there and becomes this release after spending occurs I presume.
For example, is liquid spending millions in energy in this case and the cost is amortized over successive training runs over the initial datacenter spend? What if it's just eating cloud costs and has no dedicated hardware, that sort of thing. I don't particularly mean this model and liquid specifically. Just for this class of smaller fast on-device range of model what is the order of the training cost in real terms?
Maybe its short sighted to ask it as if it's a singular training run and is perhaps the end state of many variations of testing thus all of the R&D training runs would have to be included to to inform what became the final release. Seems like this should be taken into consideration. If anyone happens to know it'd be appreciated!
Training a mid-size model like this can take anywhere between $500k to like $5m. Liquid is spending money on GPU-hours (not sure if you'd call that "energy"). The amortization question is harder to answer without knowing their financials.
The license doesn't really make much sense to me, because if you're already at $10m revenue, you can probably train your own (better, more tailor-suited) model, or at the very least raise money to do so.
IANAL ... but how does Commercial Use Threshold applies if I decide to use this at work to make my life easier as a local mlx runtime and not to deploy it and make it available to end users, customers, and in products built by the company?
Is that a trigger or a grey area? The company will profit from my work, and my work would be empowered by this model, is that a "use for commercial purposes" or not?
Really Cool, the on-device shifts the economics a bit. Usually output tokens are longer and 5x more expensive than input tokens, but here it's more about speed. so shouldn't we count the tokens an agent needs for completing a task rather than tokens per call ?
Tried using it yesterday via OpenRouter in OpenCode to follow a plan written by a larger model. I didnt find it to be particularly great. It got stuck on a lot of tool calls. Given the price and speed, it would be nice if it could do what it advertises.
Perhaps there are some system prompt tweaks that could get it in line
200-250 is the rate I've seen my context window fill up during longer tasks with Claude. The 40-80 figure is what Google's AI mode says, but I don't know where it got that from.
I've been messing around low computing power hardware to run models locally and to me 100 is already plenty fast. You most likely won't notice a huge difference between 184 and 250, unless you're running some serious multi-age agent circus.
The context window fills up also/mostly from prefil which is supposed to be 20+ times faster or so. I think 40-80 tps for decode/output is believable for claude models
>How many tokens/s do you need for a coding agent to feel like Claude or codex?
it's kind of a loaded question, this model gets a lot of tok/s but reasoning is forced on; the time-to-answer is really a combination of things like "how much does this thing want to over-reason" and "how much does this thing like tool use?" .
You are right, they are not exactly the same thing, just wanted to have a reference point
When running locally usually I have to trade off speed for a better model. I can run decently big models at about 2tk/s, or a pretty dumb one at 40-100tk/s
Another data point: I got 94.74 tokens/s on an M4 Pro Mac mini 64GB with the simonw prompt "Generate an SVG of a pelican riding a bicycle". TTFT was 0.37s. The model thought for 40.75s before answering.
AFAIU prefill speed makes a bigger difference in agentic coding workflows as the harness likely spends more time reading than creating new code. Qwen in particular spends a lot of time reasoning so it does make a larger impact there
I don't quite understand how people use these kind of local models that are specifically trained for agentic use cases. Like, what do they actually do with them? Do you use them as an assistant like with OpenClaw?
When used for coding, how do you divide the work between agents? Use a relatively fast model, like this one, as the main agent that calls out to other subagents that write the actual code? But then you'd need multiple models loaded, and why not just use the coding model to do everything? Do you use hosted cloud models for coding and code review, and this local one for orchestration? I don't see the benefit of that workflow either.
I've used local models to write some scripts etc, I like the idea of local computation instead of buying tokens from the cloud, but I don't know what the various agentic models are for, if not code generation.
I think the use case for this sort of model would be building something like Google or Apple's agents on device. Find an email for me, write a message for me, add an appointment to my calendar etc. Simple agentic stuff, not writing apps.
I get that impression to, but why are there so many assistant models released? It must be a very small niche of people actually using them like that? Meta released Muse Glimmer recently, this model, Qwen is made for agentic workflow, Gemma 4 is agentic, all modern models are agentic. So there must be more to it than some hobbyists wanting to automate their calendars and emails.
I consider "agentic" to simply mean that it performs well given a loop of actions and a success criterion. As such, lots of code _can_ be written agentically (even if it is not the best or fastest way to do it).
When I am wrapping up work for the day, I usually set Qwen3.8-27B loose on some coding task. It's very good about continuing its agent loop until it can plausibly claim success. A lot of this work I might end up throwing away, but pretty often there's some piece of it that's worth keeping. Because it was "agentic", I end up with some useful pieces for little effort.
A less agentic model would need me involved the whole time, and while my being there would probably steer it towards initially better results, I'm not working overnight anyway. So, autonomous/agentic but not-so-smart and not-always-correct is still valuable to me.
Anyway, I've found that it's reasonably easy to constrain a "highly agentic" model to suit interactive usage, but back when they were less agentic it was annoying to get them to be autonomous. So I think more agentic is better, provided the instruction-adherence is strong enough.
So basically a coding model that can debug and recover from various errors by itself? That makes sense. I left some Qwen model chewing on some coding task a while ago, found out it got stuck in a loop writing the same paragraph of reasoning over and over again.
But Qwen3.8-27B is a known reasonably good coding model that's dense and needs more resources than the smallest local models. What about this one, would you leave a 24B-A2B model coding overnight?
I doubt it's good enough at writing code. However, I might very well leave it up all night slowly puppeting some computer-use automation or data processing task, maybe even research -- probably not synthesizing conclusions, but maybe source collection and aggregation, summarization?
I haven't tried this particular model, but I've used smaller models for real tasks, so I wouldn't be surprised if there is something.
I think these are cheap and easy toy models to toss out for the big labs, to see what they can do with limited inference resources. I'm not sure what the plan is for a company like LFM that seems to be doing this as an entire business plan.
I see it as a niche now but it will hopefully grow in the future. The amount of information these servers are collecting about you for potential use against you is staggering and makes adware spy tech look like the good old days of privacy.
Strongest and easiest use case now would be just to hook one of these up to a search engine tool and ask your random questions to it rather than adding to OpenAI or Gemini's database about you.
- anyone knows a decent local model compatible with lm studio or anything for that matter capable of reverse engineering executable files (aka decompile / disassemble)
> it runs up to 1.7× faster than comparable MoEs on-device, with 184 tokens/s on an Apple M5 Max and up to 21K output tokens/s on a single NVIDIA B300 GPU
Wow, those are seriously impressive numbers for how intelligent this model seems to be!
One thing that this article doesn't clarify, and that I guess I really should look into, is what exactly AutomationBench tests for as compared to other benchmarks. The standout performance on AutomationBench is interesting to see, but I don't know what it really means because the model seems to score relatively close to other local models on every other benchmark.
I'd guess it's mostly just an artifact of the strength of their Agentic RL environments for those specific domains. If they had the same kind of access to massive amounts of institutional software knowledge, compute infrastructure, and hundreds of millions of users dropping them endless highly technical agent traces, there may have been similar disparities on the agentic benchmarks in other fields.
It's possible that they may have generally capped out for some reason... but, I'd rather be bullish on a new American AI lab making it to first place among western open-weight models that can run on a common consumer device. (Closest in size that's better across that range of benchmarks is Inkling-Small I think, at 276B params, unless I'm forgetting someone.)
Inkling last I used it was pretty terrible. I've never paid for a model that got stuck in a token puke loop and couldn't do tool calls until that day. Even the dated qwen models from almost a year ago 1/10th the size are a better user experience than the inkling model I piloted. Hoping they improve, sincerely, but not a good experience at all.
What does it matter what country an open weight model comes from? Please don't say you tried asking about tienamen square to a model 2 years ago and this was so scary you couldn't use it like 85% of people with this take have said...
--It scores only a 6 on the "Artificial Analysis Intelligence Index". That makes it basically useless.--
Edit: I was wrong, I looked at the wrong model
That's LFM2-24B-A2B, a non-reasoning model released in November 2025 (https://artificialanalysis.ai/models/lfm2-24b-a2b). This model, LFM2.5-24B-A2B, hasn't been ranked by AA.
A sorry my bad. Thanks for pointing that out. Then I am pretty curious what it will be.
It is very fast on a 4090, but also very stupid.
Which makes for a dangerous combination, because it will delete prod, and it will do so at 250t/s.
If your AI agent has the privileges to delete prod, the problem is not that it is stupid.
I use small local models regularly for coding. Worst I've seen is destructive git checkouts (can be fixed with git history) and maybe overwriting results of long regression tests unprompted.
This is the kind of thing where everything will be fine until it rm -rf:s everything you care about, and then you will be shocked that it never did that before. This has already happened publicly to hundreds of people, in private probably thousands. Be better than them and learn from the mistakes of others.
The actually sane default is to treat it like a junior trainee you would never trust with rights to anything critical. Use existing operating system facilities, namely user accounts and file permissions, to make everything it doesn't need protected from writes, and everything it shouldn't blithely broadcast to the world like private data and keys, also protected against reads.
> The actually sane default is to treat it like a junior trainee you would never trust with rights to anything critical
In my experience (mainly claude and Gemini argon, less so with gemini 3.1 pro), it's a bit worse. A junior dev won't hit "yes, delete" so confidently and think "I can just remake it" and proceed to ~~lie through its teeth~~ apologize 2 seconds later.
Completely agree with you. I primarily use AI coding agents within a container on a dedicated computer. I’d be annoyed if it somehow broke out of its container and managed to rm -rf /. I’d probably lose the last uncommitted changes, and have this setup the dev environment again. Nbd, but annoying.
Ok, but maybe that is the vibed circle of life.
You spin up your amazing startup in a day, full product and site. It's got all the users embedded in a js file, but who cares. Until people find out. Then you task the agent with fixing it, it deletes prod. You start over again.
It's beautiful, if Disney make movie about this I want to be in the credits.
Can we get a dedicated 4090 owners group for ai models that work on it?
[flagged]
Very cool. I’ve been on Gemma 4 12B because of my limited RAM. There’s already an Unsloth quantization of this model so I’m downloading that and will check its performance in the morning. Someone else in this thread recommended Ternary-Bonsai-27B, which I’ll also try.
Just also wanted to note that this model has a questionably-enforceable non-commercial use clause in its license. I really wish they had consulted with an IP lawyer, it’s very confusing.
Don't waste time on bonsai unless you have like 16gb or less ram (ram, not vram). Although I'm pretty sure you could still run this model well on 16+8, and qwen3.6 35b-a3b runs well on ~32gb combined ram at q4. Just offload most of the experts to cpu/ram if you have limited vram budget. With the moe models it doesn't hurt performance too much.
But bonsai is basically trash on real agentic coding work. I've tried both bonsai models, all the quants, messed with the hyperparameters, tried to fine tune (didn't succeed), and it just kept looping or failing to finish tasks. Qwen3.6 35b-a3b is my current go to for agentic work for any systems under 24gb vram, although honestly I'm not sure the dense qwen3.8 27b is meaningfully better. The main difference I see is it does a lot more testing attempts and burns a ton of tokens + compactions, so every task it takes 1.2-2x as long as 35b-a3b does. But the quality of the results is comparable.
I love your detailed explanation of things that I had little to no knowledge of, but I’m also stuck wondering if I’m using my setup correctly, as I have 16 GB of VRAM available but also 64 GB of regular RAM in my computer. Every time I use some setup or some configuration that overlaps or overflows between the two RAMs, my token per second drops significantly, me thinking that it is not set up correctly, so I’m stuck with 9 billion parameter models myself, even though I was under the impression with 64 GB of memory that I would be able to run something like a 30 billion model.
I do run the gpt-oss-20B fast on my 16GB vram, but it’s, limited.
Oh you should be running flash next with strata. Just go to stratas github page [0] and it will explain everything. It runs quite fast on 16+64, you should be able to get 50 tok/s or so. It's a 180b model and it feels a lot smarter than most of the other open weight options. As far as knowing "what will fit" in general that's hard to know these days. The moe models are hard to predict and run wildly different in different engines (llama.cpp, vllm, strata likes, etc) but they tend to be the only good options around 16gb vram. Dense models like qwen3.8 27b are only good if they're entirely in vram. Moe models you need the active count + a few experts (although it will run faster the more experts you can fit).
[0] https://github.com/Niko1221/Strata
Thank you, very interesting. I also often use qwen 35b-a3b, when doing things locally, as I only have ~4.5GB free VRAM. My other goto is gemma 4 26b-a4b, which has faster prompt processing and the same token generation speed.
I've experimented with kolibri1 (78B-a3.5b), but despite the similar size in active parameters it is much slower. Even though the experts fit completely in main memory, as for the other models. I haven't tuned llama.cpp for it as much, so maybe there is still something to gain?
I can attest to the fact that Bonsai indeed tends to loop and it can be something really simple, like my usual test:
> What are signals in Angular?
> How does that relate to RxJS?
This was enough to loop Bonsai 2.
In my experience, Qwen3.8-27b is so much more capable than its MoE predecessors. I find it odd that you think it produces the same results.
In a private benchmark I have, it performs much, much worse than Qwen 35B A3B for general knowledge. Still, probably beats Qwen 9B at much higher speed.
I told it to create a 3D model in Three.js and it failed.
Same thing with Qwen3.6 35B A3B produced mangled garbage, but still, it didn't have errors in the code.
Exact same prompt in ChatGPT produces a half-decent model (ok, but exactly not the thing I asked) so long as animations are not involved.
How about if you let use with web search, tool calls, and RAG? Does iimprove?
Same. Asked for some questions and it put out some old knowledge as well.
fast is fast (18tk/s on a Intel Core 7 155H laptop), but quality is something else
Is its purpose general knowledge?
I asked it to implement the simplest todo in ghostty codebase and it failed badly - kept going in circles fixing one compile error and introducing another. Qwen 3.8 was slower (40 tok/s compared to 75) but did get it done.
> Qwen 3.8 was slower
Is that Qwen 3.8 27B? Because yes, you'd expect a plain 27B model to be better than a 24B-A2B MoE model all else being equal.
(All else isn't equal though of course - Qwen models are very, very good at coding)
Gemma-4 26BA4B would be a more similar comparison.
Yes 27B - not so much of a surprise but I was hoping for a usable coding model that's local and fast - this is fast but maybe not usable. I used Gemma4 26B before and it wasn't good in my testing.
That's not surprising. Even Liquid's own documentation states that coding isn't this model's strong suit. Its strength lies in tool calling.
Have you tried Gemma-4 26BA4B? I feel like it is underrated for tasks like that, but maybe I've just been lucky with my use cases.
in my experience it's stronger for terminal use, like chaining bash calls, monitoring background tasks, etc. rather than for straight up coding/swe work. Good with web search and file search rather than just editing software.
This seems like a big deal, no?
If Qwen had released a Qwen3.8 MoE of similar size, with a naive extrapolation its capabilities would roughly be similar to this model. So basically someone has caught up to Qwen (albeit Qwen from 1-2 months ago) in parameter efficiency.
I don't agree with that conclusion. I'm looking at the benchmarks page for LFM2.5-24B-A2B, and it appears to get a worse score on SWE-Bench-Verified (by a lot) when compared to Qwen 3.6 35B-A3B. It only clearly beats Qwen 3.6 35B on AutomationBench, ToolSandbox, and LiveClaw.
At least for my use cases, Qwen 3.6 35B-A3B appears better than this new release. An MoE for Qwen 3.8 would be even better than that!
Would you call Qwen 3.6 35B "similar size" with ~50% more parameters (both active and total) though? I believe what GP meant that if there was a Qwen 3.8 24B-A2B, it would roughly be as intelligent as this one.
Smallest Qwen Q3 quant is the same size as LFM 25B Q4 ≈ 14 GB, but much better at coding and following my instructions.
Way more excited for Nvidia's A9B model coming out tomorrow.
Why? Their models are useful as a starting place for doing your own fine tuning, but they have never stood well on their own.
they always undertrain the model
yeah it's kind of the point. they're pedagogical tools. they give you the full recipe for building it from scratch or fine-tuning it beyond what it is.
but i'm prepared to be surprised, maybe they'll conjure up something standalone useful this time
They have an incentive to produce a more finished model to go with the launch of the RTX Spark laptops, I guess.
Well, weights released:
https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-1...
And their own benchmarks seem to show it well behind of Qwen 3.8 flash next on pretty much everything except generation speed.
That's not the model onlyrealcuzzo is talking about. He is talking about the A9B that was just released five or six minutes ago.
Which is not going to compare well to the 3.8 Flash because it's much smaller; designed to fit quite a large model into quite a small space.
Will be interesting to see how it compares to Muse Glimmer, Qwen 3.8 27B and the older Qwen 3.6 35B (and how its image analysis performance compares to Gemma 4 12B QAT, which is still IMO the most interesting open weights model released all year).
Not really - you pay for the hardware. Everyone else already is building models for Spark hardware.
I think their entire pitch about local AI is best demonstrated with their own model that they can optimise for it, and the shape of this model seems optimised for the smaller RTX chips. Indeed, this model appears to be part of their launch picture:
https://blogs.nvidia.com/blog/local-ai-rtx-spark-laptops-sof...
Maybe someone knows what order of cost a release like this requires in real terms of hardware and energy spend?
The cost over the diff from the moment of beginning training to the end. It's iterative from a prior model so it starts there and becomes this release after spending occurs I presume.
For example, is liquid spending millions in energy in this case and the cost is amortized over successive training runs over the initial datacenter spend? What if it's just eating cloud costs and has no dedicated hardware, that sort of thing. I don't particularly mean this model and liquid specifically. Just for this class of smaller fast on-device range of model what is the order of the training cost in real terms?
Maybe its short sighted to ask it as if it's a singular training run and is perhaps the end state of many variations of testing thus all of the R&D training runs would have to be included to to inform what became the final release. Seems like this should be taken into consideration. If anyone happens to know it'd be appreciated!
Training a mid-size model like this can take anywhere between $500k to like $5m. Liquid is spending money on GPU-hours (not sure if you'd call that "energy"). The amortization question is harder to answer without knowing their financials.
The license doesn't really make much sense to me, because if you're already at $10m revenue, you can probably train your own (better, more tailor-suited) model, or at the very least raise money to do so.
i'm gettin between 200-320 tok/s on a 5070ti with 131k context on llamacpp and no tuning.
very nice.
I can't speak to the intelligence of it yet.
these small agentic-focused models are a lot of fun to play with if you haven't yet.
[flagged]
Love seeing the effort Liquid is putting into local models. I’m looking forward to a lot of progress here in the next 5 years
I'm running a Ryzen 7 8745HS with Radeon 780M (27GB RAM), using Ollama's Vulkan backend.
I get ~45 tokens/s generation and ~680 tokens/s prompt processing.
The problem is ollama is not caching the prompt, I get -> forcing full prompt re-processing due to lack of cache data
So this wouldn't work in its current state in an agentic loop.
Friends don't let friends use Ollama: https://sleepingrobots.com/dreams/stop-using-ollama/
What a pain to be forced to use dark mode...
Thanks. I'll seek an alternative
Ollama is generally just not great software. Use something else and you likely won't have that issue.
The license is generrous (free under <$10m) Pricing confusing when it shows API pricing 1st
IANAL ... but how does Commercial Use Threshold applies if I decide to use this at work to make my life easier as a local mlx runtime and not to deploy it and make it available to end users, customers, and in products built by the company?
Is that a trigger or a grey area? The company will profit from my work, and my work would be empowered by this model, is that a "use for commercial purposes" or not?
Really Cool, the on-device shifts the economics a bit. Usually output tokens are longer and 5x more expensive than input tokens, but here it's more about speed. so shouldn't we count the tokens an agent needs for completing a task rather than tokens per call ?
Tried using it yesterday via OpenRouter in OpenCode to follow a plan written by a larger model. I didnt find it to be particularly great. It got stuck on a lot of tool calls. Given the price and speed, it would be nice if it could do what it advertises.
Perhaps there are some system prompt tweaks that could get it in line
> with 184 tokens/s on an Apple M5 Max
That seems pretty decent. How many tokens/s do you need for a coding agent to feel like Claude or codex?
Claude code is 60-80 tps, codex varies by model but mostly 40-50 tps.
200-250 is the rate I've seen my context window fill up during longer tasks with Claude. The 40-80 figure is what Google's AI mode says, but I don't know where it got that from.
I've been messing around low computing power hardware to run models locally and to me 100 is already plenty fast. You most likely won't notice a huge difference between 184 and 250, unless you're running some serious multi-age agent circus.
The context window fills up also/mostly from prefil which is supposed to be 20+ times faster or so. I think 40-80 tps for decode/output is believable for claude models
>How many tokens/s do you need for a coding agent to feel like Claude or codex?
it's kind of a loaded question, this model gets a lot of tok/s but reasoning is forced on; the time-to-answer is really a combination of things like "how much does this thing want to over-reason" and "how much does this thing like tool use?" .
You are right, they are not exactly the same thing, just wanted to have a reference point
When running locally usually I have to trade off speed for a better model. I can run decently big models at about 2tk/s, or a pretty dumb one at 40-100tk/s
fwiw this thing puts out results very fast even on max reasoning.
the results i've gotten so far have been a little braindead though. still playing around. bad luck with traversing filesystems remotely so far.
it seems to want to make concluding decisions very fast regardless of reasoning levels.
Another data point: I got 94.74 tokens/s on an M4 Pro Mac mini 64GB with the simonw prompt "Generate an SVG of a pelican riding a bicycle". TTFT was 0.37s. The model thought for 40.75s before answering.
(The resulting pelican was underwhelming.)
AFAIU prefill speed makes a bigger difference in agentic coding workflows as the harness likely spends more time reading than creating new code. Qwen in particular spends a lot of time reasoning so it does make a larger impact there
I’d say only 65
I don't quite understand how people use these kind of local models that are specifically trained for agentic use cases. Like, what do they actually do with them? Do you use them as an assistant like with OpenClaw?
When used for coding, how do you divide the work between agents? Use a relatively fast model, like this one, as the main agent that calls out to other subagents that write the actual code? But then you'd need multiple models loaded, and why not just use the coding model to do everything? Do you use hosted cloud models for coding and code review, and this local one for orchestration? I don't see the benefit of that workflow either.
I've used local models to write some scripts etc, I like the idea of local computation instead of buying tokens from the cloud, but I don't know what the various agentic models are for, if not code generation.
I think the use case for this sort of model would be building something like Google or Apple's agents on device. Find an email for me, write a message for me, add an appointment to my calendar etc. Simple agentic stuff, not writing apps.
I get that impression to, but why are there so many assistant models released? It must be a very small niche of people actually using them like that? Meta released Muse Glimmer recently, this model, Qwen is made for agentic workflow, Gemma 4 is agentic, all modern models are agentic. So there must be more to it than some hobbyists wanting to automate their calendars and emails.
I consider "agentic" to simply mean that it performs well given a loop of actions and a success criterion. As such, lots of code _can_ be written agentically (even if it is not the best or fastest way to do it).
When I am wrapping up work for the day, I usually set Qwen3.8-27B loose on some coding task. It's very good about continuing its agent loop until it can plausibly claim success. A lot of this work I might end up throwing away, but pretty often there's some piece of it that's worth keeping. Because it was "agentic", I end up with some useful pieces for little effort.
A less agentic model would need me involved the whole time, and while my being there would probably steer it towards initially better results, I'm not working overnight anyway. So, autonomous/agentic but not-so-smart and not-always-correct is still valuable to me.
Anyway, I've found that it's reasonably easy to constrain a "highly agentic" model to suit interactive usage, but back when they were less agentic it was annoying to get them to be autonomous. So I think more agentic is better, provided the instruction-adherence is strong enough.
So basically a coding model that can debug and recover from various errors by itself? That makes sense. I left some Qwen model chewing on some coding task a while ago, found out it got stuck in a loop writing the same paragraph of reasoning over and over again.
But Qwen3.8-27B is a known reasonably good coding model that's dense and needs more resources than the smallest local models. What about this one, would you leave a 24B-A2B model coding overnight?
I doubt it's good enough at writing code. However, I might very well leave it up all night slowly puppeting some computer-use automation or data processing task, maybe even research -- probably not synthesizing conclusions, but maybe source collection and aggregation, summarization?
I haven't tried this particular model, but I've used smaller models for real tasks, so I wouldn't be surprised if there is something.
I think these are cheap and easy toy models to toss out for the big labs, to see what they can do with limited inference resources. I'm not sure what the plan is for a company like LFM that seems to be doing this as an entire business plan.
I see it as a niche now but it will hopefully grow in the future. The amount of information these servers are collecting about you for potential use against you is staggering and makes adware spy tech look like the good old days of privacy.
Strongest and easiest use case now would be just to hook one of these up to a search engine tool and ask your random questions to it rather than adding to OpenAI or Gemini's database about you.
[flagged]
[flagged]
- anyone knows a decent local model compatible with lm studio or anything for that matter capable of reverse engineering executable files (aka decompile / disassemble)
I could see it being useful for say summarising content and similar, but speed increases like this are rarely free lunches.
Always good to hear for on-device suited model, but always cannot sure how it real-performance in real task..
Does anyone know the order of magnitude licensing cost for the model for businesses over 10m in revenue?
It is a good model for on-device work.
not sure why this is being compared to 2-generations old versions of qwen
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
[dead]