The old DeepSeek V4.1 Flash has been my workhorse model for the past month (after running tests comparing w/ Qwen 3.8 Flash Next, GLM 5.3 Flash, MiMo 2.6 Flash (including the MOPD fix)) and I've run tens of billions of tokens through it and pushed it fairly hard in both private internal benchmarking and in my own workloads to figure out where it's strong/weak.
Partly because I run it on very fast local infra (200+ tok/s) and partly because I'm guaranteed it's not being changed on me, I've generally preferred it over proprietary subscription models for daily tasks, as it seems about as capable at mid-level tasks (I reserve Astra/Opus mostly for planning/review).
I've replaced the previous model with 1017, and while it's early, a few observations:
- It's CoT has changed dramatically. I find the heavy Astra distillation claims to be plausible
- It has strong benchmarks scores, and from what I've seen online it's much better at 3D games and such, but that's not so relevant for my work. On my internal evals it is more token efficient, but it actually doesn't score any higher than the previous model
- It's only been a couple days, but it does seem a bit more focused and reliable. DS4 0731 and DS4.1 have been a bit sloppy (as in not-careful, although they did tend to fall into Claudisms as well), and 1017 seems to be a bit sharper.
- I'm running it through some more challenging work, but I'm not sure if it's really at GLM-5.3 level, especially for cyber/RE/hardware poking, which GLM has excelled at.
I find these local infra costs crazy heavy duty. I always assume these folks are getting their values worth or that infra is serving multiple people's requests at and justifying the cost. Like I once thought of buying a 48-64 GB RAM laptop (not even clsoe to what these models need) and I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access. So I assume the usage is also heavy duty, or if nothing then for privacy and safety and then I guess the price, if one can pay, will totally be worth it.
Everything's been trending down so far, so until evidence shows otherwise, I'd assume it will continue that trend. IMO the best time to buy local infra will be when that trend begins to asymptote.
There is so much down pressure, I cannot imagine the token/s price going up. It has been going down and will continue doing so, we are far from a plateau IMHO
API access does not have a 5-hour or weekly limit.
I run local models, too, but every discussion about these models uses the local quantized versions and the hosted versions interchangeably. They’re not interchangeable. The quantized versions you can run on mid-grade consumer hardware are a step down in quality. That might be fine or might not depending on your use case but it needs to be acknowledged. It’s not until you get into really expensive hardware setups that you can run something like DSv4 Flash at Q8, and you’re not breaking even on that relative to a cloud provider.
Keeping everything local is about the only reason to run local. The other discussions about saving money are missing something.
You just need to imagine someone making big bucks in tech + being a little bit money ignorant, having a big house and a nice basement and cheap energy prices.
Than it just makes sense that they already do this.
But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision.
> because I'm guaranteed it's not being changed on me
This is something underrated. With an open weight model I can evaluate the performance for my use case, and know for a fact the model will stay the same over time. No arbitrary “model welfare” rules being introduced later on, no undocumented change to capabilities or performance
Somehow GPT 6.1 Sol (medium) is cheaper per task ($0.21 vs $0.26) and faster per task (2.9 vs 4.9 minutes) vs v4.1 Flash 1017 (max). I hope AA adds other reasoning effort results of Deepseek so it'd be more realistic to compare it with other models, because IMO using models at max effort is very unrealistic for most people, they then take a huge amount of time and tokens for a small performance uplift.
Well yes, thats the entire business model of benchmarking companies. If you pay attention to AA, some models are on the leaderboard second they're released while others come days later, which to me implies that theres some level of coordination happening with them and a few labs.
Yeah, this always bugged me. Then again, unlike American models, surely Chinese models are far too expensive for them to test them at every difficulty ;)
I don’t know about 1017 but according to my limited testing of 0910 and previous v4-flash-0731 on some of my use cases, max simply used a lot more tokens (almost twice as many IIRC) for little to no benefit over high. So yeah I really don’t see a reason to use max.
For DS41F (both the original and 1017) I've found low (25/100) reasoning to be the optimal for my tasks at least (max doesn't tend to better, and if you can't do it w/ low imo you should just switch to a smarter model). At low it's about 1/3-1/2 max's token count (and price and speed).
A bit like influencers of the benchmarking world. Milking it while all this trends. Are there financial/tie-up disclosures that are audited for benchmarks?
AA prices are always kind of hand wavy for open models. I mean if you rent a cheap gpu and run inference yourself...or heck, what about a cpu instance with a bunch of ram, you can do that with llama.cpp for under a $1/hour. Ok not that, so are we talking about the median spot price per token from a basket of inference providers? Which ones? They actually vary quite a bit, even on openrouter, for this particular deepseek checkpoint it's between $0.15 and $1.20 ... that's like 8x spread...
Do the math yourself, it's actually $0.08 not the AA claimed $0.10 (and that's the more expensive of the two pricing tiers). That's a pretty large error bar.
Also many of the providers are now doing "peak" and "offpeak" pricing (deepseek.com itself for instance) so it can differ dramatically. These "peak" times depend on where the service is most popular. You can basically ride "off-peak" pricing by cascading from global provider to global provider.
The majority of the time taken in an inference call is doing the inference so the extra latency of traveling across an ocean is actually a fairly small wall-clock tax here. People still seem to be partial to nearby servers for ai-related workloads but really, try some decent ones 10,000 km away, you probably won't notice.
I know AA thinks highly of the price metric (I've been to their events, even have their hat, they brag about it) but I have learned to ignore it as noise.
It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
But for closed-source models with only a handful of providers (sometimes just 1) great, the price is a very reasonable number.
For the open models a more useful thing would be like "100 tok/s spot price build" and then it's using pcpartpicker or something in some kind of partnership and can query a build for buying a machine right now that can achieve a certain benchmark number and maybe something for less interactive workloads like "5 tok/s". Of course getting some base unit of power cost in there might be important as well since the power bill is the majority of what you pay for when you rent a commodity gpu.
this one is especially annoying ... i can run these on lots of hardware but there's vast differences in many aspects.
Whether I'm getting a good deal isn't what I'm thinking about if I'm getting garbage results. It's more about why am I wasting my time on the wrong tool.
I mean what I really want is a more reliable classic eval benchmark for the decision models that I can easily deploy, like the official ollama ones: https://ollama.com/library?sort=newest
perplexity has a 27b decision
then there's a 3b from liquidai and another 600M one. Then there's one called kev and bespoke has nimble-v3. clef from cloudflare is there along with strands-decider from aws. Then there's laya and tev1 ... I mean the AA page seems to be the only real third party doing benchmark and evals on it but most of these are missing from it. It has a lot to be desired.
Maybe an agentic eval framework that tests anything that hits 100 downloads would be a decent weekend project...
Also there's like wildly different tasks. choices versus noul have way different performances and also if you are doing things like text classification such as using it for theme extraction, I mean this does work with nimble-v3 but d1 gives mostly noise and pplx-decider is in between and their performance degradation curves are dramatically different depending on the input-token length.
This is all going to be bitter lessoned eventually but I can't just sit on my hands and wait around 12 months.
So now I have a proxy that looks at the input size and do dynamic routing and sometimes fire off both and do ensemble weightings ... I mean why do I have to be clever here, this is a generic problem.
I had to discover all this myself and I really shouldn't have to do that...I guess I'm just expecting some quant results from AA that they aren't in the business of providing.
this matches my experience tbh. 6.1 sol on medium is quite good at getting things right the first time without overthinking and i prefer it over xhigh or max usually. so i'm not surprised it ends up cheaper.
What I really want to know for models like this is if I can have them run on really lame consumer hardware (consider 128gb of ram, 2080ti) with extremely low TPS, but with reasonable enough intelligence that you can walk away and expect something to bake really nicely while you sleep.
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
"Model like this" not sute about ds v4.1 but qwen3.8 flash next is currently running on 10gb 3080 + 64 gb ddr5 ram at respectable 40-70t/s with 1200pp at max context
I have a 16GB 5060 and 32GB system. RAM - are there any recommended Qwen 3.8 Flash models worth running on this ? I am using Nemotron 30B A3B at the moment
Fwiw I think you can run 3.8 flash (next?) on your setup with SSD streaming.
There's this video (1) that I found, and my understanding is that flash next can be very efficient wrt ram needed, by using a "dictionary" (or something similar).
32 gb of ram is gonna be your blocker for iq3, u will have to go iq2 and thus the output will suffer. At the plus side, it will spew faster than iq3.
Iq3xxs fits barely on my 10+64gb, so much that I am agresivelly cashing checkpoints to disk instead of keeping them in memory, moved vision from ram to disk too etc, I am sitting at few gigs left RAM and few hundred MB free on gpu.
I use DwarfStar DeekSeek V4 Flash on my Mac Studio with 128gb of RAM and it's pretty good - decent TPS and coding ability. Apparently the V4.1 model can also be run by streaming weights from the SSD - I've not tried this yet because it results in a lower TPS, but eventually this version of 4.1 should also be usable like this.
expert streaming with cache-conditional routing is very powerful (you can run 4.1 Flash on two Strix Halos and a 9070XT at around 50-70TPS decode in ~full quality), and that setup might be able to at least do a few tok/s with it.
About half the t/s of an online provider, heat, noise, power consumption and on top of that you only get 75% of the intelligence? Sounds like a waste of time to me.
Half the token/s - sure, no way around beating big ass data centers.
Heat, noise and power consumption are actually quite ok - i'm running 2x3090s, capped at 250W and i haven't seen heat and noise being an issue, sitting right next to it. Might be helping that it's getting colder.
75% of the intelligence with 2x3090s, with a 5090 if you have it it's around 90% and these are first day builds too, typically people hone in and improve these all the time.
It's true local doesn't make sense value wise against online, but if you have the hardware, having "unlimited" tokens with no 5h limit, no weekly limit, is super nice.
As luck would have it, you can just pay the provider and use their service. Others may want to run locally for privacy reasons or just for the fun of it. I think at-home AI is incredibly awesome and I applaud the enthusiasts pushing it forward. Just because that's where it's at today doesn't mean it's the best we can do.
Plus, you know, this thread was about running AI on "lame consumer hardware". There are more power efficient options available.
I use only providers because I don't have the hardware, but if, and that's a big if, the only moat left for humans is our ability to steer and make bigger picture decisions with our personality, I think it makes sense not to want to send all that conversation data to a 3rd party.
Strata is intriguing and it definitely looks like there's something useful happening there with weight streaming at usable speeds, but I've not managed to have it serve a full answer without inference crashing before long. Pity.
Nvidia stopped manufacturing the 5090, so you and people like us cannot run these models on our computers and are forced to use American SOTA so they don't lose the game...
For some reason the providers for MiMo 2.6 Pro are all very slow. I'd much rather use something quick. DeepSeek was great for that.
I haven't tried MiMo ultraspeed.
Mimo from Xiaomi has safety refusals on things compltely unrelated to safety. It's good when it works but you need another model to continue the work when Mimo craps out.
It does. Even looking at cases where a both a better and a worse model accomplish a given goal the difference between a solution and a good solution is important. And will save you time by not having to iterate as much with it.
For this model in particular, even if it didn’t get stronger and it simply thought less that is a big win on its own. DS4 can be served at a hit tok/s, but it takes a while because it thinks for so long.
Really wild that Deepseek's model is 536+16B (passive+active) parameter model and its really comparable to Mimo v2.6 Pro which is the current open source SOTA model which is at (978+42)
This is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.
I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.
Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.
The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D
I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)
At risk of being blunt: Why the surprise that a model that activates 12B parameters is faster than one that activates 16B parameters which is faster than one that activates 42B parameters?
It feels like that would be the expected result, all else equal.
I think it's because of the fact that Deepseek at 16B can be comparable enough in tasks to Mimo with 42B
I am just extremely impressed and surprised by the fact at how small we are going. My dream of local inference is perhaps coming close.
Regarding 12B one (Nemotron), still, there are other models even comparing it to qwen 9b, its tps difference is: 435 tokens/s vs 68 tokens[0]
I am not sure if this is because of how AA does its calculations for the speed but my main point was that I am impressed by the speed improvements happening in general hardware. I remember this speed when it was only expected from cerebras or groq.
Also speaking of this I found K2 Horizon[1] which seems to have 1,061.5 tps at 23b and it seems to be really close to actual open source (sharing their training code, pre-training, post-training, checkpoints etc.), I am quite impressed by them as well! [And I think that I wouldn't have found them if it wasn't for their token speed perhaps]
I guess what I'm really saying is just that being impressed with a model's size and being impressed with a model's speed are roughly the same thing for similar architectures and hardware.
Yes you are correct about it, though aside from a minor differences within the architecture itself, its right that being impressed with token speed is being impressed with model's size.
So you are right about it and I think that I am just excited about how deepseek although having the same architecture as previous times is just able to make consistently good upgrades and just honestly impressed by it all :-D
The old DeepSeek V4.1 Flash has been my workhorse model for the past month (after running tests comparing w/ Qwen 3.8 Flash Next, GLM 5.3 Flash, MiMo 2.6 Flash (including the MOPD fix)) and I've run tens of billions of tokens through it and pushed it fairly hard in both private internal benchmarking and in my own workloads to figure out where it's strong/weak.
Partly because I run it on very fast local infra (200+ tok/s) and partly because I'm guaranteed it's not being changed on me, I've generally preferred it over proprietary subscription models for daily tasks, as it seems about as capable at mid-level tasks (I reserve Astra/Opus mostly for planning/review).
I've replaced the previous model with 1017, and while it's early, a few observations:
- It's CoT has changed dramatically. I find the heavy Astra distillation claims to be plausible
- It has strong benchmarks scores, and from what I've seen online it's much better at 3D games and such, but that's not so relevant for my work. On my internal evals it is more token efficient, but it actually doesn't score any higher than the previous model
- It's only been a couple days, but it does seem a bit more focused and reliable. DS4 0731 and DS4.1 have been a bit sloppy (as in not-careful, although they did tend to fall into Claudisms as well), and 1017 seems to be a bit sharper.
- I'm running it through some more challenging work, but I'm not sure if it's really at GLM-5.3 level, especially for cyber/RE/hardware poking, which GLM has excelled at.
200+ tok/s on local infra? That's crazy. How much did you pay for your hardware?
I find these local infra costs crazy heavy duty. I always assume these folks are getting their values worth or that infra is serving multiple people's requests at and justifying the cost. Like I once thought of buying a 48-64 GB RAM laptop (not even clsoe to what these models need) and I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access. So I assume the usage is also heavy duty, or if nothing then for privacy and safety and then I guess the price, if one can pay, will totally be worth it.
Looking ahead, do we think token costs will remain where they are today? Go down? Up?
Your investment in hardware is of course fixed, the token cost amortized over time.
I suppose the other question is, do we think down the road that the models capable of being run locally will improve?
Everything's been trending down so far, so until evidence shows otherwise, I'd assume it will continue that trend. IMO the best time to buy local infra will be when that trend begins to asymptote.
There is so much down pressure, I cannot imagine the token/s price going up. It has been going down and will continue doing so, we are far from a plateau IMHO
> I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access
The advantage is no more 5-hours or weekly session limit anymore. Also no chance your proprietary code ends up in someone repo.
API access does not have a 5-hour or weekly limit.
I run local models, too, but every discussion about these models uses the local quantized versions and the hosted versions interchangeably. They’re not interchangeable. The quantized versions you can run on mid-grade consumer hardware are a step down in quality. That might be fine or might not depending on your use case but it needs to be acknowledged. It’s not until you get into really expensive hardware setups that you can run something like DSv4 Flash at Q8, and you’re not breaking even on that relative to a cloud provider.
Keeping everything local is about the only reason to run local. The other discussions about saving money are missing something.
I still think only privacy is relevant to run on your own infra, never cost.
near.ai tries to solve this
That is literally my next sentence :D
> The advantage is no more 5-hours or weekly session limit anymore
That is not a concern with pay-as-you-go models like DeepSeek. It is stupid inexpensive.
You just need to imagine someone making big bucks in tech + being a little bit money ignorant, having a big house and a nice basement and cheap energy prices.
Than it just makes sense that they already do this.
But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision.
Unless you have solar and batteries there aren't cheap energy prices :).
I'm german, i pay $45 US Cents / 40 cents in Euro.
Sooooo :D
If the only variable is money, local models will never make any sense at these prices of infra/subscriptions.
For a lot of people, there are other variables though.
This has to be a distilled model? I don't see any route to 200 tokens/s without serious datacenter hardware
you mean quantized? it’s only got 16B active parameters so it can be pretty fast at decode.
sorry, yes, quantised
CoT = Chain of Thought (I had to look it up)
Thanks for the insights. If you edit your post to include the definition, I'll delete mine.
> because I'm guaranteed it's not being changed on me
This is something underrated. With an open weight model I can evaluate the performance for my use case, and know for a fact the model will stay the same over time. No arbitrary “model welfare” rules being introduced later on, no undocumented change to capabilities or performance
Somehow GPT 6.1 Sol (medium) is cheaper per task ($0.21 vs $0.26) and faster per task (2.9 vs 4.9 minutes) vs v4.1 Flash 1017 (max). I hope AA adds other reasoning effort results of Deepseek so it'd be more realistic to compare it with other models, because IMO using models at max effort is very unrealistic for most people, they then take a huge amount of time and tokens for a small performance uplift.
I wonder what AA's funding and affiliations are. I get the sense that they have their hand on the scale at least a bit.
Well yes, thats the entire business model of benchmarking companies. If you pay attention to AA, some models are on the leaderboard second they're released while others come days later, which to me implies that theres some level of coordination happening with them and a few labs.
https://fixupx.com/aarnogau/status/2076059071246193012
Yeah, this always bugged me. Then again, unlike American models, surely Chinese models are far too expensive for them to test them at every difficulty ;)
I don’t know about 1017 but according to my limited testing of 0910 and previous v4-flash-0731 on some of my use cases, max simply used a lot more tokens (almost twice as many IIRC) for little to no benefit over high. So yeah I really don’t see a reason to use max.
And not only your testing, I have seen that claim a lot of time online. I never tested it, though.
For DS41F (both the original and 1017) I've found low (25/100) reasoning to be the optimal for my tasks at least (max doesn't tend to better, and if you can't do it w/ low imo you should just switch to a smarter model). At low it's about 1/3-1/2 max's token count (and price and speed).
A bit like influencers of the benchmarking world. Milking it while all this trends. Are there financial/tie-up disclosures that are audited for benchmarks?
AA prices are always kind of hand wavy for open models. I mean if you rent a cheap gpu and run inference yourself...or heck, what about a cpu instance with a bunch of ram, you can do that with llama.cpp for under a $1/hour. Ok not that, so are we talking about the median spot price per token from a basket of inference providers? Which ones? They actually vary quite a bit, even on openrouter, for this particular deepseek checkpoint it's between $0.15 and $1.20 ... that's like 8x spread...
They have some math here https://artificialanalysis.ai/methodology but ok how are the prices weighted among the providers... and even if you do that 7:2:1 math it doesn't match the providers. Specifically the model in question: https://artificialanalysis.ai/models/deepseek-v4-1-flash-101... doesn't match the first listed provider https://deepinfra.com/deepseek-ai/DeepSeek-V4.1-Flash-1017
Do the math yourself, it's actually $0.08 not the AA claimed $0.10 (and that's the more expensive of the two pricing tiers). That's a pretty large error bar.
Also many of the providers are now doing "peak" and "offpeak" pricing (deepseek.com itself for instance) so it can differ dramatically. These "peak" times depend on where the service is most popular. You can basically ride "off-peak" pricing by cascading from global provider to global provider.
The majority of the time taken in an inference call is doing the inference so the extra latency of traveling across an ocean is actually a fairly small wall-clock tax here. People still seem to be partial to nearby servers for ai-related workloads but really, try some decent ones 10,000 km away, you probably won't notice.
I know AA thinks highly of the price metric (I've been to their events, even have their hat, they brag about it) but I have learned to ignore it as noise.
It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
But for closed-source models with only a handful of providers (sometimes just 1) great, the price is a very reasonable number.
For the open models a more useful thing would be like "100 tok/s spot price build" and then it's using pcpartpicker or something in some kind of partnership and can query a build for buying a machine right now that can achieve a certain benchmark number and maybe something for less interactive workloads like "5 tok/s". Of course getting some base unit of power cost in there might be important as well since the power bill is the majority of what you pay for when you rent a commodity gpu.
> It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
Almost no one is running DeepSeek V4.1 Flash on their laptop, so comparing based on API pricing seems perfectly reasonable to me.
well no of course not ... but they do this for qwen 3.8 27b, and decision models like this https://artificialanalysis.ai/articles/artificial-analysis-d...
this one is especially annoying ... i can run these on lots of hardware but there's vast differences in many aspects.
Whether I'm getting a good deal isn't what I'm thinking about if I'm getting garbage results. It's more about why am I wasting my time on the wrong tool.
I mean what I really want is a more reliable classic eval benchmark for the decision models that I can easily deploy, like the official ollama ones: https://ollama.com/library?sort=newest
perplexity has a 27b decision then there's a 3b from liquidai and another 600M one. Then there's one called kev and bespoke has nimble-v3. clef from cloudflare is there along with strands-decider from aws. Then there's laya and tev1 ... I mean the AA page seems to be the only real third party doing benchmark and evals on it but most of these are missing from it. It has a lot to be desired.
And you go over to HF and it's even more mysterious https://huggingface.co/models?pipeline_tag=text-classificati...
Maybe an agentic eval framework that tests anything that hits 100 downloads would be a decent weekend project...
Also there's like wildly different tasks. choices versus noul have way different performances and also if you are doing things like text classification such as using it for theme extraction, I mean this does work with nimble-v3 but d1 gives mostly noise and pplx-decider is in between and their performance degradation curves are dramatically different depending on the input-token length.
This is all going to be bitter lessoned eventually but I can't just sit on my hands and wait around 12 months.
So now I have a proxy that looks at the input size and do dynamic routing and sometimes fire off both and do ensemble weightings ... I mean why do I have to be clever here, this is a generic problem.
I had to discover all this myself and I really shouldn't have to do that...I guess I'm just expecting some quant results from AA that they aren't in the business of providing.
I was quite sure they use the API pricing (by default) for open weight models that have official providers? Deepseek does have their own API.
I think they need another version of the charts with "subscription prices" at least. Lots of people are not paying API rates.
this matches my experience tbh. 6.1 sol on medium is quite good at getting things right the first time without overthinking and i prefer it over xhigh or max usually. so i'm not surprised it ends up cheaper.
What I really want to know for models like this is if I can have them run on really lame consumer hardware (consider 128gb of ram, 2080ti) with extremely low TPS, but with reasonable enough intelligence that you can walk away and expect something to bake really nicely while you sleep.
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
"Model like this" not sute about ds v4.1 but qwen3.8 flash next is currently running on 10gb 3080 + 64 gb ddr5 ram at respectable 40-70t/s with 1200pp at max context
Indeed, I m surprised how little attention qwen-3.8.-flash-next seems to be getting while it's so well fitted for 128gb unified machines.
I have a 16GB 5060 and 32GB system. RAM - are there any recommended Qwen 3.8 Flash models worth running on this ? I am using Nemotron 30B A3B at the moment
Fwiw I think you can run 3.8 flash (next?) on your setup with SSD streaming.
There's this video (1) that I found, and my understanding is that flash next can be very efficient wrt ram needed, by using a "dictionary" (or something similar).
1 - https://youtu.be/IH8XmxiwliQ?si=lJPvYzi3mHNsiw8r
32 gb of ram is gonna be your blocker for iq3, u will have to go iq2 and thus the output will suffer. At the plus side, it will spew faster than iq3.
Iq3xxs fits barely on my 10+64gb, so much that I am agresivelly cashing checkpoints to disk instead of keeping them in memory, moved vision from ram to disk too etc, I am sitting at few gigs left RAM and few hundred MB free on gpu.
I use DwarfStar DeekSeek V4 Flash on my Mac Studio with 128gb of RAM and it's pretty good - decent TPS and coding ability. Apparently the V4.1 model can also be run by streaming weights from the SSD - I've not tried this yet because it results in a lower TPS, but eventually this version of 4.1 should also be usable like this.
expert streaming with cache-conditional routing is very powerful (you can run 4.1 Flash on two Strix Halos and a 9070XT at around 50-70TPS decode in ~full quality), and that setup might be able to at least do a few tok/s with it.
You sure can, https://github.com/lxe/dsx gets you this deepseek on 2x3090s and 128 gb ram at about 30 t/s
At around 75% of the full model intelligence
About half the t/s of an online provider, heat, noise, power consumption and on top of that you only get 75% of the intelligence? Sounds like a waste of time to me.
Half the token/s - sure, no way around beating big ass data centers.
Heat, noise and power consumption are actually quite ok - i'm running 2x3090s, capped at 250W and i haven't seen heat and noise being an issue, sitting right next to it. Might be helping that it's getting colder.
75% of the intelligence with 2x3090s, with a 5090 if you have it it's around 90% and these are first day builds too, typically people hone in and improve these all the time.
It's true local doesn't make sense value wise against online, but if you have the hardware, having "unlimited" tokens with no 5h limit, no weekly limit, is super nice.
I have not capped my 2x4090s so my room gets cosy warm.
But here my price per kwh is 40 cents (euro).
But if you integrated it with solar system and a heat exchange, this will be feasable
As luck would have it, you can just pay the provider and use their service. Others may want to run locally for privacy reasons or just for the fun of it. I think at-home AI is incredibly awesome and I applaud the enthusiasts pushing it forward. Just because that's where it's at today doesn't mean it's the best we can do.
Plus, you know, this thread was about running AI on "lame consumer hardware". There are more power efficient options available.
I use only providers because I don't have the hardware, but if, and that's a big if, the only moat left for humans is our ability to steer and make bigger picture decisions with our personality, I think it makes sense not to want to send all that conversation data to a 3rd party.
You can get something relatively capable and usable with your configuration and Strata: https://github.com/Niko1221/Strata
Strata is intriguing and it definitely looks like there's something useful happening there with weight streaming at usable speeds, but I've not managed to have it serve a full answer without inference crashing before long. Pity.
I dont think its strata problem, its been running on my system for days without any crash.
Nvidia stopped manufacturing the 5090, so you and people like us cannot run these models on our computers and are forced to use American SOTA so they don't lose the game...
Mimo 2.6 pro scores 46 and is half the price (though a lot less throughput). Im not sure why nobody has made the same fuss around it
Hmm if you just compare raw token prices on Openrouter right now, Mimo 2.6 Pro is roughly 4x the price.
Does deepseek v4.1 flash 1017 really take that much longer per task?
For some reason the providers for MiMo 2.6 Pro are all very slow. I'd much rather use something quick. DeepSeek was great for that. I haven't tried MiMo ultraspeed.
Mimo from Xiaomi has safety refusals on things compltely unrelated to safety. It's good when it works but you need another model to continue the work when Mimo craps out.
Just want to note that this model still does not work (read: sucks) for recognizing sheet music.
The only model that works for me is Astra, but it's expensive.
does the exact model matter anymore? pick one and develop context with it, imho, and just get to work.
then again, I'm not a heavy AI user, so maybe I'm missing something.
It does. Even looking at cases where a both a better and a worse model accomplish a given goal the difference between a solution and a good solution is important. And will save you time by not having to iterate as much with it.
For this model in particular, even if it didn’t get stronger and it simply thought less that is a big win on its own. DS4 can be served at a hit tok/s, but it takes a while because it thinks for so long.
Really wild that Deepseek's model is 536+16B (passive+active) parameter model and its really comparable to Mimo v2.6 Pro which is the current open source SOTA model which is at (978+42)
This is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.
I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.
Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.
The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D
I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)
high tps and braindead model is a dangerous and fun thing to watch.
give it browser access and it's a speed-run to a cloudflare bot detection freak-out in minutes.
At risk of being blunt: Why the surprise that a model that activates 12B parameters is faster than one that activates 16B parameters which is faster than one that activates 42B parameters?
It feels like that would be the expected result, all else equal.
I think it's because of the fact that Deepseek at 16B can be comparable enough in tasks to Mimo with 42B
I am just extremely impressed and surprised by the fact at how small we are going. My dream of local inference is perhaps coming close.
Regarding 12B one (Nemotron), still, there are other models even comparing it to qwen 9b, its tps difference is: 435 tokens/s vs 68 tokens[0]
I am not sure if this is because of how AA does its calculations for the speed but my main point was that I am impressed by the speed improvements happening in general hardware. I remember this speed when it was only expected from cerebras or groq.
Also speaking of this I found K2 Horizon[1] which seems to have 1,061.5 tps at 23b and it seems to be really close to actual open source (sharing their training code, pre-training, post-training, checkpoints etc.), I am quite impressed by them as well! [And I think that I wouldn't have found them if it wasn't for their token speed perhaps]
[0]: https://artificialanalysis.ai/models/comparisons/nemotron-3-...
[1]: https://artificialanalysis.ai/models/k2-horizon-375b-a23b & https://ifm.ai/blog/k2/
I guess what I'm really saying is just that being impressed with a model's size and being impressed with a model's speed are roughly the same thing for similar architectures and hardware.
Yes you are correct about it, though aside from a minor differences within the architecture itself, its right that being impressed with token speed is being impressed with model's size.
So you are right about it and I think that I am just excited about how deepseek although having the same architecture as previous times is just able to make consistently good upgrades and just honestly impressed by it all :-D
It's too bad they couldn't make it smaller, but still very nice improvement.
The new model:
DeepSeek V4.1 Flash 1017 (Max) 552B A16B
Matches
GLM 5.3 753B A40B
MiMo 1T A42B
Nicely done!
I'm running it at 3 b.p.w. and it really feels better than Flash 4 at 4 b.p.w. for me. Some of these EXL3 calibrated quants are like magic.
It also has 196B ngram params that sit on disk. So it's really pretty comparable to GLM 5.3 in size.
my daily model now