Amazing to me that something as wild as this is now basically a trope.
I don't mean this negatively; open weights AI and open source tooling is about the only thing that gives me hope.
The fact that there are dozens of these things hints at the beginning of the dull normalisation of openly available, non-monopoly AI tools, which is what is needed if we are all to survive.
The obvious thing that I can coming see is high bandwidth flash, which as far as I know is under active development by multiple companies.
From what I've read one architecture is basically RAID0 at the flash chip/chiplet or PCB level, combining many fast flash chips into one wide channel and then turning that into a wide fast streaming storage system. Fast as in competitive with DRAM, at least for streaming a lot of data, which is what most AI inference wants.
Flash, even very fast flash, is much cheaper than DRAM per GB.
I'd like a specialized local inference node consisting of 2TB high bandwidth flash, 16-32GB of DRAM (for attention and other state), and a big TPU. That could run frontier size models in something the size of a large mini PC like an Intel Nuc. The lower cost of flash would probably move most of the cost to the processor(s).
I wonder how feasible it would be to DIY a board layout for basically RAID0 flash that's pretty good? A bunch of flash chips in parallel. Use something like a cheap fast ARM or RISC-V chip as the flash controller and have it do striping and present itself as an abstract drive over a thunderbolt or PCIe interface.
Currently, the fastest way to connect flash is via PCIe 5.0 x4. Right?
That's ~16 GB/s. Even if you do RAID0 of 4 such drives, that would get you 64GB/s. That's not a lot. Compare that to 1792 GB/s for RTX 5090: it's 100x faster than NVMe flash.
If you have a 16-lane PCIe 5.0 card with 4 M.2 slots on it you get 64 GB/s from the 4 SSDs installed on it, the same as the speed of transferring data to a discrete GPU.
Desktop CPUs support only one such slot (normally used for the GPU), together with another PCIe 5.0 M.2 and one independent PCIe 4.0 M.2. That sums to 88 GB/s for a consumer motherboard. A beefy CPU like an AMD 9950X should be sufficient for AI inference limited by the SSD throughput, even without a discrete GPU.
On a server MB, you may have e.g. 6 16-lane PCIe 5.0 slots, so you could use one slot for a discrete GPU and 5 slots for SSDs, for 320 GB/s throughput. At this point the price would be quite high even with the smallest available PCIe 5.0 SSDs, but it would still be low in comparison with the price of datacenter GPUs or of an equivalent amount of DRAM.
That's why I think you'd have to do something board-level at least, not just plugging existing modules or SSDs together. You'd have to stuff flash chips on a novel board with a novel controller architecture.
The work I've seen at places like Samsung and Intel is chip level.
Why hasn't this been done already? No market until recently. Existing flash is fast enough for SSD use cases, and making it fast enough to compete with DRAM makes it more expensive. Meanwhile it's not a true DRAM replacement because it's slow to write, slower than DRAM for true random access, and can wear out with many write cycles. It only makes sense for applications that are read-mostly, streaming, and require huge data, and that was a small market until AI.
Correct, but that's comparable to streaming from CPU memory to GPU memory, which seems to work OK for MOE models. Then again, if model sizes are so large that you need SSDs, maybe not so much.
I can't justify this at all, but it feels like there's headroom in the currently-available hardware, which the current shortage will increasingly force us to optimise into. Then when the choke-point relaxes the upper bound of what's achievable will be even higher than it would have been.
There's plenty of optimisations still possible if you optimise for specific GPU families. Eg see Hipfire[1] and q38rocm/HaloPFX[2], they're so much more faster than llama.cpp for consumer AMD GPUs.
OK, yes, this I do agree with but I was mostly factoring these in (because I did that research a couple of weeks ago while speccing out a hypothetical R9700 box)
One of the architectural changes expected in the next series of Qwen models appears to be well suited to SSD streaming.
But also, there probably still is a lot of room for tuning and better utilizing the streaming capability, e.g. better speculative streaming to have the activated experts loaded in RAM by the time they're needed.
Same as big data centers siphoned up all the SSDs before their prices became reasonable for consumers, same with HDDs, or RAM every generation. They’ll be high before they’ll be reasonable, and then they’ll be cheap once their next generation comes along.
Yeah, started in the late 70s (1978) with the StorageTek STC 4305, which was 45MB in a large cabinet, cost $400,000, and was from what I can find the first SSD. From there up until the 2000s they served as expensive but crucial networking infrastructure, when in the early-mid 2000s the industry moved from RAM to NAND, and the price-per-GB of NAND became reasonable enough to enter the consumer space as a proper form of storage drive, and not just for USB sticks or memory cards.
For comparison, here is the rough trajectory of price:
1978: $9.1 million/GB
1985: $920,000/GB
1990: $50,000/GB
1995: $1,000/GB [0]
2005: $100/GB
2006 is when the first commercial SSDs hit market in laptops, and by 2010 they were ~$3/GB, which brought them to wide adoption.
[0]: 2000 was a stagnant year, prices stayed the same from my research between ‘95 and the new millennium before dropping in 2005.
I have one of my own. My reason for this is that you can get up to 2x performance compared to llama.cpp on the same model weights. It is optimized for single model to serve it well.
I did, the sd.cpp take too much time on my laptop to generate image, the issue is known and it did not seem like it gonna be resolved, so I build my own, and that's why https://github.com/netdur/qwen-image-cplus
llama.cpp has strict policies on AI-assisted contributions. Most of these specialization projects are done by coding agents. So llama.cpp won't take them (which is fair).
llama.cpp is a generalized inference engine; therefore, model-specific optimizations should remain outside its core architecture. When developing a bespoke inference application, it is standard practice to fuse operations and heavily optimize the compute kernels for the specific target model.
Most of these custom engines exist because the more general purpose projects make contributing much harder than maintaining a custom engine (somewhat understandably). Consequently they take forever to adopt optimizations, and it is somewhat pointless if llama.cpp or vllm only reach performance parity with custom engines when the LLM architectures being optimized for are no longer the best ones.
The benefit of llama.cpp etc is just ease of use for people not already familiar with the local AI landscape.
If you had read TFA, you would have seen that the reason is obvious.
The speed obtained by this optimized program is many times higher than the current speed reached by llama.cpp when reading weights from an SSD.
It is unlikely that llama.cpp will include any time soon code capable of reading SSDs at the speed limited by hardware.
If the SSD files are memory-mapped, you need to provide at appropriate points in the program the correct madvise hints (the Linux-specific advices, not the useless POSIX advices) to free the unused memory and populate the memory that will be soon used, in such a way as to guarantee that there will never be any page faults while reading the weights.
Otherwise, you must used asynchronous I/O with O_DIRECT, as in this project, which requires a significant reorganization of the program in comparison with using memory-mapped files, to guarantee that the desired data is always available at the right time and the inference computation overlaps perfectly with reading continuously from the SSDs.
> Using a GPU has, once, caused a driver reset that took the desktop with it.
Only once? This has happened to me so many times with LLM-assisted development. I was writing an OCR engine and pathological shapes would cause OOM that would take the display out. On other occasions, the LLM would try to load 24GB of bf16 weights onto a 16GB GPU without understanding the consequences.
Have to encode this in logic: keep some space reserved for the display, and check before allocation. But some errors are bound to slip through.
---
Inspired by DwarfStar4 and Strata, I am trying to build my own gemma4-specific inference engine designed to run on gfx1201.
It is a lot of work. You cannot simply vibecode one in two days.
It will be when done. I have a lot of stuff in the works and switch between them based upon my current interest. I like plotting and writing stories and Gemma is one of the nicer writers.
The goal is to get the 31B working on a 16GB card. Which means weights have to be < 4 bits. The 9070 XT has wmma support for int4. Which doubles compute performance on pp compared to i8/fp8. tg is limited by bandwidth.
Not sure if the measurement method used for this is actually valid or reliable enough, but for reference, Codex 6.1 Sol Standard runs at 19.9 tok/s according to a benchmark I recently saw: https://www.reddit.com/r/codex/s/b4mUszSOv1
The numbers you povided are for output/generation. Input is around 5-20 thousand for Astra, and 16-60k for Luna. So a few hundred times faster.
Most of the tokens consumed by agents is reading stuff - docs, tool outputs, code etc. Previewing a content of a dumb single page of html is a few thousand tokens.
If you use claude or some other harness you can ask it how many cache writes/input tokens are there in avg session, and how much time it would take if preprocessing speed was just 60 tok/s.
I’d say 2-4 years to make it feasible, but still much slower than datacenters. Custom chips may change the equation.
But because prefill is parallelizable, datacenters will have an edge here for a while. For your one request, a datacenter may push 1.7kW power to a single machine to give you a result in a few seconds. The rest of the time machine will stay busy with other people’s traffic so it’s not a problem. For you, unless you’re a billionaire, you won’t have 1.7kW of processing power running idle 99.9% of time.
In 1-3 years we’ll have datacenters switch to dedicated chips that are way more energy efficient, and such chips will ultimately trickle down to PCs.
BTW - Apple is already introducing LLM specific architecture. They introduced Neural Accelerator (nit to be confused with Engine) that speeds up their GPUs 4x for LLM specific tasks. M5Ultra gets 100 tflops vs nvidia Rtx6000 getting 500tflops. „Just” 5x faster, but it’s beginning to look relevant.
Still 25 slower than a server farm B200. And when you ask Chat/Claude you get the full speed of B200.
What I’m waiting for is consumer dedicated LLM boards, a similar way we had soundcards and first gen 3d accels. No GPU, no 3D, purely LLM compute. I can see that happen in 2-3 years.
What you said about the RTX 6000 was kind of my point. Local AI is feasible today, but only few people will be willing to drop 20k+ on a graphics card. I'm less optimistic than you on the timing of the trickle down, but I hope you're right! :)
I would guess the assumption is that "agentic work" should be happening asynchronously in the background on a machine that's also used for foreground interactive work, where the slower the inference is, the longer the machine's GPU and [V]RAM are "hogged" by it at a time, and so the more the machine's performance seems to degrade.
I have a couple of V100 and am working on making Flash Next really fast on them. So far I am at like 193 t/s decode on a specialist 3-bit quant. On shorter prompts. Around 109 t/s on a ggml/llama fork at depth before I gave up and started writing the engine from scratch. I guess I will end at about 220-250 t/s at 128k context size with about 3K prefill on these cards.
https://stealthgoat.com/deepswe-flash-next.html (Total AI pile). I benched this quant for 3 days on DeepSWE in a 16GB VRAM gaming PC. It held 90 t/s on avg I think. It’s solid. The authors claims of minimal intelligence loss hold. Sorry it renders bad in mobile will fix it soon.
It feels great to have a local model ripping at these speeds. I now use Flash NeXT as an active agent managing part of the engine build process and let it do some coding on easier chores :)
What’s the practical use for you? 300/s decode is still way too slow for agentic behaviour, but I can see some niches - processing documents in background, semantic search etc
There are many of these stream-experts-from-SDDD projects now. What's needed is a guide to all of them and maybe for someone to take the best features of each of them and merge them into one uber project.
My impression based on a cursory look at the research is that the caching algorithm is absolutely vital, non-obvious, model-dependent, and potentially prompt-dependent to make sure you're holding onto the right experts.
This could be a game changer for me. The best I've been able to do so far is ~8 tok/s with Qwen 35B A3B, 32GB RAM. But then again I just got an OpenCode Go plan yesterday and have yet to figure if I can even consume my quota on DeepSeek Flash as the need to plan things well is bottlenecking me. Might be a while before I try the local thing again.
I think this is still the tip of the iceberg in terms of this method.
There is no particular requirement for this to be limited to a single SSD.
What happens when someone makes a GPU Card with enough SSDs attached to feed every RAM chip individually.
Based on how vibe-coded this whole thing is, I’m guessing the newer models weren’t in the training data and the agent just picked what it knew existed.
I wonder when we will start seeing small commercial accelerators integrated with NVMes; could potentially use far more bandwidth than typical NVMe drive can push thru the four lanes.
Given that you'd have probably a lot more capacity just to get to required bandwidth it could be used to switch very quickly between models (as all stored there would essentially be "loaded")
Seems like a useful innovation on the desktop level, but on the server level, isn't it already pretty normal to see machines with 24x high-performance (i.e. individually high-bandwidth) NVMe drives in them, that can then be used in software RAID0 to achieve pretty massive read bandwidth?
Well, it's not my field, but pci5.0 x4 drives are the fastest I found with a quick Google. So 24 of those would top out at ~400 GB/s. That's also where HBF starts out, for one device. The higher grades are faster, but maybe NVMe will be as fast by the time they are out.
But the win here on servers is almost certainly power. High speed signals are power hungry, and the power scales with the length of the wire. HBF is designed to go in the same package as the GPU, so the distance is much shorter than with discrete drives.
I think fits in a RAM is the first step, a real practical question is if in the mean time you can still use the machine for basic tasks like use browser watch YouTube/VS Code etc.
PS: There's heat you've got to worry about a bit, especially in very high speed drives, but that's about it (practically speaking). Bit rot prevention consumes some write cycles too but that is minuscule in comparison.
Amazing to me that something as wild as this is now basically a trope.
I don't mean this negatively; open weights AI and open source tooling is about the only thing that gives me hope.
The fact that there are dozens of these things hints at the beginning of the dull normalisation of openly available, non-monopoly AI tools, which is what is needed if we are all to survive.
Hardware is becoming the choke-point. If that gets resolved, things will start to change even faster
The obvious thing that I can coming see is high bandwidth flash, which as far as I know is under active development by multiple companies.
From what I've read one architecture is basically RAID0 at the flash chip/chiplet or PCB level, combining many fast flash chips into one wide channel and then turning that into a wide fast streaming storage system. Fast as in competitive with DRAM, at least for streaming a lot of data, which is what most AI inference wants.
Flash, even very fast flash, is much cheaper than DRAM per GB.
I'd like a specialized local inference node consisting of 2TB high bandwidth flash, 16-32GB of DRAM (for attention and other state), and a big TPU. That could run frontier size models in something the size of a large mini PC like an Intel Nuc. The lower cost of flash would probably move most of the cost to the processor(s).
I wonder how feasible it would be to DIY a board layout for basically RAID0 flash that's pretty good? A bunch of flash chips in parallel. Use something like a cheap fast ARM or RISC-V chip as the flash controller and have it do striping and present itself as an abstract drive over a thunderbolt or PCIe interface.
Currently, the fastest way to connect flash is via PCIe 5.0 x4. Right?
That's ~16 GB/s. Even if you do RAID0 of 4 such drives, that would get you 64GB/s. That's not a lot. Compare that to 1792 GB/s for RTX 5090: it's 100x faster than NVMe flash.
If you have a 16-lane PCIe 5.0 card with 4 M.2 slots on it you get 64 GB/s from the 4 SSDs installed on it, the same as the speed of transferring data to a discrete GPU.
Desktop CPUs support only one such slot (normally used for the GPU), together with another PCIe 5.0 M.2 and one independent PCIe 4.0 M.2. That sums to 88 GB/s for a consumer motherboard. A beefy CPU like an AMD 9950X should be sufficient for AI inference limited by the SSD throughput, even without a discrete GPU.
On a server MB, you may have e.g. 6 16-lane PCIe 5.0 slots, so you could use one slot for a discrete GPU and 5 slots for SSDs, for 320 GB/s throughput. At this point the price would be quite high even with the smallest available PCIe 5.0 SSDs, but it would still be low in comparison with the price of datacenter GPUs or of an equivalent amount of DRAM.
That's why I think you'd have to do something board-level at least, not just plugging existing modules or SSDs together. You'd have to stuff flash chips on a novel board with a novel controller architecture.
The work I've seen at places like Samsung and Intel is chip level.
Why hasn't this been done already? No market until recently. Existing flash is fast enough for SSD use cases, and making it fast enough to compete with DRAM makes it more expensive. Meanwhile it's not a true DRAM replacement because it's slow to write, slower than DRAM for true random access, and can wear out with many write cycles. It only makes sense for applications that are read-mostly, streaming, and require huge data, and that was a small market until AI.
Correct, but that's comparable to streaming from CPU memory to GPU memory, which seems to work OK for MOE models. Then again, if model sizes are so large that you need SSDs, maybe not so much.
I can't justify this at all, but it feels like there's headroom in the currently-available hardware, which the current shortage will increasingly force us to optimise into. Then when the choke-point relaxes the upper bound of what's achievable will be even higher than it would have been.
I don't think there's much room that isn't being exploited by this SSD expert streaming stuff.
There may well be headroom within the software/architecture of LLMs but once you get down to streaming tokens, it is kind of running into the physics?
There's plenty of optimisations still possible if you optimise for specific GPU families. Eg see Hipfire[1] and q38rocm/HaloPFX[2], they're so much more faster than llama.cpp for consumer AMD GPUs.
[1] https://github.com/warpfront/hipfire
[2] https://github.com/julianmb/q38rocm
OK, yes, this I do agree with but I was mostly factoring these in (because I did that research a couple of weeks ago while speccing out a hypothetical R9700 box)
One of the architectural changes expected in the next series of Qwen models appears to be well suited to SSD streaming.
But also, there probably still is a lot of room for tuning and better utilizing the streaming capability, e.g. better speculative streaming to have the activated experts loaded in RAM by the time they're needed.
There is a new standard for High Bandwidth Flash that is under development.
> Asianometry: High Bandwidth Flash Is Coming
https://www.youtube.com/watch?v=3nTpW52nioI
The problem is will the big ai companies siphon it all up before consumers even get a chance?
Same as big data centers siphoned up all the SSDs before their prices became reasonable for consumers, same with HDDs, or RAM every generation. They’ll be high before they’ll be reasonable, and then they’ll be cheap once their next generation comes along.
Did data centre SSD rollout really happen significantly before consumer SSD rollout?
I'd always assumed that the first generation of SSDs were really tested on consumers.
Yeah, started in the late 70s (1978) with the StorageTek STC 4305, which was 45MB in a large cabinet, cost $400,000, and was from what I can find the first SSD. From there up until the 2000s they served as expensive but crucial networking infrastructure, when in the early-mid 2000s the industry moved from RAM to NAND, and the price-per-GB of NAND became reasonable enough to enter the consumer space as a proper form of storage drive, and not just for USB sticks or memory cards.
For comparison, here is the rough trajectory of price:
1978: $9.1 million/GB
1985: $920,000/GB
1990: $50,000/GB
1995: $1,000/GB [0]
2005: $100/GB
2006 is when the first commercial SSDs hit market in laptops, and by 2010 they were ~$3/GB, which brought them to wide adoption.
[0]: 2000 was a stagnant year, prices stayed the same from my research between ‘95 and the new millennium before dropping in 2005.
Not all of it, but certainly enough to keep the price high.
Dibs on the name 'DeltaAccel'
Every few weeks, someone stumbles on what llama.cpp has had for years again, and reimplements it. I'm not sure why.
I have one of my own. My reason for this is that you can get up to 2x performance compared to llama.cpp on the same model weights. It is optimized for single model to serve it well.
I did, the sd.cpp take too much time on my laptop to generate image, the issue is known and it did not seem like it gonna be resolved, so I build my own, and that's why https://github.com/netdur/qwen-image-cplus
Have you considered contributing upstream?
llama.cpp has strict policies on AI-assisted contributions. Most of these specialization projects are done by coding agents. So llama.cpp won't take them (which is fair).
No, it doesn't. AI contributions are welcome if you understand the code. Ping me directly if you want to have something upstreamed.
> if you understand the code
Well, I don't mean to sound rude or anything but that can be a lot to expect from maintainers of those projects.
As I said, it's better for projects like llama.cpp to be selective in this regard.
Well even that is okay. I can use my agent to extract something useful from it.
llama.cpp is a generalized inference engine; therefore, model-specific optimizations should remain outside its core architecture. When developing a bespoke inference application, it is standard practice to fuse operations and heavily optimize the compute kernels for the specific target model.
Most of these custom engines exist because the more general purpose projects make contributing much harder than maintaining a custom engine (somewhat understandably). Consequently they take forever to adopt optimizations, and it is somewhat pointless if llama.cpp or vllm only reach performance parity with custom engines when the LLM architectures being optimized for are no longer the best ones.
The benefit of llama.cpp etc is just ease of use for people not already familiar with the local AI landscape.
Because with AI everybody can reinvent the wheel with ease
If you had read TFA, you would have seen that the reason is obvious.
The speed obtained by this optimized program is many times higher than the current speed reached by llama.cpp when reading weights from an SSD.
It is unlikely that llama.cpp will include any time soon code capable of reading SSDs at the speed limited by hardware.
If the SSD files are memory-mapped, you need to provide at appropriate points in the program the correct madvise hints (the Linux-specific advices, not the useless POSIX advices) to free the unused memory and populate the memory that will be soon used, in such a way as to guarantee that there will never be any page faults while reading the weights.
Otherwise, you must used asynchronous I/O with O_DIRECT, as in this project, which requires a significant reorganization of the program in comparison with using memory-mapped files, to guarantee that the desired data is always available at the right time and the inference computation overlaps perfectly with reading continuously from the SSDs.
Probably because its poorly documentrd.
it does it automatically. not sure what kind of documentation ppl need.
If i run ./llama-server -m huge-model.gguf it'll just work when the model is larger than ram? Ok
Qwen 3? (Jumanji) What year is it?
What a waste of time.
Strata already does this with Qwen3.8-Flash-Next: https://github.com/Niko1221/Strata
If it uses different methodology I'd be happy to see it on HN, even if it doesn't include the flavor of the month model.
It lists a bunch of models it supports, including Qwen3.8-Flash
I see zero mention of qwen3.8-flash. only two hits talking about qwen3.8-27b: https://github.com/search?q=repo%3Aprabanta-dev%2Fjanas+qwen...
Are we hallucinating now too, not just the bots?
We've always hallucinated
Ahh, shit, you're right. I misread it as Flash.
<50GB/s bit stream? What a waste of time. Buy more DRAM.
> Using a GPU has, once, caused a driver reset that took the desktop with it.
Only once? This has happened to me so many times with LLM-assisted development. I was writing an OCR engine and pathological shapes would cause OOM that would take the display out. On other occasions, the LLM would try to load 24GB of bf16 weights onto a 16GB GPU without understanding the consequences.
Have to encode this in logic: keep some space reserved for the display, and check before allocation. But some errors are bound to slip through.
---
Inspired by DwarfStar4 and Strata, I am trying to build my own gemma4-specific inference engine designed to run on gfx1201.
It is a lot of work. You cannot simply vibecode one in two days.
Is the inference engine open-source?
It will be when done. I have a lot of stuff in the works and switch between them based upon my current interest. I like plotting and writing stories and Gemma is one of the nicer writers.
The goal is to get the 31B working on a 16GB card. Which means weights have to be < 4 bits. The 9070 XT has wmma support for int4. Which doubles compute performance on pp compared to i8/fp8. tg is limited by bandwidth.
amd bros
If anyone needs to see what 62 tokens/second looks like: https://token.qusai.pro/?speed=62
Which seems sufficient for mortals who are not running gas town, but want a local model.
Not sure if the measurement method used for this is actually valid or reliable enough, but for reference, Codex 6.1 Sol Standard runs at 19.9 tok/s according to a benchmark I recently saw: https://www.reddit.com/r/codex/s/b4mUszSOv1
60 toks/sec may be ok for pure chat or token generation, but not for token preprocessing nor any sort of agentic work.
>.. nor any sort of agentic work.
may I ask why?
oai luna runs at 60-70 toks/s; astra runs around 35-45 , are you just worried that your agentic work can't spread across a lot of parallel work?
The numbers you povided are for output/generation. Input is around 5-20 thousand for Astra, and 16-60k for Luna. So a few hundred times faster.
Most of the tokens consumed by agents is reading stuff - docs, tool outputs, code etc. Previewing a content of a dumb single page of html is a few thousand tokens.
If you use claude or some other harness you can ask it how many cache writes/input tokens are there in avg session, and how much time it would take if preprocessing speed was just 60 tok/s.
Latency is the remaining hurdle for local AI. We'll get there, but my prediction is that it will take around 10 years.
I’d say 2-4 years to make it feasible, but still much slower than datacenters. Custom chips may change the equation.
But because prefill is parallelizable, datacenters will have an edge here for a while. For your one request, a datacenter may push 1.7kW power to a single machine to give you a result in a few seconds. The rest of the time machine will stay busy with other people’s traffic so it’s not a problem. For you, unless you’re a billionaire, you won’t have 1.7kW of processing power running idle 99.9% of time.
In 1-3 years we’ll have datacenters switch to dedicated chips that are way more energy efficient, and such chips will ultimately trickle down to PCs.
BTW - Apple is already introducing LLM specific architecture. They introduced Neural Accelerator (nit to be confused with Engine) that speeds up their GPUs 4x for LLM specific tasks. M5Ultra gets 100 tflops vs nvidia Rtx6000 getting 500tflops. „Just” 5x faster, but it’s beginning to look relevant.
Still 25 slower than a server farm B200. And when you ask Chat/Claude you get the full speed of B200.
What I’m waiting for is consumer dedicated LLM boards, a similar way we had soundcards and first gen 3d accels. No GPU, no 3D, purely LLM compute. I can see that happen in 2-3 years.
What you said about the RTX 6000 was kind of my point. Local AI is feasible today, but only few people will be willing to drop 20k+ on a graphics card. I'm less optimistic than you on the timing of the trickle down, but I hope you're right! :)
I would guess the assumption is that "agentic work" should be happening asynchronously in the background on a machine that's also used for foreground interactive work, where the slower the inference is, the longer the machine's GPU and [V]RAM are "hogged" by it at a time, and so the more the machine's performance seems to degrade.
My self hosted LLM runs at 60-ish tok/s and does great at agentic tasks. I think that’s about the threshold where it stops feeling annoyingly slow.
60tok/s prefill or tok gen? If you use gpu and keep it in vram then I you most likely mean token generation, nit prefill.
I have a couple of V100 and am working on making Flash Next really fast on them. So far I am at like 193 t/s decode on a specialist 3-bit quant. On shorter prompts. Around 109 t/s on a ggml/llama fork at depth before I gave up and started writing the engine from scratch. I guess I will end at about 220-250 t/s at 128k context size with about 3K prefill on these cards.
https://stealthgoat.com/deepswe-flash-next.html (Total AI pile). I benched this quant for 3 days on DeepSWE in a 16GB VRAM gaming PC. It held 90 t/s on avg I think. It’s solid. The authors claims of minimal intelligence loss hold. Sorry it renders bad in mobile will fix it soon.
It feels great to have a local model ripping at these speeds. I now use Flash NeXT as an active agent managing part of the engine build process and let it do some coding on easier chores :)
What’s the practical use for you? 300/s decode is still way too slow for agentic behaviour, but I can see some niches - processing documents in background, semantic search etc
Good luck when you need to prefill context..
There are many of these stream-experts-from-SDDD projects now. What's needed is a guide to all of them and maybe for someone to take the best features of each of them and merge them into one uber project.
My impression based on a cursory look at the research is that the caching algorithm is absolutely vital, non-obvious, model-dependent, and potentially prompt-dependent to make sure you're holding onto the right experts.
It doesn't look like a solved problem yet.
[flagged]
This could be a game changer for me. The best I've been able to do so far is ~8 tok/s with Qwen 35B A3B, 32GB RAM. But then again I just got an OpenCode Go plan yesterday and have yet to figure if I can even consume my quota on DeepSeek Flash as the need to plan things well is bottlenecking me. Might be a while before I try the local thing again.
Then check out those:
https://github.com/prabanta-dev/janas <- this, focuses on Qwen3-Next-80B-A3B.
https://github.com/Niko1221/Strata <- Qwen3.8-Flash-Next. I use this (tuned) and I cannot understate how satisfied I am.
https://github.com/antirez/ds4 <- DeepSeek 4 family + more.
s/understate/overstate/ I mean, I get up to 90 tok/s on consumer hardware (16 GB VRAM + 64 GB RAM) using IQ3_S, which so far is holding up quite well.
I'll have to check that out. Never gone that low in quants as I always assumed the quality would be pretty bad.
I think this is still the tip of the iceberg in terms of this method.
There is no particular requirement for this to be limited to a single SSD. What happens when someone makes a GPU Card with enough SSDs attached to feed every RAM chip individually.
Why are the models being used so old (in llm years anyway). Why no Qwen 3.8 ?
Based on how vibe-coded this whole thing is, I’m guessing the newer models weren’t in the training data and the agent just picked what it knew existed.
Qwen 3.6 & 3.8 have the exact same model architecture. In theory, it should just be a drop-in replacement.
https://hfviewer.com/compare/qwen3.6-27b-vs-qwen3.8-27b
I sure hope this path keeps getting taken to the point we can do effective things on an M5 MacBook Air with 16gigs of RAM ..
I wonder when we will start seeing small commercial accelerators integrated with NVMes; could potentially use far more bandwidth than typical NVMe drive can push thru the four lanes.
Given that you'd have probably a lot more capacity just to get to required bandwidth it could be used to switch very quickly between models (as all stored there would essentially be "loaded")
AMD Radeon Pro SSG, launched ten years ago at SIGGRAPH: https://www.tomshardware.com/news/amd-radeon-ssg-1tb-gpu-mem...
When High Bandwidth Flash comes out, most likely: https://chipsandcheese.com/p/hot-chips-2026-applying-high-ba...
Seems like a useful innovation on the desktop level, but on the server level, isn't it already pretty normal to see machines with 24x high-performance (i.e. individually high-bandwidth) NVMe drives in them, that can then be used in software RAID0 to achieve pretty massive read bandwidth?
Well, it's not my field, but pci5.0 x4 drives are the fastest I found with a quick Google. So 24 of those would top out at ~400 GB/s. That's also where HBF starts out, for one device. The higher grades are faster, but maybe NVMe will be as fast by the time they are out.
But the win here on servers is almost certainly power. High speed signals are power hungry, and the power scales with the length of the wire. HBF is designed to go in the same package as the GPU, so the distance is much shorter than with discrete drives.
I think what we're going to see first is essentially gpus with nvme (or at least flash) bolted on, not sure if that's also what you're saying or not.
I think fits in a RAM is the first step, a real practical question is if in the mean time you can still use the machine for basic tasks like use browser watch YouTube/VS Code etc.
I'm waiting for this idea to get generalized out to networks.
e.g. each expert lives on one node of the network and the model gets streamed.
Let's make ssds more expensive.
a.k.a. MSSDGA
I'm always excited to see more open source apps like this.
Zoomers discover swap?
it is probably lookahead / an oracle to prefetch
Couldnt engines already do this albeit super slow?
Honestly, I’m more worried about losing my SSD especially with where the prices are today. Pretty sure this is going to put a lot of strain on it.
Reading from an ssd is free
TIL! Thank you kind internet stranger.
PS: There's heat you've got to worry about a bit, especially in very high speed drives, but that's about it (practically speaking). Bit rot prevention consumes some write cycles too but that is minuscule in comparison.
Not for QLC.
would this work on a virtual disk, say w s3 or dropbox
You could prompt it before going to bed yes
Sure, it’ll just be absurdly slow.
Maybe next you can run an LLM off of some old MFM drives
Not online services. The overhead of TCP and latency I imagine would really lag performance. A SAN over SFP, maybe.
Wouldn’t that cause a lot of egress fees?
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
I think GitHub repos should have a new tag listing the model used to write it along with the token cost.
I don’t want to bother reading anything sub-Fable, and this one looks sub-Fable.