I like the in-depth description. Everything from the naming convention of the models (and how much RAM they require) as well as all the components needed underscores just how complicated this all still is.
I suppose I am waiting for AI-in-a-Box to come along so I can (painlessly) join in.
(I'm sure wrangling with all these esoteric aspects of LLMs though is fun for some people.)
Hehe I really am just working it out as I go along - I promise it is fairly painless. Hugging Face allow you to specify your machine and then browse models that fit. And then you can just vibe out 'oh this one is a bit slow let me try another' etc
No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minutes to run. It's free. The machine is just sitting there anyway. I have tried using qwen-coder and opencode on my M5 Max 128gb and compared to claude code it's painful. I did setup a workflow where claude plans, qwen executes (unattended overnight, again b/c it's slow) and then claude reviews. I benchmarked this several times and I ended up using MORE tokens with claude because it had to 'fix' all the qwen issues. While the code it produced was 'good enough' the fixes were worth it so I just stick to coding task using API models (codex and claude).
It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.
It sounds like they are doing something similar to what I described in my other post below. Personal media station.
That can be done on hardware that quite a lot of people basically just have and don't use 24/7 to the max - because it is their gaming machine or their programming and compiling workhorse, for example. Of course you are paying for additional electricity but even with napkin-math instead of a "proper" calculation, you are unlikely to pay more for running your own instead of something commercial (and that can be offset further with some of the "modern" electricity contracts and/or PV and battery storage). Especially if we are talking about a stack that runs most of/all the time when you are not using your machine and makes LLM calls regularly while running.
The work in software/admin to get whatever you want set up is similiar no matter which infrastructure you use.
true but if you're actually running k8s and similar workloads, chances are it might eat memory that LLM requires.
you'll also notice these articles rarely specify their context window in tokens, because it is small, usually 30k to 70k tokens and it gets slower as it fills up.
I actually have a Mac Mini M4 Pro with 48G. I gave the k8s example because this is what I was doing with it.
Was because I am back to using Linux as my workstation.
My Mac Mini is now a headless server for llama.cpp.
So, you are right that for these workloads , I would not be using the Mac Mini for k8s AND llama.
Another thing going against using a Mac for Linux containers is that there are no solutions that I know that properly manages memory : memory is given to the Linux vm , but never fluctuates if the needs in the vm are less than the initial request.
I know Orb Stack does that but is it proprietary. I think UTM does it , but not sure I would use UTM instead of Lima, Colima , multipass , etc to run containers.
> It's very possible that it costs you more than a cloud mode would
...which is almost always true in a single request/reply mode and never true in batch mode. Single request usually 2x-3x more expensive than cloud and batch mode 2x-3x cheaper. Now, for narrow tasks, a finetuned tiny 8b model would dramatically outperform SOTA frontiers for a fraction of price, esp. on energy efficient hardware like Apple.
Local is never cheaper than cloud because they can do batch inference, and that means you load model weights once to produce 128 tokens on 128 sessions in parallel not 1 token on 1 session like local models. Local models rarely get to high utilization factor, they spend most of their time waiting.
If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.
Local models can absolutely run in batch, what are even talking about?
> If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.
Even if you ran sequentally, single session, a _finetuned_ tiny (8B) local model on narrow tasks would abolutely mog SOTAs, any of it - Fable, Opus, Sol you name it.
> Local models can absolutely run in batch, what are even talking about?
I think the point was that if you aren't running your local machine at 100% for 24 hours a day then a cloud - with multiple clients - that is, will be more efficient.
Can you share a bit more about your bifrost and llama swap setup? I’m facing memory constraints and am looking for a managed model solution that will help with hot swapping loaded models and stay-warm concurrency. Ideally with prioritization.
What do you want to know? Just start llama-swap with the models i have downloaded, add llama-swap as a provider in bifrost, expose the models you want and they become available in one single endpoint you can use in anything like opencode, openwebui or anything that speaks openai.
Yes, that makes sense. Some of my models currently run in ollama while others require their own inference servers. I’m curious about custom inference servers in bifrost and the ability to orchestrate keeping some models warm in memory while evicting and cycling other models. All of which span different providers in bifrost (I think). Obviously I can get the mechanics from an agent, more wondering about any experience with something similar.
i have a 512gb ram m3 ultra mac studio setup with a gas city that runs one of my companies. today was the first time ever that a local model (GLM5.3 8-bit) was able to match fable5 in our tests.
GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds.
• 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill.
• Runs beside our whole agent city on one box with ~130 GB to spare.
• Review test: caught 6 of 6 planted P1 defects, zero false positives, same score as the frontier model we pay for.
• CRM test: 11 of 11 required records extracted, zero wrong writes, 45 minutes, first local model to clear the bar.
• Serving a 131k-token window today; the model itself supports 1,048,576. Widened to 4 concurrent slots and still have 50gb+ of excess ram.
granted my cto still isn't moving all of our inference to glm5.3 but we've identified 40%+ that is currently handled by fable that we're routing locally instead and will do concurrent requests to verify/compare responses for a while.
You still have electricity and capital investment. Envelope math suggests cheap electricity is costing you something like $0.50/mtok and the opportunity cost on the capital tied up and lost in the unit purchase and resale is going to cost you something like $2/mtok at 100% utilization (so, frontier model prices or higher at real utilization), and you don't benefit from any elasticity.
Hosted GLM 5.3 flash is like $0.15/mtok in $0.50/mtok out
Time to completion also must be considered. If I have to wait around for hours for a prompt to complete locally and I’ll need to iterate quickly, I’m better off hosted than local. If it’s “free” and slow it may just not be worth it.
Props to your parent commenter for including context size. Because 131k context window is prohibitively small for my coding workloads so I know a 512GB Mac won't cut it.
I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the performance is more than good enough. I'm not running the MLX version of the Gemma model, if I did the inference speed would likely be a bit better. I wouldn't use them for coding features though.
My perf sucks compared to yours. Added it to the post - same model averages 325 tok/s in processing prompts, and 34 tok/s in token generation. What am I doing wrong..?
Wow, that's just about half the perf. I'm not sure what you're doing differently, though our hardware is a bit different: I am on a Macbook Pro M4 Pro, while you're on a Mac Mini.
I would try a different version of the model from HuggingFace while ensuring it's MLX. I'm also using LM Studio, not oMLX, and I've seen some threads like these:
Some examples (keep in mind this is all indefinitely free for me, no burning quota away):
1. Getting information (such as information about hardware unfamiliar to me) when not connected to the internet, which happens occasionally in my case.
2. Continuing to learn Rust by way of toy examples, puzzles, and comparing aspects of various solutions, for example from LeetCode.
3. Reformatting data, for example from a PDF to a markdown table, or converting receipt images to text.
4. Simple translation/explanation (e.g. I'm teaching my wife one of the languages I speak but sometimes may not know/have the words to explain the full nuance of a translated word).
5. Summarization. One of the webnovels I'm reading has some very boring parts I don't want to slog through, in those cases I simply make the LLM summarize that part and move on.
Etc., you get the idea. It's not unusable for coding, but it would make many mistakes when making a whole feature and the context lengths are limited to around 30k-40k tokens by my RAM. I could give it access to the web but I simply use an online model when I need that sort of thing, again partly due to the context limit.
Edit: The MLX version of Gemma 4 26b a4b does about 62 tok/s.
I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B, or a pair of DGX Sparks running DSv4 flash, or better, 2x6000 RTX Blackwells. Those are the kinds of rigs that the local model enthusiasts are running. With the GPU setups, you’re looking at generally >100tps generation in single stream, and >10k tps of prefill, so it’s snappier than Claude code, which somewhat makes up for it being dumber.
That said, it is really cool to be able to run an LLM on eg a Mac laptop. Just not a better experience on almost any metric for interactive use than eg Claude Code, beside privacy and guardrails.
>I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B
How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed.
Not sure, I haven't run it, I've just been running DS V4 Flash non-stop since it came out, and that's replaced a lot of my Claude Code usage. People seem very impressed, though, it seems like it trades vram/world knowledge for extra thinking time, which I think is a good trade for local. tbf, I've heard luna's not great at coding. Fast and good for things like classifiers, summarization, though.
A friend and I were actually discussing today how benches show Luna Max at about par on coding with Sol Medium, but how it's nowhere near in reality. We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n, because variance in performance is killer with large coding projects. Consistency is what lets you actually build on this stuff.
It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
That's not even remotely close to being true, even once you account for capex. You have to look at the actual usage, look at the token limits. Even if you're paying Anthropic $200k/month for scale-tier, you're going to blow through your token limits trying to run max output 24/7. Three users running Opus 4.8 at max non-stop will probably clean your monthly allowance from daddy Dario in less than a week.
With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive. It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause. And you get the full month like that, your monthly token limit is the time in a month. That cluster, the electrical upgrade, the cooling setup, and the electricity to run it all costs less in 2 months than your maximum affordance from Anthropic does in the same time period. Two billing cycles, and realistically it's more like two weeks. In 4 quarters you've wasted over a million. Like, what are we talking about here?
Now if you aren't using AI all that much, which is perfectly valid, and especially if you aren't using it at its absolute maximum, the story changes. Because even though at that point you're not paying nearly as much in electricity to run the cluster anymore, you still have the $300k+ capex to get the setup in the first place. But if we're not redlining it non-stop, then we're not really talking about performance anymore, are we? If your org never comes close to hitting token limits, it's probably because AI is rather marginal for you. Which again, is perfectly valid. I don't even use AI professionally.
Fact of the matter is, if your corp can justify the capex for a cluster and makes heavy use of AI, you are literally burning money by not having one in your building. The numbers are painfully obvious. Even deepseek isn't as cheap. This is before we get into things like LoRAs, custom inference pipelines, etc. which you know are kind of important if you actually care about model performance.
> With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive.
Pretty expensive is an understatement. You couldn’t buy one of these if you wanted to right now. If you could it would be multiple hundreds of thousands of dollars.
> It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause
You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users. I don’t know how you think it’s going to run 8 of them at the same time. Did you mean 8 concurrent sessions?
Your math is way off across this post. If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months, it wouldn’t be some little secret that we only discover in a comment online.
> Pretty expensive is an understatement. [...] If you could it would be multiple hundreds of thousands of dollars.
Obviously, I quantified both the operating expense and the capital expense in my post. What I find curious is that you're quoting me talking about the operating expenditure, and changing the topic to be about the buy-in like these are interchangeable things. You don't think that this is a crucial and important distinction?
> You couldn’t buy one of these if you wanted to right now.
You could have spent all of 5 seconds of searching rather than just assuming[1]. You're not buying an Nvidia Superpod™.
> You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users.
That's certainly fair a point. Although in the English language, especially in legal contexts, the multi- prefix is used inclusively for fractional values. That is it's strictly >1, not >=2. IE an 18 month contract is a multi-year contract, or a $1.6 million dollar asset is a "multi-million" dollar asset. But this is uninteresting semantics.
You are right, but it also doesn't matter. The gap is just that big. You can run 1 single user of Kimi K3 and still not even come remotely close to the $70k or so that a single Opus 4.8 user can burn over the course of a month on left on max. An honestly lowballed amount I know from anecdote. The per-token cost is just really expensive.
> Your math is way off across this post.
You made one technical point above, one that doesn't ever arrive at a relevant rebuttal to the substance of my post. But please, I'd love to hear you elaborate, especially because I didn't actually give much math at all.
If you want math though, here's the math. Let's say you are paying a ridiculous amount of money for electricity, a price nobody in the US pays -- $2 per kilowatt hour. That's about 5x the average rate in California, 4x as in Hawai'i. 17kW @ $2/kWh * ~8766 hours in a year puts that cluster's electrical costs at just shy of ~$298k annually assuming it takes no breaks. Let's make matters worse and round that up to $300k. It's also assuming you didn't invest in a solar hookup for your building, which I don't know why you haven't at this point, especially if you're installing a CDU for your new cluster. 12 months of Claude burning $70k a month is $840k. For a buy in of, you know what, let's call it $500k. Why not? It still doesn't matter. The operating cost is so much lower it's paid for itself plus an additional $40k in the first year. Even at a ridiculous penalty in electricity that nobody pays, even overinflating the amount of money you'd pay for the cluster and the infrastructure to get it set up, it's not even remotely close for a single user where the gap is smaller (IE, you're not wasting "a million dollars" in a year by maxing out the $200k scaling limit every month)
You can of course trot out the point that oh, in 12 months this setup will be extremely outdated! It doesn't matter. If the work it was doing today was useful, it will be useful next year too. And with the rapidly encroaching diminishing returns from parameter scaling, you're probably going to be just fine for a while. Maybe grab a quantized version of a newer Chinese model at the end, before grabbing a newer generation of AMD node. Those MI400s are looking pretty sweet after all.
> If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months
If you're locked in, then you're locked in. But don't pretend like you're saving money. You're not.
> it wouldn’t be some little secret that we only discover in a comment online.
Why does this have you so nasty and defensive? It's not a "little secret" that running your own infrastructure is cheaper. Of course it is. You know what else is cheaper? Owning your own office building out in the sticks, rather than leasing part of one in the city. Not everybody can make that work, there are no free lunches after all.
History repeats, these same exact lines were rolled out ad nauseum during the cloud craze. Datacenters are businesses, not charities. Frontier companies rent quite a fair amount of their infrastructure. Even if they resold that compute below cost (they don't), there's a pretty steep cliff before the economics start to look attractive.
> You could have spent all of 5 seconds of searching rather than just assuming[1].
I guarantee this will not ship to you any time soon.
The current lead time on these GPUs in measured in years. If you didn't place an order for this a long time ago, it's not coming this year.
Being able to add it to an online configurator does not mean anything right now.
> 12 months of Claude burning $70k a month is $840k
Your math is completely useless with these arbitrary numbers pulled out of the air.
If you want to begin calculating payback period you'd need to look at token costs, cost per task, utilization rates, and so on.
> The operating cost is so much lower it's paid for itself plus an additional $40k in the first year.
You went from paying back in a couple months to paying back in a year but you still haven't even talked about tokens or concurrency.
You're also neglecting the fact that hosted tokens are going down in price at a rapid rate. If someone was paying $70K per month in tokens for Opus this month, that same level of compute is going to be much cheaper 12 months from now.
> Why does this have you so nasty and defensive?
Not nasty or defensive, just tired of these armchair claims that it's easy to go out and buy an 8 X MI355X box from people who obviously have no idea what the hardware lead time is like right now, or who haven't considered the actual math on token costs and payback times. You're still making a lot of claims without a single discussion of cost per task or token.
It usually boils down to people trying to convince themselves that keeping their macs hot and with very little ram to spare only to get sub 50 tokens per second on a subpar lobotomized (quantized) model is worth it.
And I'm not even considering their time spent fiddling, fine tuning configs to adjust for ram, updating/benchmarking models, etc. Which is probably more expensive than the mac so the math is even more wrong.
> I guarantee this will not ship to you any time soon.
The assumption, the starting point, is that you have a line on the hardware. Asking around, some distributors have a 6 month lead time on Instinct GPUs, which curiously enough is about how long you'll be twiddling your thumbs waiting for the cooling loop to be put in. Yes things take time.
> Your math is completely useless with these arbitrary numbers pulled out of the air.
Your dismissal is worthless if you can't even be bothered to provide a counter-example. You've not provided a single iota of quantified reasoning beyond my original not accounting for the space used for the context of concurrent users.
> If you want to begin calculating payback period you'd need to look at token costs, cost per task, utilization rates, and so on.
Now go back and carefully reread my original post. Yes, if you are not actually redlining an LLM for a billing cycle, the capex starts to be way more relevant for this setup. Otherwise, our constraint is time and our unit of measure is $/hr.
If you want to compare token cost, it may shock you to learn that Kimi K3 without speculative decode on this setup is slightly under twice as fast as Opus 4.8 max. That's still true when fast is compared with K3 with speculative decode, and now Claude is twice as expensive as a base rate. Oops. We're already burning more money over a period of time, looking at tokens we're screaming even further ahead.
> You're also neglecting the fact that hosted tokens are going down in price at a rapid rate.
Cool. Call me when Opus 4.8 max is $0.50/million. In 4 years you could have bought the 200 acres of land down the road from your building, started a 5MW solar farm subsidiary that you'll expand over time, and as soon as your connect is up, dropped the opex of the cluster down to its maintenance costs. That subsidiary will pay the loan required to spin it up back irrespective of your primary business. When you own your own shit, you can play your own game, stack your cards deep. Have a little bit of business acumen. Fuck what The Valley is doing, that is an ecosystem fully enslaved by economic nihilism, money isn't grounded there.
> Not nasty or defensive, just tired of these armchair claims that it's easy to go out and buy an 8 X MI355X box from people who obviously have no idea what the hardware lead time is like right now
This motte-bailey routine is both nasty and defensive, particularly when you keep prosecuting a geist of numeric justification that never arrives. All I've gotten from you is vague dismissals, one borderline irrelevant technical argument, moving goalposts and missing the point. Granted, not as egregiously as other people in this chain thinking we're talking about running 100B models on a Mac, I'll give you credit for that. But this whole time, we're just talking past each other. You make realistic points and I try to bring you back to context, but you have to work with me here too.
The point was that these companies are not selling to you below cost, they're not even selling to you at-cost. Just use your head. Venture capital isn't a magic wand. Frontier companies are in the red because they're in non-stop expansion operations at massive scales. Anthropic has an operating profit of half a billion dollars[1]. They are not selling you API usage below cost.
"You can of course trot out the point that oh, in 12 months this setup will be extremely outdated! "
But interestingly still extremely valuable on the second hand market.
The capital expense isn't the amount laid out. It's the rental cost of obtaining that capital, less the depreciation on the fixed asset over the period in use.
Going back to the OP, Apple gear is well know for having good resale values, which means the capital outlay isn't anywhere near as much as some people think.
>You couldn’t buy one of these if you wanted to right now.
You can: https://www.exxactcorp.com/Exxact-TS4-149591758-E149591758 . You can get thousands of tps of GLM 5.3 output out of this thing, which grades around Opus 4.8. Payoff is around 1 year vs. spot prices on these GPUs, including power.
Ah thanks for the solid info, too bad. I'd seen them come up as a pretty good price for 6000 RTX's in the past, which seem generally pretty available, good source for those?
Yeah, they're good source. But the price for those GPUs is 5 figs even with the nvidia startup program nowadays. Also, I went back and looked. Most of my GPUs are actually from Central Computers who were great, but Exxact is real too. So "lots of" was inaccurate.
Also, the lead time I quoted was for individual 8x nodes.
I can't tell from the ad -- it says "supports" 8x MI350X GPUs, but does that mean "includes" 8x MI350X GPUs? For $300K I'd certainly hope so, but I'm assuming not.
A system with 4x RTX 6000s costs about $60K these days, and can (as you note) trade blows with Opus 4.8 if not Fable. In fact, it'll give you a better pelican than Fable 5.1, and in less time.
Well, they are if you're into animating pelicans. :-P But yes, in the general case Opus is a better match.
And Opus is no slouch. I'm satisfied that GLM 5.3 is just as strong as Opus. Z.AI has promised/bragged that they will be at Fable 5.0 level by the end of the year or early next year, and I don't see any reason to doubt them.
Baseline yeah. But part of the reason you run open models is how much nicer fine tuning them is. Granted, you probably don't want to try and make LoRAs on a 4x RTX6000 setup, but you could if you really wanted to and there are other ways to modify models. And yes, if you're good at it, you can turn a piddly mid-range model that's only good at benchmarks into a heavyweight clanker (for a specific domain).
Ha fair, I'd definitely confirm with a salesperson before wiring them $300k. But most of the signs on the configurator seem to point to it including the GPUs? Not going to make 30k BTUs/hr of heat without the 8kw of GPUs.
Here’s an experiment: purchase an anthropic pro max subscription for $200/m. Now go buy the hardware to run DeepSeek’s equivalent. In a year, who spent more?
In normal times in which hardware used to depreciate (lately that's not the case and HW even appreciates, but let's not get distracted), if you calculate only with depreciation costs, plus the fact that when you have such a setup, it'd take many 200$ subs to cover your lack of limits in the other, I think it'd not be a clear victory for any side.
If you just ask "who spent more in the first year" (100% depreciation) then even with 5-6 max accounts, buying HW will be a couple of times more expensive. But when does it make sense to ask that question?
Maybe the SotA models will need better hardware so your investment will not be useful after a year or you'd need very expensive upgrades? But then (as in Fable case) subscribers need to spend more too.
It’s not so clear after 5 years that you’ll come out ahead. You’ll have spent $20k. The apple computer owner will probably be running local models that are better than today’s frontier on the same hardware.
Idk where you live, but where I am running the M5 Ultra Mac Studio at max rated power 24/7 for a month costs C$42.
The considerations against Apple hardware are 1) hardware advancements 2) early access to the best models. But it’s really not that clear.
(The other guy who thought hosted models on openrouter are cheap has spent $100k in 5 years.)
Open weight models have been getting better/smaller every year.
Also, from what I can tell, MLX inference is not as well optimized as CUDA, and the M5 Ultra has additional kinds of AI compute which is unavailable on other M models. With the massive 1.2 TB/s 512GB Mac studios coming out, I think MLX will get a lot more attention.
In short: Todays models should run faster next year, and next year's models should also be more efficient.
It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.
They're not cheap at all. I did one xhigh Qwen 3.8 27B agentic coding task last week via OpenRouter and it cost me like $10.
99% of the cost was in input tokens, I only used like 100k ish output tokens. It was a one shot task asking the agent to implement proxy injection to Guice. It did a pretty amazing job.
If you were to use hosted LLMs for a lot of agentic coding, a maxed out M5 Ultra Mac Studio would pay for itself in under a year.
I've been hosting Qwen3.8-27B myself. On my endpoint it's $0.30/1M in, $0.10 cache, $2.03 out - so those agent turns that re-send the same prefix get a lot cheaper when cache hits. UI at inference.tiyuvta.ai/app if you want to try it. Hosted is up to 210 tok/s and 280ms TTFT with reasoning off.
Qwen is weirdly expensive. Deepseek v4 flash is dirt cheap. You'd need at least 128gb of ram to run this model and in my experience, a days work with it costs around 80 cents.
So I ran the math, assuming the agent takes 75 turns per 200k context, with deepseek v4 flash it costs around $2.57 to reach 1M context in 375 turns. Cached input costs scale quadratically with # of agent turns.
Considering that I hit the 1M compaction multiple times per day with codex, it would definitely cost at least $5-8/day to use deepseek how I normally use codex.
Here's the thing that's a little different about data centers; we can tell from Anthropic and OpenAI that they're capacity constrained. Inference demand is there. I notice Cerebras doesn't offer much directly any more, all their capacity is getting completely sucked up by B2B sales. Grok did overbuild, but Anthropic was so desperate for more compute they ate their pride and leased the excess capacity.
That means all these data centers are being heavily utilized by actual end user inference demand. Well, some is research on new models, but a lot is actual end user demand. No one has given an explanation of why peoples usage would decline.
On top of that, margin on inference appears to be decent. It's model training that's a serious financial burden.
And maybe that's where there will be a slowdown, maybe the market doesn't justify spending as much on R&D as it does, but the end demand for inference is there.
Does that justify these stock prices? That's a different question. But the housing boom left behind endless rows of empty homes because demand disappeared. The 'dot com' boom left behind thousands of miles of dark fiber that'd been built out well ahead of demand for bandwidth. I can see the stock market having a giant sell off, but I don't see data centers sitting idle in that same fashion.
The number of planned data centers and the scale is pretty nuts.
Here's what you have to believe:
- AI demand is at least several times larger than what can currently be satisfied, or will grow. (This one I can buy, but...)
- AI chips (GPUs, TPUs, compute-in-memory, whatever else is being studied) will not get significantly more efficient than they are now. It will not be possible in, say, 5-10 years, to do 2X or 4X or 10X more AI requests per rack than is possible now. I think this one's the single most likely thing to be false, since all computing history contradicts it.
- Edge devices (PCs, laptops, specialized but smaller scale AI compute nodes) will never be powerful enough to run frontier models at a reasonable price that's appealing for professionals, enthusiasts, or businesses, and there will never be a market for this. None of the demand will be served on-device or near-edge. AI must all go in giant data centers.
- AI models will not become significantly more efficient than they are now. There are no large gains on the table from better model architectures, better training, more efficient quantizations, better harnesses, etc.
If all those things are true, than the current planned like 4X-10X increase in data center capacity makes sense. If even one or two of them are not true, then the planned data center build-outs start looking excessive. If all four are not true, it's a total bubble that will crash and burn. Answer is probably somewhere between, but how far toward bubble? That's why I picked a number like "only 20% ever gets built." It might be as high as 50%. It ain't gonna be 100%. The planned built-out is batty.
Oh I forgot two more...
- Data center capacity currently serving non-AI work loads does not shrink through either reduced demand, more efficient software, or (most likely) faster chips and denser RAM. If that happens, more pre-existing DC space can serve AI work loads.
- Orbital solar powered compute nodes never happen. If this happens (free power! much less political opposition!) then terrestrial data centers have significant competition.
I'm the author - hello! Added to the post! Qwen averages 325 tok/s in processing prompts, and 34 tok/s in token generation. That isn't instant, but it's quick enough that I never really think about it.
I took this thread and summarized it with Qwen3.6-35B-A3B, it had 1400 tps prefix and 60 tps completion. Very good performance. Using oMLX on MacBook M5 Pro 64GB.
Are you using the right configuration for your own CPU?
On a Laptop with 32 GB RAM and Iris Xe integrated graphic card, I get between 11-18 Tokens/Second with Qwen 3.8 27B and llama.cpp with sysl Intel optimisations. Same results with the vulkan back end, although sometimes it ends in weird segmentation faults due to the memory consumption.
Nice setup; but, for simple tasks or questions, AI is currently free? And it will probably stay free, as I don't see Google starting to charge for using AI on its search engine? So costs can't be a motivation for running small models locally?
For more complex or important tasks, costs, autonomy and privacy matter, but then so does performance/quality.
So I'm not completely convinced it's really worth it; but it's tempting!
I'm the author - hello! I talk about it in the blog post - knowing what's being run, knowing where it's being run, and not having anyone else control it.
I figure most free AI is free as in free electricity in the coffee shop. You're welcome to use it for small reasonable loads, but try to build anything off of it and you'll soon find yourself barred from the establishment.
And that's probably good, otherwise the free ai would just be unavailable for everyone else
You are correct. In large part, the cost of something like Gemini on a very basic Google AI plan provides far more utility than local LLMs for coding assistance.
There are 2 main reasons for running local LLMS.
1. Process private data/work with uncensored models.
2. Use a large amount of inference that would quickly blow through rate limits and/or run up API costs.
The thing that is critical for 2 is that a) you have to have a sweetspot between a pretty good model, which means largest parameter counts, and fast enough token generation where you can run agentic loops. The latter is needed because you aren't going go get the "intelligence" of larger models to form shell commands and run tools to figure stuff out, so the only way around that is to have custom agentic loops to force the model into doing what you want, which results in more text processing.
From my testing, Gemma4:31b is basically the only local model that can be relied upon to produce accurate results. Qwen models chase benchmarks, which results in MoE models (thus the A3B in the model, i.e 3 billion parameters are only active during inference). In general, these are good for very specific tasks, but fail to be accurate in considering cross task data, whereas Gemma, being fully active does a much better job. If you only need to do a very specific deterministic task, those models are pretty good.
As an aside though, if your task involves pure text processing (for example take html data, make it into a markdown document), you can also additive train Gemma270M quite easily all on CPU, and on a decent CPU it gets like 50-100 tok/sec, no need for any extra hardware.
The thing with Macs is that while they can run those models and larger models no problem, the tok/sec is very slow. This limits effectively what you can do with the models. On the M4 that the poster mentioned, Gemma:31b will run about 20 tok/sec. That means that when you wants to write a whole code file or process large context, you have to wait for it to do things. Compared to workflow with larger models, where file generation often takes like <10 seconds, it takes a while to adapt.
The only benefit of using Macs is the price for Mini and cheaper studios. However, once you reach the total cost of about 2.5k (note that the M4 statedin the article us about 2k), building a gfx card rig is the way to go. You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models.
Maybe, "Free as in free WiFi?" Like WiFi, the models you can use for free online aren't the highest quality, and can be pulled any time.
The models used in TFA are halfway in between the traditional "free as in beer" software. Open weight means once you download it, it continues to work forever; and you can also do your own RL on them; but you can't really see what went into their training, nor train a new one yourself from scratch.
Getting undeserved safety filter refusals, watching LLM accidentally reading what you might not find comfortable showing, and hitting free tier/basic subscription quotas, are not fun. Going local removes somewhat irrational anxiety that come from those.
Recent performance data on my M1 Max 32GB MacBook using oMLX. I have been working on identifying suitable model and config for my use case and system. Using a refactor and suggest improvements prompt for a specific Django code block using VSCode Cline extension.
Qwen3.8-27B-4bit generally runs out of output token before completing the task though excellent partial results.
Ornith-1.5-35B-A3B-MLX-4bit seems to get in the loop often specially with tool calls.
Qwen3.6-35B-A3B-mxfp4 seems to be optimal with speed and quality output.
I am going to test Qwen3.6-35B-A3B-4bit soon with same code block just to check my intuition that any derivatives don't seem to perform better than the originals.
I got 400 pp tps on a 10k token input. Your numbers seem suspiciously low, maybe the input was too short to measure properly? And this dense 27B is slow, the MoE A3B models get to 1000 tps.
> The main reason to run local: cloud APIs are rented land. They can change their pricing, hit your usage limits, or swap the model being served behind the scenes whenever they feel like it.
Apple is working from the 'desktop' up to beefy servers with 64GB+ RAM. Nvidia is working from the 'datacenter' down to beefy racks with terabytes of RAM.
There isn't really an overlap yet.
Individual Nvidia cards exist on desktops but they're not really oriented for regular inference so individual developers are left with Macs or datacenter resources as their options.
Maybe a silly question, but is there a reason/advantage to using mac minis over any other kind of small computer/laptop, ie running linux?
My understanding of using a mac mini for ai (ie running a claw bot or whatever) is to have it 'always on' and a better price/performance profile than a cheap vps.
Is there performance (silicon processor?) so unique? As I guess it's not their graphics units. I see tonnes of people using mac minis for AI, to the point it almost became a meme.
Edit: yes I know this article is about local models, my question is a bit more general.
There are probably three main reasons: 1) unified memory (but you can also that with DGS Spark / Strix Halo), 2) access to your Apple account, so you can have a bot handle your iMessages, email, calendars 3) energy efficiency.
Apart from that, if it doesn't work out you still have a Mac Mini, which in itself is more desirable for many than a DGX Spark or Strix Halo if you have no AI use case.
The main difference total other laptops of non-Apple make is/was unified memory (graphic VRAM + RAM) architecture. No need for an extra dedicated graphics card to get 64+ GB VRAM.
Quite a lot of "local doesn't work" in here - unfortunately, often with not much details about what the people actually want to use their models for. Which I'd be curious about.
I, personally, do use frontier models in the cloud for a lot of (meta-)cognitive analyses that are heavy enough to have me run against the limits of payed accounts regularly - so I'm neither a Luddite nor stingy with cash in this case.
However: I have pretty good experiences with local models as well. My solid but hardly extreme desktop (with one RX 9070 XT 16GB) mostly serves gemma4:12b and specialized models (embedding) to my local network. This is for general use like simple queries, simple code, reformatting and the like but also for two specific tasks that are permanently running:
a) It's connected to Home Assistant (as a second stage after very simple "turn light XY on" commands which get processed without LLM). So, I can mumble into my smartwatch "computer, how much gas do we have in the warp core and how much energy did the bussard collectors make from the cosmic dust today?" (or describe a more complex light scene or create an automation I want or whatever).
The phone transcribes that - with a local model on device - and fires it to the desktop who has agentic access to HA, looks through the sensors and data, sees that I've tagged my solar panels and battery with nerd vocabulary. It makes the right conclusion, converts a few units and gives me back a nice overview. All hands-free while I'm sitting on the toilet.
b) It's the LLM backend for a personal radio station run by a fleet of nerdy/quirky AI DJs who's archetypes are represented more than well enough in the latent space of the "small" model to produce funny results. The DJs can produce consistent, individual segments and programs, run a playlist that works well for me (based on multi-layered audio analysis that also uses local LLMs), respond to song wishes and generally produce much better recommendations than Spotify ever could for me. And you can also put multiple of them in the "studio" to create hilarious crossovers that you would not get from a commercial entity because the IP owners would rather shoot each other in the face.
All of this doesn't even max the available resources, so I can shovel F5-TTS into the VRAM as well and have all my DJs have good, locally created voices (or voice clones of Captain Picard and Han Solo, if I wanted to) based on zero-shot voice cloning.
--> Far from "unusable". It just depends on the task. And I neither have to hand my keys to the Navidrome server nor to my Smart Home to any entity outside my local network.
> Running a large model locally comes down to one thing: how much RAM it actually needs in memory.
Not completely true. It's memory AND memory bandwidth. You can have 1tb of memory but if you have awful memory-bandwidth you'll also have slow tok/s. A3B helps with this, but so does MTP.
From my experience, you'd be better off running the dense 27b-mlx with MTP than the 3.6 version with A3B. You say your model is ~20GB of ram, but the 3.8:27b-mlx is 18GB and gets me very reasonable tok/s, and greater speed if you disable thinking when not required.
The dense 27b Qwen on M4 Pro has a prompt processing speed of around 125tok/s which makes it ok to ask a quick question but impossible to use in an agent, as processing the first prompt of the agent with the tools and instruction can easily be 10 000 tokens
In this case the 35b a3b makes sense as it has a PP speed of around 800tok/s
True. This then boils down to a quality vs speed decision. the 3.8 27b is far better than 3.6 A3B from my experience. I'm happy taking the speed hit, given local models aren't as intelligent as frontier models. Anything that can get me closer to my CC experience both in reasonable speed and intelligence is worth it. With that said CC can also be slow at times, so it's locally the difference in experience is not always noticeable.
I experiment a lot with local LLMs, particularly small ones like Qwen3.5 4B and 9B. I have build multiple experiments to make harnesses that use these models for code generation, planning, local search, etc.
These are really good models but the harness has to be built around them. I have a ton of generated system prompts for specific purposes. Even parts of a SolidJS stack, for example Route management, has its own prompt. These are experiments but the results are real. If we build harnesses around small models, we can build a locally running WYSIWYG editor which works on plain text prompts.
The performance, in simple tokens/second, is not the most important factor. For many private data points, like emails, I would rather have a local graph based search and LLM on top where the harness is specific to problems like calendar, contacts, finance, etc.
I run all experiments on an 16GB M4 Mac Mini but coding agents building the harness are a mix of Codex, Claude Code and opencode.
I really like these show and tell style posts. I’m always curious how people have their setups and what tools they use. Also the blog has a nice theme and is easy to read.
I wanna get a desktop Mac for local ai so that I don’t turn my laptop into a delta 15k rpm fan when I run things.
If author runs the local model for privacy reasons, then I don't understand why they give Telegram access to all their conversations. It's well known that Telegram doesn't end-to-end encrypt bot accounts.
The beauty of this is that you can just swap out the platform and everything remains as it's the same backend. You make a really good point, one that I haven't really considered, but I also only have so many hours in the day to be fiddling around with stuff.
I just use Gemini Pro which comes bundled with a Chromebook; when the 'free' year runs out I buy another, initiate the free years Gemini again and then put the 'as new' Chromebook on eBay and get most of my money back. So Gemini Pro costs me less than £2 a month. And for my needs (investment research) that works well and is a pretty cheap compromise.
Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running models locally, if there was a way to inference the model of your choice at a reasonable cost by effectively time-sharing a B300 rack through some privacy-protecting intermediary, would you consider that?
If there was a "Mullvad of GPU clouds", would that solve the privacy concerns?
Chutes, Near AI, Phala and Tinfoil all offer various privacy assurances around inference. Some of the bigger providers also offer "zero data retention".
The problem I have with these is that the guarantees aren't strong enough (Phala, Near) or the models are old (Tinfoil). Chutes is mostly pretty good (cryptographic security all the way to the GPU) but I'm not sure it's possible to cryptographically verify the precise source code they run on the mode.
yes, and it's already some offerings like that but they all cost a lot because they only good for "I have some idea of workload for N hours or days" lets rent it and run. That fine for some experimentation but if you think about renting something 24/7 even for example to share it with the friends that will cost at least 4x from any API prices as result (something like rtx 6000 48gb will cost ~$470/m).
runpod.io is essentially this. You can rent the hardware for cheap in small time slices. I do this whenever I need to do a lot of embeddings, fast. I have an agent skill that will estimate the optimum hardware to reserve for the time/price constraints of the job, and you can spin up temporary inference for cheap via their API as well.
There will always be a reason to run frontier models, but local models are well at levels that assist with stuff that don't need that level of complexity.
It depends on your use case, but the smaller Gemma 4 models or qwen3.6:9b would probably run OK on that. I recommend trying it, even just for fun. It‘s easy with omlx.
Not enough for coding. 48GB is minimum for a non-lobotomized coding model like qwen, and you'd likely want 64GB to have long context and not kernel panic when Chrome opens.
You could run one of the smaller Gemma models to have a chatty Wikipedia.
> You do not know what these companies do with your data once they have it. They might limit how it gets used, they might sell it, they might expose it.
This is the burning question for me, what are they doing with our hard work.
I'd have thought that sherlocking a user's $10M business would be too high risk, given the billions at stake if real evidence of this happening was found.
However, OpenAI are currently being sued by Apple for trade secret theft, and the way it was done seems to be abundantly idiotic.
yes, share performance, numbers if you can, also i wonder if you figured out a way to do a 2way audio with local models, or even explored that. I have a very similar setup but not too happy with the token speed, will try omlx though !!!
Have a macmini m4 32G, not the pro version, previously everytime I tried local LLM is a bit disappointing, and I finally decide to not waste time and perhaps in the future invest a better hardware to server more modern and dense model
I am curious is what is the 80% request served by this setup, I was using it for OpenClaw which run serveral cron jobs that discover stuffs over the wide internet, check my support system's unanswered tickets, browser X and some social media for me to filter the valued ones(though I have to say even with GPT 5.6 sol, the quality is low for the timeline X sent to me)
Btw, Tailscale is quite cool and did a good job, I was using it to serve the local LLM and connct the openclaw on a Linux Machine to it.
You have tried Qwen 3.8 27B before coming to this conclusion, I hope? It's an incremental improvement over 3.6, but I mostly want to make sure you didn't just try running some old junker before coming to this conclusion.
I like the in-depth description. Everything from the naming convention of the models (and how much RAM they require) as well as all the components needed underscores just how complicated this all still is.
I suppose I am waiting for AI-in-a-Box to come along so I can (painlessly) join in.
(I'm sure wrangling with all these esoteric aspects of LLMs though is fun for some people.)
Hehe I really am just working it out as I go along - I promise it is fairly painless. Hugging Face allow you to specify your machine and then browse models that fit. And then you can just vibe out 'oh this one is a bit slow let me try another' etc
I've seen https://www.lucebox.com/ as an interesting option
It’s $7900! “Interesting” and “option” are doing a lot of heavy lifting there :).
... if they are doing heavy lifting, are they ... load-bearing? :)
I was hoping for a software "LLM in a Box".
No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minutes to run. It's free. The machine is just sitting there anyway. I have tried using qwen-coder and opencode on my M5 Max 128gb and compared to claude code it's painful. I did setup a workflow where claude plans, qwen executes (unattended overnight, again b/c it's slow) and then claude reviews. I benchmarked this several times and I ended up using MORE tokens with claude because it had to 'fix' all the qwen issues. While the code it produced was 'good enough' the fixes were worth it so I just stick to coding task using API models (codex and claude).
> It's free
It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.
It sounds like they are doing something similar to what I described in my other post below. Personal media station.
That can be done on hardware that quite a lot of people basically just have and don't use 24/7 to the max - because it is their gaming machine or their programming and compiling workhorse, for example. Of course you are paying for additional electricity but even with napkin-math instead of a "proper" calculation, you are unlikely to pay more for running your own instead of something commercial (and that can be offset further with some of the "modern" electricity contracts and/or PV and battery storage). Especially if we are talking about a stack that runs most of/all the time when you are not using your machine and makes LLM calls regularly while running.
The work in software/admin to get whatever you want set up is similiar no matter which infrastructure you use.
> the cost is included in your electricity bill
Acting like an extra $20 on my electric bill is equivalent to a $200/mo subscription is... a take.
If you buy the computer specifically for inference it is more expensive than cloud, but if you had it anyway it's free.
True, but how many people (realistically) buy a computer with 48GB+ of RAM?
48gb of vram.
a machine like this is about a years rent for most people.
a small car for most others.
I think he’s talking about the Mac Mini unified memory.
48G RAM is pretty useful if you want to run k8s locally for tests / exploration
true but if you're actually running k8s and similar workloads, chances are it might eat memory that LLM requires.
you'll also notice these articles rarely specify their context window in tokens, because it is small, usually 30k to 70k tokens and it gets slower as it fills up.
I actually have a Mac Mini M4 Pro with 48G. I gave the k8s example because this is what I was doing with it.
Was because I am back to using Linux as my workstation.
My Mac Mini is now a headless server for llama.cpp.
So, you are right that for these workloads , I would not be using the Mac Mini for k8s AND llama.
Another thing going against using a Mac for Linux containers is that there are no solutions that I know that properly manages memory : memory is given to the Linux vm , but never fluctuates if the needs in the vm are less than the initial request.
I know Orb Stack does that but is it proprietary. I think UTM does it , but not sure I would use UTM instead of Lima, Colima , multipass , etc to run containers.
Try smolvm microvms from https://smolmachines.com - among other benefits they only consume host resources if they're actually used.
Free heating in winter.
Though much less energy efficient than a heat pump.
> It's very possible that it costs you more than a cloud mode would
...which is almost always true in a single request/reply mode and never true in batch mode. Single request usually 2x-3x more expensive than cloud and batch mode 2x-3x cheaper. Now, for narrow tasks, a finetuned tiny 8b model would dramatically outperform SOTA frontiers for a fraction of price, esp. on energy efficient hardware like Apple.
Local is never cheaper than cloud because they can do batch inference, and that means you load model weights once to produce 128 tokens on 128 sessions in parallel not 1 token on 1 session like local models. Local models rarely get to high utilization factor, they spend most of their time waiting.
If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.
> not 1 token on 1 session like local models.
Local models can absolutely run in batch, what are even talking about?
> If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.
Even if you ran sequentally, single session, a _finetuned_ tiny (8B) local model on narrow tasks would abolutely mog SOTAs, any of it - Fable, Opus, Sol you name it.
> Local models can absolutely run in batch, what are even talking about?
I think the point was that if you aren't running your local machine at 100% for 24 hours a day then a cloud - with multiple clients - that is, will be more efficient.
Can you share a bit more about your bifrost and llama swap setup? I’m facing memory constraints and am looking for a managed model solution that will help with hot swapping loaded models and stay-warm concurrency. Ideally with prioritization.
What do you want to know? Just start llama-swap with the models i have downloaded, add llama-swap as a provider in bifrost, expose the models you want and they become available in one single endpoint you can use in anything like opencode, openwebui or anything that speaks openai.
Yes, that makes sense. Some of my models currently run in ollama while others require their own inference servers. I’m curious about custom inference servers in bifrost and the ability to orchestrate keeping some models warm in memory while evicting and cycling other models. All of which span different providers in bifrost (I think). Obviously I can get the mechanics from an agent, more wondering about any experience with something similar.
i have a 512gb ram m3 ultra mac studio setup with a gas city that runs one of my companies. today was the first time ever that a local model (GLM5.3 8-bit) was able to match fable5 in our tests.
GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent city on one box with ~130 GB to spare. • Review test: caught 6 of 6 planted P1 defects, zero false positives, same score as the frontier model we pay for. • CRM test: 11 of 11 required records extracted, zero wrong writes, 45 minutes, first local model to clear the bar. • Serving a 131k-token window today; the model itself supports 1,048,576. Widened to 4 concurrent slots and still have 50gb+ of excess ram.
granted my cto still isn't moving all of our inference to glm5.3 but we've identified 40%+ that is currently handled by fable that we're routing locally instead and will do concurrent requests to verify/compare responses for a while.
Hosted GLM 5.3 flash is like $0.15/mtok in $0.50/mtok out
Time to completion also must be considered. If I have to wait around for hours for a prompt to complete locally and I’ll need to iterate quickly, I’m better off hosted than local. If it’s “free” and slow it may just not be worth it.
So this is something like a $10,000 machine before RAM prices rose?
I see Apple is currently selling a 256GB M5 for about $10K, so buying October's 512GB one could be, what, $13-14K?
A $0 per-token bill is great but this is clearly not something for normal people, just some businesses.
Props to your parent commenter for including context size. Because 131k context window is prohibitively small for my coding workloads so I know a 512GB Mac won't cut it.
What sort of business can you run with this setup?
This may work for your use case, but sounds abysmally slow for any complex coding task.
I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the performance is more than good enough. I'm not running the MLX version of the Gemma model, if I did the inference speed would likely be a bit better. I wouldn't use them for coding features though.
My perf sucks compared to yours. Added it to the post - same model averages 325 tok/s in processing prompts, and 34 tok/s in token generation. What am I doing wrong..?
Wow, that's just about half the perf. I'm not sure what you're doing differently, though our hardware is a bit different: I am on a Macbook Pro M4 Pro, while you're on a Mac Mini.
I would try a different version of the model from HuggingFace while ensuring it's MLX. I'm also using LM Studio, not oMLX, and I've seen some threads like these:
https://www.reddit.com/r/LocalLLaMA/comments/1spuwir/omlx_10...
I have a (now discontinued) 64gb mini pro and I’ve found the same qwen model to be almost unusable unless I kill Thinking on each turn.
What are you using them with/for?
I do turn thinking off most of the time for both models. I made a separate comment detailing my use cases.
> enough for my needs
Which are...?
Some examples (keep in mind this is all indefinitely free for me, no burning quota away):
1. Getting information (such as information about hardware unfamiliar to me) when not connected to the internet, which happens occasionally in my case.
2. Continuing to learn Rust by way of toy examples, puzzles, and comparing aspects of various solutions, for example from LeetCode.
3. Reformatting data, for example from a PDF to a markdown table, or converting receipt images to text.
4. Simple translation/explanation (e.g. I'm teaching my wife one of the languages I speak but sometimes may not know/have the words to explain the full nuance of a translated word).
5. Summarization. One of the webnovels I'm reading has some very boring parts I don't want to slog through, in those cases I simply make the LLM summarize that part and move on.
Etc., you get the idea. It's not unusable for coding, but it would make many mistakes when making a whole feature and the context lengths are limited to around 30k-40k tokens by my RAM. I could give it access to the web but I simply use an online model when I need that sort of thing, again partly due to the context limit.
Edit: The MLX version of Gemma 4 26b a4b does about 62 tok/s.
What kinds of tasks are you using this for?
I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B, or a pair of DGX Sparks running DSv4 flash, or better, 2x6000 RTX Blackwells. Those are the kinds of rigs that the local model enthusiasts are running. With the GPU setups, you’re looking at generally >100tps generation in single stream, and >10k tps of prefill, so it’s snappier than Claude code, which somewhat makes up for it being dumber.
That said, it is really cool to be able to run an LLM on eg a Mac laptop. Just not a better experience on almost any metric for interactive use than eg Claude Code, beside privacy and guardrails.
>I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B
How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed.
[1] https://deepswe.datacurve.ai/, https://unsloth.ai/docs/models/qwen3.8#benchmarks
Keep in mind these downloadable models use 3-10x the amount of tokens as well. You really can’t beat a couple $20 subscriptions.
https://quesma.com/benchmarks/babaisbench/
The price is handing over your data, and your intellectual property.
>your intellectual property
Which already comes from Claude itself. Clearly, they don't want to train on their own product.
Looks like GLM 5.2 is coming in at <2x the tokens of Opus 4.8 (and 1/10 the cost)?
Great showing from Sol, though.
But also, it's Baba Is You :-D
Not sure, I haven't run it, I've just been running DS V4 Flash non-stop since it came out, and that's replaced a lot of my Claude Code usage. People seem very impressed, though, it seems like it trades vram/world knowledge for extra thinking time, which I think is a good trade for local. tbf, I've heard luna's not great at coding. Fast and good for things like classifiers, summarization, though.
A friend and I were actually discussing today how benches show Luna Max at about par on coding with Sol Medium, but how it's nowhere near in reality. We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n, because variance in performance is killer with large coding projects. Consistency is what lets you actually build on this stuff.
> We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n.
Thank you. I just changed my opinion on this thanks to you. I agree now, since we tend to execute LLM tasks once instead of N times anyway.
I suspect models like Fable executes the same task N times in parallel and picks best answer or merges them to for a better answer.
It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
That's not even remotely close to being true, even once you account for capex. You have to look at the actual usage, look at the token limits. Even if you're paying Anthropic $200k/month for scale-tier, you're going to blow through your token limits trying to run max output 24/7. Three users running Opus 4.8 at max non-stop will probably clean your monthly allowance from daddy Dario in less than a week.
With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive. It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause. And you get the full month like that, your monthly token limit is the time in a month. That cluster, the electrical upgrade, the cooling setup, and the electricity to run it all costs less in 2 months than your maximum affordance from Anthropic does in the same time period. Two billing cycles, and realistically it's more like two weeks. In 4 quarters you've wasted over a million. Like, what are we talking about here?
Now if you aren't using AI all that much, which is perfectly valid, and especially if you aren't using it at its absolute maximum, the story changes. Because even though at that point you're not paying nearly as much in electricity to run the cluster anymore, you still have the $300k+ capex to get the setup in the first place. But if we're not redlining it non-stop, then we're not really talking about performance anymore, are we? If your org never comes close to hitting token limits, it's probably because AI is rather marginal for you. Which again, is perfectly valid. I don't even use AI professionally.
Fact of the matter is, if your corp can justify the capex for a cluster and makes heavy use of AI, you are literally burning money by not having one in your building. The numbers are painfully obvious. Even deepseek isn't as cheap. This is before we get into things like LoRAs, custom inference pipelines, etc. which you know are kind of important if you actually care about model performance.
> With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive.
Pretty expensive is an understatement. You couldn’t buy one of these if you wanted to right now. If you could it would be multiple hundreds of thousands of dollars.
> It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause
You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users. I don’t know how you think it’s going to run 8 of them at the same time. Did you mean 8 concurrent sessions?
Your math is way off across this post. If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months, it wouldn’t be some little secret that we only discover in a comment online.
> Pretty expensive is an understatement. [...] If you could it would be multiple hundreds of thousands of dollars.
Obviously, I quantified both the operating expense and the capital expense in my post. What I find curious is that you're quoting me talking about the operating expenditure, and changing the topic to be about the buy-in like these are interchangeable things. You don't think that this is a crucial and important distinction?
> You couldn’t buy one of these if you wanted to right now.
You could have spent all of 5 seconds of searching rather than just assuming[1]. You're not buying an Nvidia Superpod™.
> You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users.
That's certainly fair a point. Although in the English language, especially in legal contexts, the multi- prefix is used inclusively for fractional values. That is it's strictly >1, not >=2. IE an 18 month contract is a multi-year contract, or a $1.6 million dollar asset is a "multi-million" dollar asset. But this is uninteresting semantics.
You are right, but it also doesn't matter. The gap is just that big. You can run 1 single user of Kimi K3 and still not even come remotely close to the $70k or so that a single Opus 4.8 user can burn over the course of a month on left on max. An honestly lowballed amount I know from anecdote. The per-token cost is just really expensive.
> Your math is way off across this post.
You made one technical point above, one that doesn't ever arrive at a relevant rebuttal to the substance of my post. But please, I'd love to hear you elaborate, especially because I didn't actually give much math at all.
If you want math though, here's the math. Let's say you are paying a ridiculous amount of money for electricity, a price nobody in the US pays -- $2 per kilowatt hour. That's about 5x the average rate in California, 4x as in Hawai'i. 17kW @ $2/kWh * ~8766 hours in a year puts that cluster's electrical costs at just shy of ~$298k annually assuming it takes no breaks. Let's make matters worse and round that up to $300k. It's also assuming you didn't invest in a solar hookup for your building, which I don't know why you haven't at this point, especially if you're installing a CDU for your new cluster. 12 months of Claude burning $70k a month is $840k. For a buy in of, you know what, let's call it $500k. Why not? It still doesn't matter. The operating cost is so much lower it's paid for itself plus an additional $40k in the first year. Even at a ridiculous penalty in electricity that nobody pays, even overinflating the amount of money you'd pay for the cluster and the infrastructure to get it set up, it's not even remotely close for a single user where the gap is smaller (IE, you're not wasting "a million dollars" in a year by maxing out the $200k scaling limit every month)
You can of course trot out the point that oh, in 12 months this setup will be extremely outdated! It doesn't matter. If the work it was doing today was useful, it will be useful next year too. And with the rapidly encroaching diminishing returns from parameter scaling, you're probably going to be just fine for a while. Maybe grab a quantized version of a newer Chinese model at the end, before grabbing a newer generation of AMD node. Those MI400s are looking pretty sweet after all.
> If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months
If you're locked in, then you're locked in. But don't pretend like you're saving money. You're not.
> it wouldn’t be some little secret that we only discover in a comment online.
Why does this have you so nasty and defensive? It's not a "little secret" that running your own infrastructure is cheaper. Of course it is. You know what else is cheaper? Owning your own office building out in the sticks, rather than leasing part of one in the city. Not everybody can make that work, there are no free lunches after all.
History repeats, these same exact lines were rolled out ad nauseum during the cloud craze. Datacenters are businesses, not charities. Frontier companies rent quite a fair amount of their infrastructure. Even if they resold that compute below cost (they don't), there's a pretty steep cliff before the economics start to look attractive.
[1] - https://www.avadirect.com/GIGABYTE-G893-ZX1-AAX4-Dual-AMD-EP...
> You could have spent all of 5 seconds of searching rather than just assuming[1].
I guarantee this will not ship to you any time soon.
The current lead time on these GPUs in measured in years. If you didn't place an order for this a long time ago, it's not coming this year.
Being able to add it to an online configurator does not mean anything right now.
> 12 months of Claude burning $70k a month is $840k
Your math is completely useless with these arbitrary numbers pulled out of the air.
If you want to begin calculating payback period you'd need to look at token costs, cost per task, utilization rates, and so on.
> The operating cost is so much lower it's paid for itself plus an additional $40k in the first year.
You went from paying back in a couple months to paying back in a year but you still haven't even talked about tokens or concurrency.
You're also neglecting the fact that hosted tokens are going down in price at a rapid rate. If someone was paying $70K per month in tokens for Opus this month, that same level of compute is going to be much cheaper 12 months from now.
> Why does this have you so nasty and defensive?
Not nasty or defensive, just tired of these armchair claims that it's easy to go out and buy an 8 X MI355X box from people who obviously have no idea what the hardware lead time is like right now, or who haven't considered the actual math on token costs and payback times. You're still making a lot of claims without a single discussion of cost per task or token.
It usually boils down to people trying to convince themselves that keeping their macs hot and with very little ram to spare only to get sub 50 tokens per second on a subpar lobotomized (quantized) model is worth it.
And I'm not even considering their time spent fiddling, fine tuning configs to adjust for ram, updating/benchmarking models, etc. Which is probably more expensive than the mac so the math is even more wrong.
> I guarantee this will not ship to you any time soon.
The assumption, the starting point, is that you have a line on the hardware. Asking around, some distributors have a 6 month lead time on Instinct GPUs, which curiously enough is about how long you'll be twiddling your thumbs waiting for the cooling loop to be put in. Yes things take time.
> Your math is completely useless with these arbitrary numbers pulled out of the air.
Your dismissal is worthless if you can't even be bothered to provide a counter-example. You've not provided a single iota of quantified reasoning beyond my original not accounting for the space used for the context of concurrent users.
> If you want to begin calculating payback period you'd need to look at token costs, cost per task, utilization rates, and so on.
Now go back and carefully reread my original post. Yes, if you are not actually redlining an LLM for a billing cycle, the capex starts to be way more relevant for this setup. Otherwise, our constraint is time and our unit of measure is $/hr.
If you want to compare token cost, it may shock you to learn that Kimi K3 without speculative decode on this setup is slightly under twice as fast as Opus 4.8 max. That's still true when fast is compared with K3 with speculative decode, and now Claude is twice as expensive as a base rate. Oops. We're already burning more money over a period of time, looking at tokens we're screaming even further ahead.
> You're also neglecting the fact that hosted tokens are going down in price at a rapid rate.
Cool. Call me when Opus 4.8 max is $0.50/million. In 4 years you could have bought the 200 acres of land down the road from your building, started a 5MW solar farm subsidiary that you'll expand over time, and as soon as your connect is up, dropped the opex of the cluster down to its maintenance costs. That subsidiary will pay the loan required to spin it up back irrespective of your primary business. When you own your own shit, you can play your own game, stack your cards deep. Have a little bit of business acumen. Fuck what The Valley is doing, that is an ecosystem fully enslaved by economic nihilism, money isn't grounded there.
> Not nasty or defensive, just tired of these armchair claims that it's easy to go out and buy an 8 X MI355X box from people who obviously have no idea what the hardware lead time is like right now
This motte-bailey routine is both nasty and defensive, particularly when you keep prosecuting a geist of numeric justification that never arrives. All I've gotten from you is vague dismissals, one borderline irrelevant technical argument, moving goalposts and missing the point. Granted, not as egregiously as other people in this chain thinking we're talking about running 100B models on a Mac, I'll give you credit for that. But this whole time, we're just talking past each other. You make realistic points and I try to bring you back to context, but you have to work with me here too.
The point was that these companies are not selling to you below cost, they're not even selling to you at-cost. Just use your head. Venture capital isn't a magic wand. Frontier companies are in the red because they're in non-stop expansion operations at massive scales. Anthropic has an operating profit of half a billion dollars[1]. They are not selling you API usage below cost.
[1] - https://www.forbes.com/sites/jonmarkman/2026/08/17/anthropic...
"You can of course trot out the point that oh, in 12 months this setup will be extremely outdated! "
But interestingly still extremely valuable on the second hand market.
The capital expense isn't the amount laid out. It's the rental cost of obtaining that capital, less the depreciation on the fixed asset over the period in use.
Going back to the OP, Apple gear is well know for having good resale values, which means the capital outlay isn't anywhere near as much as some people think.
>You couldn’t buy one of these if you wanted to right now.
You can: https://www.exxactcorp.com/Exxact-TS4-149591758-E149591758 . You can get thousands of tps of GLM 5.3 output out of this thing, which grades around Opus 4.8. Payoff is around 1 year vs. spot prices on these GPUs, including power.
> You can: https://www.exxactcorp.com/Exxact-TS4-149591758-E149591758 .
No, you can get a quote for possibly being allocated one in the distant future.
The backlog for these is huge. You cannot buy one any time soon.
Ah gotcha. Have you tried to order something like this in the past?
I have quoted large nodes from this supplier and have lots of^W^W GPUs from them for personal use. Current lead time is more than 30 months.
They're a good provider but you have to be a big shot buying NVL72s before you're getting anything within your payback period.
Yeah, you're looking at the scale of ... Nscale ... buying in the order of 1,000 NVL72 racks to get yourself deliveries in a timely manner.
Ah thanks for the solid info, too bad. I'd seen them come up as a pretty good price for 6000 RTX's in the past, which seem generally pretty available, good source for those?
Yeah, they're good source. But the price for those GPUs is 5 figs even with the nvidia startup program nowadays. Also, I went back and looked. Most of my GPUs are actually from Central Computers who were great, but Exxact is real too. So "lots of" was inaccurate.
Also, the lead time I quoted was for individual 8x nodes.
Ah yeah, one of mine is from Central. And yeah, crazy how much they've gone up. But I can see why, they scream.
Oh my god, I completely spaced dude. Weeks, not months. Weeks. Sorry.
Oh, nice haha, way better. And no worries, I'm actually waiting on a quote/timeline from them anyway.
I can't tell from the ad -- it says "supports" 8x MI350X GPUs, but does that mean "includes" 8x MI350X GPUs? For $300K I'd certainly hope so, but I'm assuming not.
A system with 4x RTX 6000s costs about $60K these days, and can (as you note) trade blows with Opus 4.8 if not Fable. In fact, it'll give you a better pelican than Fable 5.1, and in less time.
> trade blows with Opus 4.8 if not Fable.
Okay I love the open models, but the hype is getting ridiculous. The models you can run on 4 X RTX6000 are not Fable level.
Well, they are if you're into animating pelicans. :-P But yes, in the general case Opus is a better match.
And Opus is no slouch. I'm satisfied that GLM 5.3 is just as strong as Opus. Z.AI has promised/bragged that they will be at Fable 5.0 level by the end of the year or early next year, and I don't see any reason to doubt them.
Baseline yeah. But part of the reason you run open models is how much nicer fine tuning them is. Granted, you probably don't want to try and make LoRAs on a 4x RTX6000 setup, but you could if you really wanted to and there are other ways to modify models. And yes, if you're good at it, you can turn a piddly mid-range model that's only good at benchmarks into a heavyweight clanker (for a specific domain).
Ha fair, I'd definitely confirm with a salesperson before wiring them $300k. But most of the signs on the configurator seem to point to it including the GPUs? Not going to make 30k BTUs/hr of heat without the 8kw of GPUs.
Here’s an experiment: purchase an anthropic pro max subscription for $200/m. Now go buy the hardware to run DeepSeek’s equivalent. In a year, who spent more?
That's not apples to apples on almost any dimension.
In normal times in which hardware used to depreciate (lately that's not the case and HW even appreciates, but let's not get distracted), if you calculate only with depreciation costs, plus the fact that when you have such a setup, it'd take many 200$ subs to cover your lack of limits in the other, I think it'd not be a clear victory for any side.
If you just ask "who spent more in the first year" (100% depreciation) then even with 5-6 max accounts, buying HW will be a couple of times more expensive. But when does it make sense to ask that question?
Maybe the SotA models will need better hardware so your investment will not be useful after a year or you'd need very expensive upgrades? But then (as in Fable case) subscribers need to spend more too.
It’s not so clear after 5 years that you’ll come out ahead. You’ll have spent $20k. The apple computer owner will probably be running local models that are better than today’s frontier on the same hardware.
Idk where you live, but where I am running the M5 Ultra Mac Studio at max rated power 24/7 for a month costs C$42.
The considerations against Apple hardware are 1) hardware advancements 2) early access to the best models. But it’s really not that clear.
(The other guy who thought hosted models on openrouter are cheap has spent $100k in 5 years.)
> The apple computer owner will probably be running local models that are better than today’s frontier on the same hardware.
Hardware is not magically getting more memory or bandwidth.
Believing there will be some magical optimizations to compensate for it is just dellusion.
Open weight models have been getting better/smaller every year.
Also, from what I can tell, MLX inference is not as well optimized as CUDA, and the M5 Ultra has additional kinds of AI compute which is unavailable on other M models. With the massive 1.2 TB/s 512GB Mac studios coming out, I think MLX will get a lot more attention.
In short: Todays models should run faster next year, and next year's models should also be more efficient.
Then explain how equal parameter size models can grow in capability every few months or year?
Depends on what you plan to do.
You don't need frontier models to summarise or create an email.
What're you using that monster for?
It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.
They're not cheap at all. I did one xhigh Qwen 3.8 27B agentic coding task last week via OpenRouter and it cost me like $10.
99% of the cost was in input tokens, I only used like 100k ish output tokens. It was a one shot task asking the agent to implement proxy injection to Guice. It did a pretty amazing job.
If you were to use hosted LLMs for a lot of agentic coding, a maxed out M5 Ultra Mac Studio would pay for itself in under a year.
I've been hosting Qwen3.8-27B myself. On my endpoint it's $0.30/1M in, $0.10 cache, $2.03 out - so those agent turns that re-send the same prefix get a lot cheaper when cache hits. UI at inference.tiyuvta.ai/app if you want to try it. Hosted is up to 210 tok/s and 280ms TTFT with reasoning off.
Qwen is weirdly expensive. Deepseek v4 flash is dirt cheap. You'd need at least 128gb of ram to run this model and in my experience, a days work with it costs around 80 cents.
So I ran the math, assuming the agent takes 75 turns per 200k context, with deepseek v4 flash it costs around $2.57 to reach 1M context in 375 turns. Cached input costs scale quadratically with # of agent turns.
Considering that I hit the 1M compaction multiple times per day with codex, it would definitely cost at least $5-8/day to use deepseek how I normally use codex.
There are economies of scale but there’s also a data center bubble (probably) so there might be some selling dollars for fifty cents going on.
Here's the thing that's a little different about data centers; we can tell from Anthropic and OpenAI that they're capacity constrained. Inference demand is there. I notice Cerebras doesn't offer much directly any more, all their capacity is getting completely sucked up by B2B sales. Grok did overbuild, but Anthropic was so desperate for more compute they ate their pride and leased the excess capacity.
That means all these data centers are being heavily utilized by actual end user inference demand. Well, some is research on new models, but a lot is actual end user demand. No one has given an explanation of why peoples usage would decline.
On top of that, margin on inference appears to be decent. It's model training that's a serious financial burden.
And maybe that's where there will be a slowdown, maybe the market doesn't justify spending as much on R&D as it does, but the end demand for inference is there.
Does that justify these stock prices? That's a different question. But the housing boom left behind endless rows of empty homes because demand disappeared. The 'dot com' boom left behind thousands of miles of dark fiber that'd been built out well ahead of demand for bandwidth. I can see the stock market having a giant sell off, but I don't see data centers sitting idle in that same fashion.
The number of planned data centers and the scale is pretty nuts.
Here's what you have to believe:
- AI demand is at least several times larger than what can currently be satisfied, or will grow. (This one I can buy, but...)
- AI chips (GPUs, TPUs, compute-in-memory, whatever else is being studied) will not get significantly more efficient than they are now. It will not be possible in, say, 5-10 years, to do 2X or 4X or 10X more AI requests per rack than is possible now. I think this one's the single most likely thing to be false, since all computing history contradicts it.
- Edge devices (PCs, laptops, specialized but smaller scale AI compute nodes) will never be powerful enough to run frontier models at a reasonable price that's appealing for professionals, enthusiasts, or businesses, and there will never be a market for this. None of the demand will be served on-device or near-edge. AI must all go in giant data centers.
- AI models will not become significantly more efficient than they are now. There are no large gains on the table from better model architectures, better training, more efficient quantizations, better harnesses, etc.
If all those things are true, than the current planned like 4X-10X increase in data center capacity makes sense. If even one or two of them are not true, then the planned data center build-outs start looking excessive. If all four are not true, it's a total bubble that will crash and burn. Answer is probably somewhere between, but how far toward bubble? That's why I picked a number like "only 20% ever gets built." It might be as high as 50%. It ain't gonna be 100%. The planned built-out is batty.
Oh I forgot two more...
- Data center capacity currently serving non-AI work loads does not shrink through either reduced demand, more efficient software, or (most likely) faster chips and denser RAM. If that happens, more pre-existing DC space can serve AI work loads.
- Orbital solar powered compute nodes never happen. If this happens (free power! much less political opposition!) then terrestrial data centers have significant competition.
It's a time sharing agreement, just like old-school mainframes and such. You're not getting a full machine to yourself, but a few cycles at a time.
I watched someone at a fortune 20 company get embarrassed for buying a Mac to run a 70B model in 2025.
He was a lead engineer, so after he announced it wasn't going to work, everyone pretended it never happened. But we all knew.
Sounds like a really rude workplace. Who cares if he wants to try running things locally?
What was rude? No one bothered him.
Also the entire purpose of them buying it was so the department had a LLM.
I proposed A6000. That ended up working.
I'm the author - hello! Added to the post! Qwen averages 325 tok/s in processing prompts, and 34 tok/s in token generation. That isn't instant, but it's quick enough that I never really think about it.
I can't imagine using CPU... Oh I did twice.
If you are work from home and do dishes between prompts you can get a gpt3-like result.
I found it useful when I was... Well I didn't find it useful. But an Nvidia 3060 let me ask unethical questions pretty fast.
With which model. I have a 3060 with a bunch of system ram
Old school Berkeley Sterling or an abliterared model.
That is a lot of dishes.
These are estimates: https://tokenstead.ai/find/results?hardware_id=23&use_case=c...
That's useful thanks. Also, looks painfully slow!
I took this thread and summarized it with Qwen3.6-35B-A3B, it had 1400 tps prefix and 60 tps completion. Very good performance. Using oMLX on MacBook M5 Pro 64GB.
Are you using the right configuration for your own CPU?
On a Laptop with 32 GB RAM and Iris Xe integrated graphic card, I get between 11-18 Tokens/Second with Qwen 3.8 27B and llama.cpp with sysl Intel optimisations. Same results with the vulkan back end, although sometimes it ends in weird segmentation faults due to the memory consumption.
Qwen3.8:27b-mlx on 64GB MBP M4, I can get up to 42tok/s, more often than not in the ~30 range.
> I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible
With all due respect, I'm not clear why you are so surprised ?
By your own admission its a little mini-PC with 16GB RAM, I'm not sure what miracles you were expecting ?
Its a bit like complaining Rasperry Pi performance is terrible when trying to compile the Linux kernel.
That's their point
Not surprised at all - just making the point that getting a model to run isn't that impressive if it runs at a few tokens per second.
Nice setup; but, for simple tasks or questions, AI is currently free? And it will probably stay free, as I don't see Google starting to charge for using AI on its search engine? So costs can't be a motivation for running small models locally?
For more complex or important tasks, costs, autonomy and privacy matter, but then so does performance/quality.
So I'm not completely convinced it's really worth it; but it's tempting!
I'm the author - hello! I talk about it in the blog post - knowing what's being run, knowing where it's being run, and not having anyone else control it.
Traditional search is “free” too, but you see ads. If something looks free, then you are the product.
OpenAI has started doing ads, but for Anthropic the free tier is still just a loss leader. It's basically an ad for their pay tiers.
I figure most free AI is free as in free electricity in the coffee shop. You're welcome to use it for small reasonable loads, but try to build anything off of it and you'll soon find yourself barred from the establishment.
And that's probably good, otherwise the free ai would just be unavailable for everyone else
It's free like ads are free. Or certain kinds of advice.
You are correct. In large part, the cost of something like Gemini on a very basic Google AI plan provides far more utility than local LLMs for coding assistance.
There are 2 main reasons for running local LLMS.
1. Process private data/work with uncensored models.
2. Use a large amount of inference that would quickly blow through rate limits and/or run up API costs.
The thing that is critical for 2 is that a) you have to have a sweetspot between a pretty good model, which means largest parameter counts, and fast enough token generation where you can run agentic loops. The latter is needed because you aren't going go get the "intelligence" of larger models to form shell commands and run tools to figure stuff out, so the only way around that is to have custom agentic loops to force the model into doing what you want, which results in more text processing.
From my testing, Gemma4:31b is basically the only local model that can be relied upon to produce accurate results. Qwen models chase benchmarks, which results in MoE models (thus the A3B in the model, i.e 3 billion parameters are only active during inference). In general, these are good for very specific tasks, but fail to be accurate in considering cross task data, whereas Gemma, being fully active does a much better job. If you only need to do a very specific deterministic task, those models are pretty good.
As an aside though, if your task involves pure text processing (for example take html data, make it into a markdown document), you can also additive train Gemma270M quite easily all on CPU, and on a decent CPU it gets like 50-100 tok/sec, no need for any extra hardware.
The thing with Macs is that while they can run those models and larger models no problem, the tok/sec is very slow. This limits effectively what you can do with the models. On the M4 that the poster mentioned, Gemma:31b will run about 20 tok/sec. That means that when you wants to write a whole code file or process large context, you have to wait for it to do things. Compared to workflow with larger models, where file generation often takes like <10 seconds, it takes a while to adapt.
The only benefit of using Macs is the price for Mini and cheaper studios. However, once you reach the total cost of about 2.5k (note that the M4 statedin the article us about 2k), building a gfx card rig is the way to go. You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models.
> You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models
I have a beefy Linux box with a 4090 but never took the time to set it up properly beyond simple testing; any tutorial you would recommend?
Just ollama if you are feeling lazy. With ollama, pull the model (start with https://ollama.com/library/gemma3:27b), and `ollama run gemma3:27b`
If you want to build agentic frameworks, use llama.cpp with its built in http server, and build the framework with python
Only “Free as in free beer.”
Maybe, "Free as in free WiFi?" Like WiFi, the models you can use for free online aren't the highest quality, and can be pulled any time.
The models used in TFA are halfway in between the traditional "free as in beer" software. Open weight means once you download it, it continues to work forever; and you can also do your own RL on them; but you can't really see what went into their training, nor train a new one yourself from scratch.
Getting undeserved safety filter refusals, watching LLM accidentally reading what you might not find comfortable showing, and hitting free tier/basic subscription quotas, are not fun. Going local removes somewhat irrational anxiety that come from those.
At some point and for some tasks, predictability is important if not critical.
I’d rather use a tool where I know the limitations, over a tool where the limitations and strengths keep changing.
This way I know where in the process I ought to step in and pay attention.
Recent performance data on my M1 Max 32GB MacBook using oMLX. I have been working on identifying suitable model and config for my use case and system. Using a refactor and suggest improvements prompt for a specific Django code block using VSCode Cline extension.
---
Qwen3.8-27B-4bit, Prompt Processing (PP) 66.3 tok/s, Token Generation (TG) 11.8 tok/s
Ornith-1.5-35B-A3B-MLX-4bit, PP 379.7, TG 45.8
Ornith-1.5-35B-A3B-MLX-4bit, PP 381.5, TG 46.4
Qwen3.6-35B-A3B-mxfp4, PP 389.6, TG 47.6
Qwen3.6-35B-A3B-OptiQ-4bit, PP 342.6, TG 44.4
---
Qwen3.8-27B-4bit generally runs out of output token before completing the task though excellent partial results.
Ornith-1.5-35B-A3B-MLX-4bit seems to get in the loop often specially with tool calls.
Qwen3.6-35B-A3B-mxfp4 seems to be optimal with speed and quality output.
I am going to test Qwen3.6-35B-A3B-4bit soon with same code block just to check my intuition that any derivatives don't seem to perform better than the originals.
> Prompt Processing (PP) 66.3 tok/s
I got 400 pp tps on a 10k token input. Your numbers seem suspiciously low, maybe the input was too short to measure properly? And this dense 27B is slow, the MoE A3B models get to 1000 tps.
What system? If on *M1 Max 32GB* or weaker, I will be interested in learning more about your setup.
Interesting, what's your Context Window?
The above tests were done with 24k context window. Testing was mostly driven by ChatGPT analyzing oMLX server logs and suggesting changes.
Finally, I settled on Qwen3.6-35B-A3B-4bit with 32,768 context window and 16,384 max tokens.
---
Additional results from Qwen3.6-35B-A3B-4bit (Can't edit previous comment)
Qwen3.6-35B-A3B-4bit, 329.7 PP, 41.3 TG
My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency.
I've since acquired two DGX Sparks, and it feels so much snappier.
Would you mind sharing your local Mac setup and which models you currently use and whether it’s GGUF or MLX? I’ve the hardware same specs.
Primarily used ds4[0] by antirez
[0]https://github.com/antirez/ds4
Thanks, that’s what I’m currently using too.
m5 max really fixed pp with the better matmul support, im sure the m5 ultra will be even crazier
the sparks have much slower memory bandwidth is the trade off
I believe the dgx spark is still twice as fast at prefill as the m5 max, but the ultra should get closer to parity.
Another benefit of the 2x spark setup is that you can parallelize to ~6 streams pretty efficiently.
All depends on the workflows you’re using it for.
I’m quite excited for the M7 class machines.
> The main reason to run local: cloud APIs are rented land. They can change their pricing, hit your usage limits, or swap the model being served behind the scenes whenever they feel like it.
Yes but it's easy to replace them.
The main reason should be privacy.
Tip: include pictures of your machine. Not sure if I’m alone in this, but I love to see other people’s setups
Apple did great work convincing people their unified memory was good at AI. Even AI says Apple is the best of all time at marketing.
Meanwhile the stock market has Nvidia at the top... Until everyone gets cuda.
Apple is working from the 'desktop' up to beefy servers with 64GB+ RAM. Nvidia is working from the 'datacenter' down to beefy racks with terabytes of RAM.
There isn't really an overlap yet.
Individual Nvidia cards exist on desktops but they're not really oriented for regular inference so individual developers are left with Macs or datacenter resources as their options.
Uh... Even my $700 laptop with a 3060 can run 9b models.
Maybe a silly question, but is there a reason/advantage to using mac minis over any other kind of small computer/laptop, ie running linux?
My understanding of using a mac mini for ai (ie running a claw bot or whatever) is to have it 'always on' and a better price/performance profile than a cheap vps.
Is there performance (silicon processor?) so unique? As I guess it's not their graphics units. I see tonnes of people using mac minis for AI, to the point it almost became a meme.
Edit: yes I know this article is about local models, my question is a bit more general.
There are probably three main reasons: 1) unified memory (but you can also that with DGS Spark / Strix Halo), 2) access to your Apple account, so you can have a bot handle your iMessages, email, calendars 3) energy efficiency.
Apart from that, if it doesn't work out you still have a Mac Mini, which in itself is more desirable for many than a DGX Spark or Strix Halo if you have no AI use case.
Unified memory is the answer, it’s one of the cheapest ways to get >64GB of gpu addressable memory.
The main difference total other laptops of non-Apple make is/was unified memory (graphic VRAM + RAM) architecture. No need for an extra dedicated graphics card to get 64+ GB VRAM.
No, it's just hype and people cargo culting local LLM guys buying maxed out Mac Studio for its massive and relatively fast GPU-assignable RAM.
Quite a lot of "local doesn't work" in here - unfortunately, often with not much details about what the people actually want to use their models for. Which I'd be curious about.
I, personally, do use frontier models in the cloud for a lot of (meta-)cognitive analyses that are heavy enough to have me run against the limits of payed accounts regularly - so I'm neither a Luddite nor stingy with cash in this case.
However: I have pretty good experiences with local models as well. My solid but hardly extreme desktop (with one RX 9070 XT 16GB) mostly serves gemma4:12b and specialized models (embedding) to my local network. This is for general use like simple queries, simple code, reformatting and the like but also for two specific tasks that are permanently running:
a) It's connected to Home Assistant (as a second stage after very simple "turn light XY on" commands which get processed without LLM). So, I can mumble into my smartwatch "computer, how much gas do we have in the warp core and how much energy did the bussard collectors make from the cosmic dust today?" (or describe a more complex light scene or create an automation I want or whatever). The phone transcribes that - with a local model on device - and fires it to the desktop who has agentic access to HA, looks through the sensors and data, sees that I've tagged my solar panels and battery with nerd vocabulary. It makes the right conclusion, converts a few units and gives me back a nice overview. All hands-free while I'm sitting on the toilet.
b) It's the LLM backend for a personal radio station run by a fleet of nerdy/quirky AI DJs who's archetypes are represented more than well enough in the latent space of the "small" model to produce funny results. The DJs can produce consistent, individual segments and programs, run a playlist that works well for me (based on multi-layered audio analysis that also uses local LLMs), respond to song wishes and generally produce much better recommendations than Spotify ever could for me. And you can also put multiple of them in the "studio" to create hilarious crossovers that you would not get from a commercial entity because the IP owners would rather shoot each other in the face.
All of this doesn't even max the available resources, so I can shovel F5-TTS into the VRAM as well and have all my DJs have good, locally created voices (or voice clones of Captain Picard and Han Solo, if I wanted to) based on zero-shot voice cloning.
--> Far from "unusable". It just depends on the task. And I neither have to hand my keys to the Navidrome server nor to my Smart Home to any entity outside my local network.
> Running a large model locally comes down to one thing: how much RAM it actually needs in memory.
Not completely true. It's memory AND memory bandwidth. You can have 1tb of memory but if you have awful memory-bandwidth you'll also have slow tok/s. A3B helps with this, but so does MTP.
From my experience, you'd be better off running the dense 27b-mlx with MTP than the 3.6 version with A3B. You say your model is ~20GB of ram, but the 3.8:27b-mlx is 18GB and gets me very reasonable tok/s, and greater speed if you disable thinking when not required.
Running it depends on RAM, which is what I wrote, bandwidth is important for speed. I chose my words carefully, but you are absolutely right.
Coming back to this a few hours later, I've decided to add a section to explain this to the blog post. Thank you for flagging it.
The dense 27b Qwen on M4 Pro has a prompt processing speed of around 125tok/s which makes it ok to ask a quick question but impossible to use in an agent, as processing the first prompt of the agent with the tools and instruction can easily be 10 000 tokens
In this case the 35b a3b makes sense as it has a PP speed of around 800tok/s
True. This then boils down to a quality vs speed decision. the 3.8 27b is far better than 3.6 A3B from my experience. I'm happy taking the speed hit, given local models aren't as intelligent as frontier models. Anything that can get me closer to my CC experience both in reasonable speed and intelligence is worth it. With that said CC can also be slow at times, so it's locally the difference in experience is not always noticeable.
I experiment a lot with local LLMs, particularly small ones like Qwen3.5 4B and 9B. I have build multiple experiments to make harnesses that use these models for code generation, planning, local search, etc.
These are really good models but the harness has to be built around them. I have a ton of generated system prompts for specific purposes. Even parts of a SolidJS stack, for example Route management, has its own prompt. These are experiments but the results are real. If we build harnesses around small models, we can build a locally running WYSIWYG editor which works on plain text prompts.
The performance, in simple tokens/second, is not the most important factor. For many private data points, like emails, I would rather have a local graph based search and LLM on top where the harness is specific to problems like calendar, contacts, finance, etc.
I run all experiments on an 16GB M4 Mac Mini but coding agents building the harness are a mix of Codex, Claude Code and opencode.
I have the exact same stack (macbook pro in stead of mini) and swapped from qwen to: https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B
Literally today, but it feels like an improvement.
I will give it a look!
I really like these show and tell style posts. I’m always curious how people have their setups and what tools they use. Also the blog has a nice theme and is easy to read.
I wanna get a desktop Mac for local ai so that I don’t turn my laptop into a delta 15k rpm fan when I run things.
I guess I’ll get in line for one hah.
if you like this style and want more r/LocalLLaMA and r/homelabs have a bunch
I tried the 1 bit model of Qwen3.6 27B on my M1 pro (16G) and got 13 tok/s with only 5G of ram usage.
https://x.com/mkagenius/status/2093730391429685732
(xcancel seems to have received a cease and desist)
If author runs the local model for privacy reasons, then I don't understand why they give Telegram access to all their conversations. It's well known that Telegram doesn't end-to-end encrypt bot accounts.
The beauty of this is that you can just swap out the platform and everything remains as it's the same backend. You make a really good point, one that I haven't really considered, but I also only have so many hours in the day to be fiddling around with stuff.
I think I'd use Discord instead. Clankers are happy to set it all up for you.
Discord is even worse than Telegram in the privacy aspect though!?
Depends what actors you trust. They are equally bad but personally I‘m way more comfortable with the Discord story.
> <a href="https://omlx.app">oMLX</a>
Is that supposed to be hallucination? The human or other kind. Feels like a made up URL. It's .ai, isn't it?
A literal manual typo. Good catch, will fix.
I just use Gemini Pro which comes bundled with a Chromebook; when the 'free' year runs out I buy another, initiate the free years Gemini again and then put the 'as new' Chromebook on eBay and get most of my money back. So Gemini Pro costs me less than £2 a month. And for my needs (investment research) that works well and is a pretty cheap compromise.
You didn't read the article nor understand the purpose.
The way you are using it uses the internet and datacenters. It is costly to the environment.
Running locally is a significant cost savings in comparison.
Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running models locally, if there was a way to inference the model of your choice at a reasonable cost by effectively time-sharing a B300 rack through some privacy-protecting intermediary, would you consider that?
If there was a "Mullvad of GPU clouds", would that solve the privacy concerns?
Chutes, Near AI, Phala and Tinfoil all offer various privacy assurances around inference. Some of the bigger providers also offer "zero data retention".
The problem I have with these is that the guarantees aren't strong enough (Phala, Near) or the models are old (Tinfoil). Chutes is mostly pretty good (cryptographic security all the way to the GPU) but I'm not sure it's possible to cryptographically verify the precise source code they run on the mode.
These are all on my router TrustedRouter, and more providers coming. Tinfoil has some newer ones too like GLM 5.3 now.
Phala isn't verifying all the way down but NEAR is and I know the CEO
yes, and it's already some offerings like that but they all cost a lot because they only good for "I have some idea of workload for N hours or days" lets rent it and run. That fine for some experimentation but if you think about renting something 24/7 even for example to share it with the friends that will cost at least 4x from any API prices as result (something like rtx 6000 48gb will cost ~$470/m).
runpod.io is essentially this. You can rent the hardware for cheap in small time slices. I do this whenever I need to do a lot of embeddings, fast. I have an agent skill that will estimate the optimum hardware to reserve for the time/price constraints of the job, and you can spin up temporary inference for cheap via their API as well.
i've got a "Router of GPUs" end-to-end encrypted: TrustedRouter.com.
I have an m1 Mac 64gb and look forward to trying this out
Not many people share setup with actual setup handholding so that was very G of you
Oh man, an M4 Pro. My old M1 is really starting to show its age trying to run anything bigger than 7B.
Been curious about the M4 Pro for local models. My M3 Max machine still chugs on larger LLMs, definitely eyeing an upgrade.
M4 Pro has been a solid performer for iterating on smaller local models. Much more convenient than spinning up cloud instances for dev.
If you just want chat.
Agents require at least DeepSeek pro and even that is the minimum.
You might be able to get a good model to write instructions and run it in smaller models.
Otherwise, cool your AI got the current weather.
> Agents require at least DeepSeek pro and even that is the minimum.
The Granite 4.2 models which are just recently out, are optimized to handle agentic workflows.
For local models, it's about using the right model for the right job.
Waste of time when I can pay $20 a month for sol.
versus $0 with local models.
There will always be a reason to run frontier models, but local models are well at levels that assist with stuff that don't need that level of complexity.
You could buy a $150 refurbished 16gb i5 and use that model until openAI does its IPO and has to become sane again.
But I guess a $2000 Mac is probably better if you don't care about cost or quality.
> You could buy a $150 refurbished 16gb i5
Or I can use the Mac I already have.
Your example though, Ouch!
~8B Q4. That's around 5-10 tokens a second. Base M1 16GB mac would do 15-20 tokens a seconds. That's a 6 year old machine.
You do get what you pay for it seems.
Would love to see a tutorial on this setup =D
Is there anything one can reasonably run on a mac mini M2 with just 24GB RAM or should I not even try?
It depends on your use case, but the smaller Gemma 4 models or qwen3.6:9b would probably run OK on that. I recommend trying it, even just for fun. It‘s easy with omlx.
Ok, will see if I can find some time for play. Thanks.
Not enough for coding. 48GB is minimum for a non-lobotomized coding model like qwen, and you'd likely want 64GB to have long context and not kernel panic when Chrome opens.
You could run one of the smaller Gemma models to have a chatty Wikipedia.
Figured that would be the case. Thanks.
That M4 Pro is probably a beast for quantised models. My M2 Pro handles 34B just barely; what speeds are you seeing?
> You do not know what these companies do with your data once they have it. They might limit how it gets used, they might sell it, they might expose it.
This is the burning question for me, what are they doing with our hard work.
I'd have thought that sherlocking a user's $10M business would be too high risk, given the billions at stake if real evidence of this happening was found.
However, OpenAI are currently being sued by Apple for trade secret theft, and the way it was done seems to be abundantly idiotic.
So I'm torn.
yes, share performance, numbers if you can, also i wonder if you figured out a way to do a 2way audio with local models, or even explored that. I have a very similar setup but not too happy with the token speed, will try omlx though !!!
I haven't, but that's just because it really isn't my personal usage pattern.
Have a macmini m4 32G, not the pro version, previously everytime I tried local LLM is a bit disappointing, and I finally decide to not waste time and perhaps in the future invest a better hardware to server more modern and dense model
I am curious is what is the 80% request served by this setup, I was using it for OpenClaw which run serveral cron jobs that discover stuffs over the wide internet, check my support system's unanswered tickets, browser X and some social media for me to filter the valued ones(though I have to say even with GPT 5.6 sol, the quality is low for the timeline X sent to me)
Btw, Tailscale is quite cool and did a good job, I was using it to serve the local LLM and connct the openclaw on a Linux Machine to it.
You have tried Qwen 3.8 27B before coming to this conclusion, I hope? It's an incremental improvement over 3.6, but I mostly want to make sure you didn't just try running some old junker before coming to this conclusion.
[dead]
Similar setup here with an M3 Max, it's surprisingly capable. Curious what models you're running on that M4 Pro.
Great setup! Well done!
Performance or quailty of the llm is not the draw for me.
I read those blog posts to remind of wealth gap i have with the average hackernews user.
Thank you for the encouragement, i will work harder to reach your level.
Is this another marketing paid post from Apple..?
It is not
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[dead]