It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.
Gaming used to be done in arcades for 25c a play*.
Over time NeoGeo realized they could sell you the hardware to play at home (“on-prem”) at arcade quality albeit at a price out of reach to most people. Alongside them, Nintendo and Atari gave you 80% of the fun at 20% of the cost. History will view games arcades as a flash in the pan compared to gaming at home.
Namco, Capcom, and Konami went into selling software instead of hardware but it feels hard to imagine buying a home copy of Fable** for $X,000 any time soon.
⸻
* That’s for video games. In fairground arcades you’d literally pay for tokens then use tokens for play. The analogy almost writes itself.
If you think the idea of buying Fable like model is hard, you should really try some of the latest open models like Qwen 3.8. They’re getting there very fast.
Qwen 3.8 and GLM 5.3 are my workhorses right now. Qwen for simpler stuff because it is wicked fast using 'ninfer', GLM 5.3 because it is very good a more complex stuff and less likely to get stuck so I can let it run for much longer without intervention.
I have Qwen 3.8 running under ninfer or Strata sized to fill my 5090.
From a fair bit of coding with the agent, my experience with Qwen 3.8 (Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-0 on Strata, Qwen 3.8 27B on ninfer). In my particular case, I was trying to clone a recent HN post which allowed you to visualize how out of tune your singing was relative to a song, but as a web app or portable native app.
My experience is that both those models will generate an enormous amount of noise (literally: it was running audible unit tests on my machine to test the note-detection algorithms) while coming to a solution, if they do find a solution- while Claude (whatever the latest model is) or ChatGPT (ditto) will do 1/10th the amount of work to get the right solution. I suspect if I had the resources to run GLM 5.3, it would probably be significantly faster to get to a working solution than Qwen-3.8 27B, but likely still somewhat inferior to Claude or ChatGPT solutions, and also would generate a much larger energy bill on my part.
Even 2 DGX sparks are limited to Q2 at best. As for GPUs, it of course depends, but anything under 50k just can't run it, not even Q2.
And at Q2, I think you're better of with Qwen3-coder-next on one DGX spark (or much better: on any apple silicon max, we are currently in the ridiculous situation that a second hand m1 max 64G will beat a DGX spark that's at least triple the price)
It reminds me that coin-op SF2 champion edition was like $6000 new, and you rented time with tokens. Eventually you could play a very good but more limited SNES version of SF2 at home for no tokens. And then PC hardware matured to the point where you can play the exact bits via emulator at better than original fidelity.
It will be very nice when AI gets to the emulator stage.
I'm already nostalgic for very early Dall-e LSD-like art. Ref: "Sleeping Horses - Nov 2021."
Those early GAN image generators were so weird that they produced art that felt genuinely creative. I don't know that whatever people are using to create modern Tung-Tung-Tung-Sahur instagram reel slop is even capable of producing similar things.
I'd say Nintendo gave you 10x the fun of neo geo. Yeah it looked cool, but mario and Zelda on the nes played like nothing else. And then we got street fighter 2 on snes which is better than any snk fighter I've played :)
(I'll run for cover now as I'm sure neogeo fans will disagree!)
I'll agree, since arcades could never give the experience a JRPG could. No shop would allow a player to occupy a machine all day unless they spent a lot of quarters.
> Rent-seeking is the act of growing one's existing wealth by manipulating public policy or economic conditions without creating new wealth.
Game industry has first manipulated the economic conditions by restricting access to game content (skins, modding capabilities, always online drm, etc.) that previously was included in the price (buy game, you own that game outright).
Manipulation of public policy was a more recent thing with the laws surrounding underage gambling via 'loot-boxes'.
So I think the onus is on you to show us why these behaviors, which increase the wealth the companies are able to extract per game, are not rent seeking behavior.
Changing how you sell your own products is not "manipulating economic conditions" whatsoever. If it was, that would mean that every business in the history of commerce has engaged in it.
As for the laws on underage gambling, countries all over the world are outlawing that, suing gaming companies, etc. The public is not in favor of underage gambling, and it's unclear to me how banning it increases the wealth of these companies? So I think the onus is on you to explain how this is an example of rent seeking.
Modern gaming companies can be considered rent seekers not because they charge money for their products, but because many increasingly rely on extracting additional revenue from consumers by exploiting market power, consumer lock-in, and artificial scarcity rather than creating proportionately greater value. In economics, rent seeking is the pursuit of economic gains through the exploitation of existing advantages rather than through productive activity. Developing a great game is productive; leveraging an established franchise or player community to extract additional money without providing equivalent value is rent seeking.
Consider the use of microtransactions, battle passes, paid progression, and downloadable content. These systems often exploit the time, money, and emotional investment players have already made in a game. By deliberately slowing progression, withholding content, or introducing artificial scarcity, companies can encourage consumers to spend more money not because they are receiving a better product, but because they want to avoid frustration or access something they already value. The company is monetizing the player's existing commitment rather than necessarily creating new value.
This does not mean every gaming company or microtransaction is inherently rent seeking. Selling meaningful new content is a legitimate business activity. The criticism is that parts of the modern gaming industry increasingly prioritize extracting revenue from an existing player base over developing better games. When companies use their control over popular franchises and digital ecosystems to extract additional money from consumers who have few alternatives, they exhibit the very characteristics of rent seeking that economists criticize in other industries.
You've stumbled on exactly why the big providers are lobbying for regulation and big claims that Ai is unsafe. They don't want you running these at home.
Arcades mostly failed to adapt to the advances of home consoles to provide experiences you couldn't get at home. That's on the dimwitted game suits of the 1990s who cared little about novel gameplay, networked gameplay, or better NPC AI and physics. Then mobile happened and there was a brief glorious Cambrian explosion of new ideas until the Zynga playbook and DLC. Evidence: lived through it as a small game studio developer trying to do original things. The fun experiment now and then is to break out your 1980s video game consoles and play through their libraries. Most of the games are objectively terrible but gems like Yar's Revenge are still gems today IMO.
It will take a while till prices drop to a level where affording 256GB card (even if it would be shared between few coworkers) will have any sense over buying tokens.
And the AI debt bubble price spike will probably stay there for a while.
That's going to be pretty pricey though. And I really want to see some independent benchmarks, that all looks very good on paper but there are some complications (notably: chip count, run length of the traces).
Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
My local model never says no. I never get banned. If I want to do security stuff, reverse engine, there it goes! Do I wanna chat about politics? No problem! Do I want it to analyze medical stuff? I don't get to hear that it's not a doctor and I should call 911. Do I want to explore legal docs, no problem! There are lots of things that cloud models will say no to! I read that don't even agree to help with advanced biology and chemistry topic much. I also never run out of tokens, no 5 hr limit. I can run it 24/7. I don't run out of tokens. I can run 16 sub agents at the same time, non stop! 1-4 billion tokens a month? No problem. Any agent harness I want! No restrictions. Privacy all day. My own true system prompt, not something someone has determined, no filters. Furthermore, there are many open weight cloud models that don't have equivalent online. For a while before cloud models got good at math, the best model at math was DeepSeekv2 Math and there was no API. The only way to use it was to run local. All sorts of audio models, image/video models, etc. No multiple subscription needed. Private UI for my family, kids included (I can guard what kids infer). Got a friend that really needs some tokens? I give em access through private VPN. Telling someone that they are paying for decades of Claude for local AI is like telling someone they are paying for decades of public transport when they buy a car. Sure, you are right, but ...
If you already have a machine, throwing an r9700 in it is one of the cheapest ways to get qwen3.8 27B Q6 with 100k+ context working.
Sadly, the r9700 has gone up in price, but it's still the cheapest route. I use It on an old x99 Linux desktop and it achieves 50 tps decode and 1000+ tps prefill.
The build uses my machine's ROCM and I just link it instead of embedded all that in an image, like the published ones do.
Then deploying it via docker compose to my local machine. The models I want to run are stored in a docker volume, so it's easy to switch the model while keeping the same build.
The biggest negative for me is that I don't think it's enough to do parallelization through llamacpp (I'd prefer to work on one thing faster than two things slowly). I haven't gotten vllm working yet. Also, once you have one, you constantly start thinking about a second card...
> No political censorhip? Try asking Chinese models about Taiwan.
Self-hosting means being able to use abliterated models where the superficial instructions like not describing Taiwan as independent are burned off.
Censorship and bias are related but not synonymous. I agree that it's harder to remove political bias when the training data includes it. We know many of the frontier models are heavily trained on Reddit, for example, [which likely explains why we see such heavy left wing bias.](https://trakkr.ai/bias/findings)
>abliterated models where the superficial instructions like not describing Taiwan as independent are burned off
Does this help with issues like including security vulnerabilities when you say you're in Tibet? I understood that this was burned in at the training level, not just a layer of censorship added at the last minute.
It could, but it's an inexact science. A little bit like tampering with the brain. Even a surgical procedure removing a tiny tumor can interfere with critical pathways. Usually abliterated models end up almost exactly as capable in benchmarks, but it requires the person doing the abliteration to anticipate or know about the censorship. In the case of Chinese model censorship it is believed that it is post-training, so the underlying data input is not the issue. It's how the model has been trained to use that data. Specifically, abliteration usually targets the refusal behaviour. I would guess that this example could be significantly improved, but I have not seen any abliteration attempts re this specifically.
All good points, but I am not sure who I trust less. Chinese companies or random people on Hugging Face. :) It is still a black box. And I have a feeling I could not abliterate those models on my own hardware.
I am not saying I would not use any of them at all, but I would avoid doing certain tasks with them. And with the testing I have done with abliterated models in the past, it is not 100% successful and it can still hit the same walls as original model.
I agree. I think the future for privacy conscious people is going to be an adversarial open weights model, where competing models (ideally from opposing superpowers and labs) critique each other's work to find consensus. Or at least alert for suspicious behaviour.
I mostly run Qwen3.8 Flash Next on 2xR9700 (but Intel Arc Pro B70 may be better currently, or 4x3090 is even better but out of budget, 32GiB Nvidia GCs on ebay are also usable).
I have 128GiB DDR4 (3200Hz), but 64GiB is enough, even 32GiB maybe, I just reuse it from an old project.
You may need a AMD EPYC processor and associated motherboard, don't forget a good PSU at least 1200W (300W processor + 400W by card to support spikes of consumption).
On some setup (~15k€) you can use 4 CG to go up to 128G of VRAM.
I'm not using a EPYC processor for now, so I get 170tps on decode but very slow prefill, the bottleneck is the PCIe exchange between the cards, I need to setup P2P (it means what you think it means, for GCs).
My setup supports full context (2*18 bytes so 256K), I didn't setup YaRN to get 1M of context (2*20 bytes), but it's an option with 128GiB of RAM, the visual component is extracted from RAM so a little slower.
I'm not using multiple agents but some people does, it's god like with 4 graphics cards, on 2 only, it's relatively quick but limited to single user and up to 3 to 4 concurrents requests only.
It's not frontier, but look AA of QFN, it's good enough. DeepSeek V4.1 can run on my setup, but it's slow. On dual B70, there is some PoCs to make it usable (>60tps). It improves every days.
If you need more information, look for local setup users on the Web. There is a lot of people sharing information about benchmarks and setups. L80 is my reference for now.
It's a good first step. They're not great but they kind of work for many tasks. I think the small models can be useful in context-specific tasks but the friction is high right now because the user is forced to determine whether Siri is good enough, or a better LLM is required per task. Ideally Siri would be smart enough to intelligently route. Apple claims that's how it works but even their cloud processing is far behind frontier models.
Worryingly, they're taking a walled garden approach, and they don't want to expose user data to competitors. After the monumental failure of Siri over more than a decade, I don't trust Apple to execute competently on AI.
It will not do that unless you specifically teach it to do that, or you use an Anthropic model. The issue in this case was Anthropic's safety monitoring built into the model. They record everything and flag potential illegal activity for human review. This occurs even over the API. I do not know if OpenAI does this, but I would not be surprised. I think the Chinese models are far less likely to do this (unless one resides in China).
I’m suggesting it’s highly unlikely it will call the FBI to dob you in. That affirmatively trained behaviour, not something one should expect to accidentally happen.
You see it as money spent, others might see it as an investment. OP doesn't necessarily say exactly what they use it for, but if education is the goal, $25k and spending your evenings learning about AI theory and tooling is way cheaper than a university degree and can easily result in millions more dollars in lifetime earnings given the current early state of the technology.
Besides, I regularly see people here in HN spend a LOT more than that on hobbies and depreciating assets. A few years back, people with coastal salaries and tech jobs were tripping over themselves to spend $100k on a certain brand of trendy electric car.
I can’t “invest” $25k into an LLM. The vast majority of people can’t. And I’m not spending $5k to timeshare it with 4 other people or whatever lol I currently make do with a 9070 + 32gb of ram.
AnyONE. But if you got some friends, and they got friends, then that $25k becomes smaller and stomachable. My favorite rumor out of LA is groups of friends with a Lamborghini Urus that they timeshare. No idea if it's true, but if car people can get together to buy/lease a car that's MSRP is $345k, I think $25k should be an option for anyone with friends. (Hypothetically. I've read a lot about this friend thing on the Internet!)
If 10 people are sharing it the speed advantages suddenly drop drastically. That’s $2500 a head. 5 would be $5k. Then all the rules and such you need in place
This is all this stuff that dries people to pay for frontier models. Even running modest models requires a few thousand, unless you just want really lean ones. People don’t need frontier but it’s not cheap to effectively run, say, a 16gb Qwen 3.6 or 3.8 in the 30b range. Ram is like $15/gb and GPU’s with 16gb vram are hardly cheap.
In 1993-94, I upgraded the RAM on my home 486, which started with 4MB. I doubled that to 8MB for $250, then to 16MB for another $250, and finally to 32MB (the max for the motherboard) for another $250. At the time, I was a poor undergraduate and had to give up other pleasures for these expenses. If I estimate correctly it's about $30K/gigabyte (only high end servers at the time had a gigabyte, and they would often cost $100K or more).
The lesson it taught me is that swapping is slow and that if it happens during your software development, you're much less productive. When I finally maxxed out the RAM on my machine, I could run linux + emacs + X11 + bash + g++ all at the same time, making me about 10X more productive. It's an investment and lesson that has helped me build my career over decades.
Unless I read it wrong, it sounds like this guy did not get it paid for by his employer - he runs multiple companies and is an angel investor with clearly deep pockets - and I have people talking about it being an investment in their careers. He could be using it support his businesses, yes, but that needs to be clarified and is still not a situation most people find themselves in.
This is not a straightforward business expense conversation IMO. Otherwise yes I do agree with you. A company should invest in supporting local models for their team.
Anyone? I know many people in Silicon Valley, Boston, Seattle, New York, and elsewhere who would consider that a reasonable expense if it kept them viable in the job market for a few more years.
My entire career has been based on purchasing the most hardware I could afford at home and using it near its peak. I've given up other life pleasures (dinners out, nicer car/housing) several times to max out my computational capacity.
Also hardware like this, if chosen carefully, often has a viable lifetime over 5+ years
I’m not spending $25k to “stay in the job market.” I come from film production, I’m no stranger to owning my own hardware. But that’s ridiculous. And at least people rented my gear.
In case a future user wants to know what a 2026 model would refuse to do, I needed to rotate dozens of passwords last night both on the server and on the clients. When it was time to verify that the rotated credentials worked, I asked for a script that would rate limit the credential checks. Claude/opus5 (opus!!) refused and stopped in its tracks.
In another instance, Claude was making great headway bringing a legacy Windows app up in WINE until it’s saw “auth” somewhere and shut down. In the shutdown state, Claude asked me to reword the prompt to see if I could get it to act on less dangerous instructions. But when I reworded my request, narrowed the scope to be clearly unthreatening, or even redirected the model, it’s still rejected me. I suspect Anthropic uses the less suspicious rephrasing as training material to increase their rejection rate.
I agree the local models have many nice properties, but I would not be surprised if people start to recognize that Gemini pointing you at medical services instead of attempting to diagnose you is actually a good thing in many cases (and Gemini often does do some diagnosis anyway).
It's pretty easy for a reasonable person to determine when to call 911 and when to ask an LLM. Typically when I ask an LLM, it is for something I am not willing to escalate further at the moment.
Anthropic has a significant interest in reducing liability, which in this case means mostly blocking this use case.
Saying yes is not equal to "provides the requested information." If you use models for affirmation instead of information, this is a more reasonable concern.
Have you heard the cost of health insurance and seeing a doctor? Some people don't see going to hospital as an option. Better for them to have a local model than no options. But let me tell you how I use it for health, I take all my health records, lab results, every thing, I throw it in there for a second opinion, it explains things to me, it suggests questions I should ask on follow up, possible options, etc.
The security research is probably the biggest reason to use local models as it seems like the big service providers will ban you for life if you even hint at doing that.
If I want to go out of business that is what I would do. Most of the data I have here could never be shipped to a service provider, I'd be in lawsuits for the next decade or so and die bankrupt.
That NDA case is the real argument, more than price. Token prices keep dropping anyway, but "this data can never leave the building" doesn't change with a price cut ;)
Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients.
It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money.
On a tangent: this is a really good website. I understood what you do and how you do it by reading down one page. There's a lot of projects/companies I immediately tune out from because I can't actually figure out what they do amongst all the buzzwords, animations, and excess scrolling in every dimension.
Isn't he the guy who posted about his LEGO-sorting computer vision project? That post alone taught me things that increased my career opportunities significantly.
The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.
They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.
I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.
> The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.
But, there is no need to make that bet - when prices increase, I'd look into renting a previous-gen GPU on a cloud provider. If the online GPU providers raise prices, then I would look into purchasing for on-site.
Until that happens, I'm not making any bets, because if all the token providers go bust GPU rentals are going to fall in price, and if they don't go bust after raising prices, I can still get years of GPUs for self-hosting at the price of a self-owned rig.
Well there are risks to being part of a crowd that's all compelled to make the same decision at the same time. Certain risks can only be transferred, not reduced in absolute terms.
It's not economical today to run my own hardware, but the insurance is worth it to me in the sense that if my bet is wrong, I'm still okay with paying extra for the protection.
Inference prices are going only down. It could be that some top subscription costs more in a decade, but I fully believe a full array of competent models will be extremely cheap.
Keep in mind that the prices of the big AI companies are currently mostly loss makers. Their plan is to get sufficient market saturation and dependency of their clients and if they achieve that, they will hike up the prices quite a bit.
(Of course this all depends on if their plans work out, but if not then the smaller local models will probably play a big role in the failure of their plan)
People keep repeating this, but it is only remotely close to true for the literal free plans and the small subset of consumers maxing out the pro plans.
At API pricing (i.e. what all enterprise customers are paying), inference is profitable, even if they're paying usurious rates on cost of capital for capex. Where the frontier labs are getting screwed is that they're in a Red Queen's race with each other, spending tens of billions of dollars per year training the next frontier model, and not being able to recoup the investment on that model before it's obsolete.
That's why you see the CEOs calling for government intervention to "pace the frontier": it lets them reduce investment on training without risking that their competitors keep going and make them obsolete. Unfortunately for them, the Chinese labs exist, and would love nothing more than to take the frontier, so they're in a bit of a pickle.
The people I personally know who signed the "pacing the frontier" petition (including me), or who called for it in other ways, are sincere in their concerns that things are moving too fast for safety.
People keep repeating this, but it is only remotely close to true if you evaluate inference in a vacuum without any of the other supporting costs to facilitate that capability.
Just a few years ago people were saying the same thing about WeWork: "leasing is profitable, the unit economics of individual locations is net positive", and yet...
I was thinking the economics of it the same way.. if the AI companies are burning debt and VC money to give me cheaper tokens why not take them for it?
I can always rent a GPU server for those personal, private AI needs.. heck even run a decision model to route them to the rented GPU servers either on demand or scheduled for few hours in a day…
It doesn’t have to be black or white…
For my toy projects, adventures, learning exercises, and throwaway MVPs I will prefer the subscription plans… they are cheap for what they offer today.
From this article alone it should be obvious that the long-repeated myth about frontier inference being subsidized is just that: a myth.
It's training and tuning new models that cost them money. And that's why they are trying to scare people into thinking AI will kill us all in the next 5 years; they want bans on the open-weight models because they know they won't be able to compete with them.
A few days ago we got the news that DeepSeek makes 38 dollars profit per card with their off peak pricing (in the inference server release discussion).
If that is true, either the American labs are not optimizing correctly or they make a ton of money with inference.
There are lots of reasons Chinese AI companies will always be cheaper than Western AI companies, but chief among them is that China is pouring ludicrous amounts of money into new energy generation projects.
Hydro, wind, solar, nuclear, coal, they are building it all, as we speak, at least 10x more than the US. They are aiming to not just be energy independent, but energy dominant.
You can't now (and certainly won't in the future) be able to run an AI provider in the West as profitably as you can in China.
Yeah, the way I see it short term is take the free/subsidized tokens while we can and the further out we go the cheaper compute will get for local setups.
This trope minimizes so many downsides of "frontier" models. When you can't do what you need to with a model it really isn't valuable - which is why subscriptions still exist. As soon as you're forced to pay actual token costs people who keep saying this will likely wish they had invested in something else earlier.
It's also very clear that providers are running variants of models, especially at the SotA levels. I wouldn't be surprised to find out at some point that 1-2 months after release OAI and Anthropic just run "good enough" quants to save themselves on inference (since they're losing money hand over fist). Another upside of local models is that I don't need to worry about model degradation at any given point.
If you know you will pay Claude a few thousand bucks a year a decade adds up too, owning the hardware could just as well end up cheaper because you can sell it or as others have alluded the price to rent could get much worse.
Would you really expect to be using the same hardware over the course of that decade? One of the biggest drawbacks to using on-prem hardware is keeping up with the upgrades. Leasing upgrade cycles are a thing for businesses, but for someone at home it seems like later in the decade the local system will be showing its age.
The 3090's I have would sell for more than new today... for the next couple of years or so it will be pretty bad on the hardware front. Until there is more RAM manufacturing capacity and NV has a credible competitor I don't think it will get better.
The benefit of the agent working firstly for you rather than for Anthropic or OpenAI will become increasingly apparent. We're already seeing signs: remember when Anthropic openly promised to have Claude sabotage people using it to create competitors? And OpenAI is looking for revenue opportunities from advertising, which can't end well for their users.
Just think of how user-hostile other online services have gotten, and imagine how acutely painful the proprietary LLM providers could make theirs. Thankfully, open models provide some counterbalance.
Would you use hosted models if they promised anonymity and no log retention and servers were physically located inside jurisdictions that respect privacy (Switzerland, Norway or similar, I'm not into law) if that meant better price/performance ratio than your current rig?
I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
V100S is a very good card for some workloads. And they were cheap as well.
I have 240G VRAM total which is a good reason why that rig cost as much as it does, and if I were to build it at today's prices it would be even more expensive.
I got mine just a few weeks ago for about $750 each. I am a decent programmer and have Opus optimizing the inference engine. I know enough to keep it on track, but GPU programming is something I have literally never done, before, but now I know enough about Volta and how to optimize it to be dangerous.
Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.
And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)
I tried getting some 170's when I first heard about that unlock but they were already unobtanium by then. That would have been a nice way to add a lot of VRAM.
GLM 5.3 runs pretty comfortably on this rig, even with large context.
You mean, the Chinese models, right? Like the ones that stole all the weights from ChatGPT and Claude, just like those companies stole all the IP from the rest of the world, right?
I’ve always wondered about those models. Are we just sending all of our data over to China, instead of ChatGPT or Claude? Can we trust them any more than we do ChatGPT or Claude?
Would it make more sense to just rent the infra from a Cloud provider just for amortization purposes? While...and this depending on the the Cloud provider...getting the same guarantees of governance and personal sovereignty?
Bedrock guarantees neither your inputs are shared with the LLM providers, neither your data will be used to train the next model. I dont think AWS loves AI that much...to break their business model with a violation of terms and conditions.
But I was not even suggesting that, just run your local LLM on Cloud infra. Do you really believe they are ready to risk their Cloud business just to spy on your prompts?
Well... that's a tricky one. I bought a lot of these second hand. I can sell them for more than they originally cost new, so indeed, it is not 'cheap'.
Then again, I run a business and for the business it is absolutely manageable, it is roughly the price of a single job. Privately this would be a different matter but I assume that if you would do this privately you'd simply get a single (but powerful) GPU second hand and offload to RAM. Not as fast but with llama.cpp, 'ninfer', DwarfStar and various other clones of those two you can get some pretty decent performance out of single GPUs.
I've been buying powerful hardware since the 90's, starting with SGI and then ever larger Linux boxes. I make my living with them so it's never been a huge decision to invest some of the proceeds back into the tools. But on an absolute scale, yes, it is a lot of money.
It really depends on usecase i guess. I've tried a few open models, they don't hold a candle Argon/Opus/Sonnet when it comes to my current major use case which revolves around CAD and aero simulation development work.
What's your use case where the open models are working well?
I mean IBM already has a track record of building gigantic amounts of garbage and over charging large corps and govts for it coasting on their brand name and networks to get deals. So using some cheap ass model fits their MO.
Having used their shit mountain called websphere, what an IBM CEO says is not something I take too seriously.
IBM has two faces. Their consultancy arm is the exact opposite of what I would call a good company, even so, they make a good bit of money. The hardware side of the business is still quite impressive, yes, it is too expensive but their power rack boxes are effectively a little mainframe (check the IO stats on those things) and the performance is - I have to admit - impressive. That said I would never buy it because I can get much better value elsewhere. Even so I think they could do this if they really wanted to.
They do have big advantanges because many large corps have become totally dependent on them having outsourced everything and lost internal knowledge. That internal ops knowledge is very crucial/useful to take full advatange of what AI offers. In the coming years much of it will leak into frontier models. After which corps might not need IBM/TCS etc and just rebuild internal teams to handle ops like they once did.
> Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct)
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
You need to reread what was written. No one is proposing to take away your right to leak proprietary edge to the rapacious frontier labs that need to eat most of the economy in order to survive.
You can try open models via OpenRouter and omp without having to buy or rent any GPU. I routinely try open models to see where they are. GLM 5.3 with DSFV4.1 as advisor via omp feels Opus 4.6 level, i.e. totally able to ship stuff.
I do use Anthropic and OpenAI mostly because it costs me 1/40 and 1/50 of the api prices, which is way cheaper than Chinese subscriptions, but I’m ready to jump ship when they pull the plug. Fortunately they have nothing that forces me to keep paying them, and I don’t pay the crazy enterprise subscription costs, just the personal account prices.
I also die to be able to run the OSS setup locally to stop being spied on all the time.
For context, I averaged 10 hours a day prompting LLMs from when ChatGPT came out. At the start it was just function generation, then single file modules, then modules, and now I reason, design, implement and test with them.
An interesting thing is that with omp you can plan with a smart model and then implement with a dumber one. So for example, plan with Kimi 3.1 via OpenRouter and then implement with DSFV4 with zero data retention active, sorted by cheapest, and minimal tps accepted.
I get the people who are pissed about their cushy jobs evaporating, but if you know how to use these technologies it’s very freeing because it’s like having a company of people working for you, except they are not human so they’re not wasting their life working for you instead of using it for themselves.
You can make them do pretty much everything that is fuzzy-verifiable, you don’t have to worry about their feelings or that they won’t do what you asked them to do or stop, and you can activate them when you want at scale in parallel or not use them at all for days.
For me it’s a dream honestly. I never felt so much empowered in my life while being enabled to unlock my creativity with almost no boundaries.
As the OP said really. It is very very freeing to just be able to give a task that in ye olden days you knew was going to be a pain in the arse to do, and it will do it without you. Amazing.
It's like having extra copies of yourself that you can just fork and get stuff done. E.g. the other day I was in a meeting and there were some heavy discussions about a few gnarly issues. I fired off a few prompts in parallel while the AI researches options, approaches, made a prototype etc all while I was present and paying attention still in the face to face meeting. I am so much so much more efficient as an engineer now - getting better feedback, more bonuses etc. It's never been better - I handle thinking about the requirements, the computer science and software engineering big picture etc, and the AI handles the syntax trivia for implementation.
The models are really really good now.
The only downside is that as a result of more stuff getting done, we need to have more meetings to discuss what actually needs to be worked on.
You didn't really answer the question though, how are you not nervous about your job security and how do you still see any value in software that you make with almost 0 effort?
People at work are cutting and pasting Claude output for use in routine messages on Teams, troubleshooting suggestions, and proposed architecture designs, and honestly it's some of the dumbest shit I've ever seen. I have to tell people to reword it in their own words, and I believe they often can't.
I've seen developers use Claude to make suggestions for infrastructure design that made absolutely no sense and did not consider our environment. I've seen people write English that grammatically made sense but logically did not.
I don't see enough complaints about this and it can't be just my company.
Is anyone else seeing really severe Dunning-Kruger?
I always check what AI says to me, and I use my own words. I think the average engineer is not.
That it's a degradation in some environments? There's not a lot being said about this. The laments are usually developers sad that they're prompting instead of coding. My lament isn't about the art of it, it's the human reasoning that seems to have gone away in my workplace when it comes to engineers.
Think bigger. Don't think in features. Think in business workflows.
You are a consumer, so you naturally think like the consumer you are. But consumers want cheap stuff and only big big businesses are able to provide cheap stuff at massive scale. So stop thinking in b2c. For us little businesses the money is in b2b. Businesses pay lots of money to have problems solved. They want to focus on their core business and outsource the rest.
Consumers hate paying subscriptions. Businesses love paying subscriptions. Because then they can make their problems someone else's problem and focus on what their actual business is.
Maybe it’s 1% or 10% today, but a decade ago, having/assembling the technical chops to know how to build something was a huge part of the meat / barrier to entry.
It can still be true though, because entry is nothing without sales and a continued search for product market fit. I don’t know about 1% but if you say ‘for every 100 apps launched in 2015-2020 only 1 turned into a sustainable business’ I’d say that’s in the ballpark.
If code becomes cheaper that may change the business fundamentals but it also means capturing customers becomes way harder.
In a former life I was a freelancer who built first version mobile apps for seed and pre-seed startups. I’m a decent dev, so what I built was always good enough. Virtually none of these startups survived, and there were dozens of them.
Even a decade ago marketing/growth was the massive lion's share. IMO just building something has been near meaningless since at least post-Ruby on Rails. The app stores have been inundated since the late 2010s.
I've hit a weird point myself. I'm not overly interested in building software for other people now. What I'm able to do is build all my stupid hobby project ideas for my very specific use cases and not have to worry about making them profitable in order to justify the time spent on them. I just built my own point of sale system because nothing on the market meets my needs, and my needs are extremely niche.
I sell vintage cameras in an antiques centre, but only 3 days a week. I’d like to sell online too but I have no ability to put aside stock for that, and similarly I only have access to the stock a few days a week, which makes shipping awkward. Pretty much every item is unique and needs listing separately, which is a pain. This is probably doable on some existing POS but it’s almost certainly much more expensive than just building my own (I think Shopify starts around £40 per month for what I’d need, then still require a lot of customisation). A major aspect of my business is buying stuff in the stall and tracking its eventual sale price, which tends to be a pain on other systems when it’s all one-off unique items.
I also do repairs and so I recently added a job management system into the POS, which has been really useful so far. Next step is to sort out the online store/eBay integration aspect!
Take your time to set up the tools and just build something small. Create a personal website, or a calendar that helps to track friends' birthdays, or something else of that sort. It's a good way to calm yourself down, because it will give you a good picture of both the capabilities and the limitations of AI.
I've found it very liberating that I can handle some of the tedious parts of a project to AI. In the end, I spend perhaps the same amount of time on a project, but more of it goes to important work and less gets wasted on plumbing.
In OMP just setting the "task" role (which controls the default subagent model choice) to some cheap Flash model like Deepseek or Gemini goes a long way towards making better use of your quota.
Sol and Opus both are quite eager to use subagents and since the subagent tasks are often well specified and then verified by the main model there's hardly any advantage to using anything better for the subagent anyway, so IMO it's a no-brainer.
For me switching the main model to Kimi K3.1 Pro would put me towards 100% open, but I don't know if it uses subagents effectively. Any other open models I noticed just don't utilize them very much.
People are underestimating the lock-in of Claude Code and Codex. Most companies are not buying a model, they are buying the ecosystem.
But the lock-in is not _that_ big that a big co could not pull off a migration, but it would be like migrating from Teams to Slack, not like migrating away from Adobe.
This is exactly how I got people at work onboarded to our local models. Everyone was comfortable with Claude or codex CLI so i gave them prebaked config files to use them with our local models. Ezpz
They do when they see the cost savings and ROI. It’s the one reliable thing that makes a company switch from any tech to another one.
Looking at our AI spend today vs 1 year ago we have massive changes. Anything that can bring this under control and potentially save millions a year is being explored.
We will probably end up with a combo of open weight models for most work and frontier models for key projects. For now. Who knows what the landscape looks like in 1 year.
I didn't think that IBM had gear that mortals would want to run. If you're not running a mainframe, what do they have to sell you? They still sell Power servers and it looks like they have an AI card that can be used on both. But, if we're talking about the IBM CEO talking about on-premises, what are they really talking about hardware-wise? It's not something they would sell, is it?
This is more of a statement for their consulting customers, right?
I can picture that they’d have the software side sorted out with RedHat. But I don’t know what hardware they’d push to support local on-premises models. It looks like when you buy Fusion HCI, you can choose from Dell or Lenovo as the underlying hardware supplier.
"Affordable" and "IBM" don't belong in the same sentence together, but they're happy to sell you their inference-capable hardware if you've got the wallet for it.
I don't agree with that. They made the PC revolution possible by making hardware that was at least affordable enough to be bought by a large number of businesses and individuals. It also set the stage for the clone market. I'm pretty sure that if IBM really set their mind to it that they could do this.
That doesn't detract from your overall point that they tend to make overpriced stuff, but I don't think they have to do that. Besides, what's overpriced? NVidia's GB300 workstation sells for 95K euros, a single RTX 6000 is now 20K. Surely even IBM can compete at that level.
True, even so: They still make some pretty impressive hardware. They have access to fab capacity, they know how to make chips. They have the capital. If they really wanted they could do it.
As someone who worked for the big blue not that long ago I would agree that they have the capability. The top heavy C-suite might have to get over themselves and their buzzwords and actually try to create something besides an arbitrary EPS though. I wouldn't recommend holding one's breath.
In a way it is, but I would see that as a starting point, not as the ready solution. They'd need to do what NV has done for their eco-system: fantastic libraries, a credible future upgrade path, easy ordering, manufacturing partnerships, the works. I figure besides the performance gap that they are at least 5 to 10 years behind NV with their present offering. But NVidia is dropping stitches and IBM should take advantage of that.
I think IBM sees the Spyre as a reason to sell more Power hardware but that's just not where the industry is right now and I doubt it will be enough to create sufficient demand. Who knows, I don't have any insight in what they're doing there, but there is massive gap between A GB300 workstation and what IBM currently offers. At the same time: the RAM crunch hits all parties and that's a major obstacle as well.
Yeah. For all the jokes we make about paperclip maximizer, we've figured it out. It's not paperclips, it's RAM.
That's the thing though. Software is cheap with AI, but Nvidia has Cuda. If software doesn't matter anymore (it does) because it's so cheap now, then Nvidia's moat evaporates to IBM's Spyre. In reality "just a little bit of software" has inertia and cannot be dislodged quickly, quietly, or easily. cf Microsoft.
That. And not starting every sale with 'call us' where they first want you to tell them how much you can afford to spend. I buy a fair amount of gear and if the sales process starts with some sleazeball sales guy instead of a mouseclick on a local webshop then I'm out.
As for CUDA: I think CUDA is a formidable moat but I also think that that is precisely the kind of thing where the likes of IBM could (if they really wanted to) make things happen. They're already in the space with Redhat and they have ample knowledge about drivers and low level code. Putting together a team that either makes a drop-in replacement or alternatively, that starts supporting their custom hardware for the big players in the inference space (VLLM, llama.cpp and Torch) would make all the difference. They'd have to keep doing it though, even if the initial results were lower than expected because that market is not going to turn on a dime.
You'd think that with a 250K workforce and an active line of hardware including an AI accelerator that those people are still working there. I could be wrong, but someone has to be doing that work unless IBM in turn outsources it.
I know. But: they have gobs of money, they still have some talent, they still have a hardware department that I'm pretty sure would jump at a chance to give NV a run for their money. This would require a massive long term investment and a commitment that I'm not sure they would have the stomach for.
They do? They are a spec in the ocean relative to big players offering inference and training their own models. Pretty sure one thing they are struggling with is the money to compete successfully in the ai game this time around.
10 billion give or take, market cap 250 billion, 250K employees. That's not a 'spec in the ocean' by normal standards, and IBM is a mature company, not an inflated balloon.
When that happened I'd been working at Tandy (EU version of Radio Shack) for a while and even though we sold computers they were not taken serious until IBM came along and then suddenly we couldn't supply them fast enough.
Yes, that was the first time I had a 'proper' processor to play with. A Dragon 32 (CoCo clone from the UK).
I've never seen Tandy used outside of Europe but then again, I had not traveled as much then as now so it's definitely possible they used the brand on other continents as well. I wonder how many trees were cut for those catalogs :)
It was only a difference in the store names. They were sold at Radio Shack stores in US. Tandy, originally a leather company, bought Radio Shack in the 60s).
Yes but unironically? IBM should be what NVIDIA would be competing with their NVL72 and DGX products. It doesn't make sense that IBM is just watching NVIDIA take its lunch.
I just can't believe the price of hardware these days, a 3090, a graphics card released in September 2020, can still be worth £1500 or more. Honestly everywhere is sold out below around £1800. Can you really trust them not to have been rinsed on eBay and have an extremely short usage - they are still around £1150 used there!
We need a step change in the amount of silicon chips produced. This situation has been going on now for over 2 decades, initially bitcoin but ever since various high compute scenarios keep causing GPU shortages and now RAM and storage as well. This one isn't likely to go away anytime soon so we need a drastic increase in production to bring pricing down to reasonable levels.
I dunno, I always thought the AI craze started from ChatGPT which, in turn, was mostly driven by Google research papers.
Yes it would have costed them more to train the first few versions of ChatGPT if it wasn't for the crypto mayham, but it would still happen. The hardware was always available and still improving regardless of crypto or not.
It's going to boil down to cost of the hardware vs. pay per use cloud models. I'm on the 100$/month chatgpt plan (~88euro). With all the resets, credits, and all the rest I typically don't run out of tokens. This is fine for me. I might go to the 200$/month plan at some point.
In comparison, my highend mac book pro costs 105 euro per month (leasing cost). That's not quite enough to run the bigger models and besides running the GPU on a laptop constantly makes it a bit hostile for my lap. And the model I would need a lot more memory for the bigger models. It's nice for experimenting though.
If you get your own GPUs in some kind of server form, you could run those locally or in a data center. Or you just get hardware resources from your favorite cloud provider. Hetzner (one of the more affordable ones) GPU servers start at around 200 euro per month. I know some people that use that with open weight models. Works fine, especially if you can share the resources between multiple users.
At which point of course you are doing the same as cloud based providers are doing. But probably with a bit worse economics. It all boils down to whether you are actually saving money or paying for a lot of underutilized hardware that's running 24x7 when your people actually only show up about 8x5.
I don't think most companies should be doing that. It sounds like they'd end up with a lot of idling over capacity at a premium. The only valid reasons to do this would be securing access to GPUs or some very urgent need to keep things on premise (which a lot of companies insist is true for them until they realize how much they are going to pay).
> It's going to boil down to cost of the hardware vs. pay per use cloud models. I'm on the 100$/month chatgpt plan (~88euro). With all the resets, credits, and all the rest I typically don't run out of tokens. This is fine for me. I might go to the 200$/month plan at some point.
It's also about "insurance protection" against the AI company raising the prices or changing their terms (e.g. becoming more frugal with respect to token drops).
The speed at which you can switch is probably proportional to your dependence. But the thing with insuring against risk is that you need to cover all of them and not just cherry pick them.
If you are running your own models on your own hardware you take a bunch of risk as well. Is your hardware capable enough to run your models? Do you have enough of it and access to more when you need to? And is the stuff that you picked still going to be good enough in six months? And can you trust those model providers that you picked to keep up with the rest of the industry or are they going to fall behind and leave you stuck with obsolete but still very expensive technology.
Of course, this is an article about IBM who are in the business of selling consulting and services at a premium. They would probably love companies to self host their models on infrastructure that IBM supplies and consults on (for obvious reasons). The question is, should you buy what they are selling or can you do better? My simple observation here is that if companies like IBM are getting excited it's probably because they are looking to separate a lot of companies from some pretty significant cash. So, I'd be wary to buy what they are selling. But that's just me.
In a rapidly evolving market with extreme levels of uncertainty around tech and what will happen next, being reluctant to get pinned down on the wrong solutions is probably a good instinct. The biggest risk here is making the wrong bets. And you need to look at what it's going to cost you if you get it wrong and what it's going to cost you to fix it.
In this context yes, but for example at my job we are using LLMs to evaluate some outputs of cheaper/simpler models and to add context to them. This gets expensive very quickly with per-token pricing and would pin our entire business on the whims of one or two providers. With self-hosted open-weight LLMs we can just operate like we always have, it's a different kind of processing but otherwise we are free to do as we please wrt compute.
So it matters a lot that these open-weight models are good enough; we would likely not be rolling out this new processing if it was fully dependent on Anthropic and OpenAI.
I think, despite the desperate attempt at IBM to remain relevant (and in business), their CEO has a point here that do you really need SOTA models for everything?
I had a really nice time with GLM5.2 for a month or two - it was, in my experience, approx Opus 4.5ish level of capability when I think Opus 4.8 was the new hotness
Even today, I use Gemini 3.8 Flash as my daily driver for most coding. Its fast and good, even if it is not the absolute SOTA at coding benchmarks it really is very good. Yes I have tried Opus 5.5 and 4 Argon etc and yes the are better, but do I need that?
I think that we are going to be at the stage soon-enough where models are "good enough" for 99% of people's needs. The SOTA will keep on marching ever-onwards I am sure, but I would not be surprised if people stick with previous-gen/previous-previous-gen/etc models because why upgrade?
If this does play out like this, I do wonder how that works out for the frontier labs (specifically OpenAI and Anthropic, less so Meta & Google) - if open models or even just cheaper Sonnet-level models from a few generations ago are good enough for basically everyone's daily work, where is the revenue? how can you justify pumping billions and billions into new generation models and datacenters etc when there is no driving need to use them?
For work in which you are in the loop, absolutely. Do people not value their own time or something? Why in the world would you use less than the best LLM available to you? You're not going to run out of the $20 plan even if you use Opus 5.5 at xhigh and code 8 hours/day (assuming you're doing it piecewise using a Claude product) so what exactly are you conserving by using something inferior?
For large continuous volumes of programmatically submitted LLM tasks, sure.
I have single sentence prompts in one of my legacy codebases that it takes Opus 5.5 over $1000 in API spend to return an answer correctly on. I am not getting by with a $20 plan.
That's my point - do you need the best available to you? What if tomorrow your employer said no more 5.5, or Anthropic IPOs and suddenly investors want to see a profit and that 20USD a month for Opus 5.5 becomes 20USD per-day or even per-hour? We're already seeing corps restrict their employees token usage.
Could your survive on say Opus 4.8? Or GLM 5.3?
I bet you could if you had to.
I think it's the same argument about cars. Why wouldn't you want a Ferrari? Do you not value your time? Why not go as fast as physically possible?! You need the absolute fastest, most-capable, best-handling car for every single journey you take! Or... Back in reality not every journey you take is trying to lap the track as quickly as possible and just any reasonable car will get you where you need to go.
This is my point. Will Anthropic and OpenAI continue to exist selling Ferraris when actually people realise they can get by with a Toyota Corolla 99% of the time? Ferrari exists in the car world so I guess there is an argument for "yes". I am not sure myself - Ferraris are after all rare and exotic cars ... I suspect that even if we are not there now, we are already seeing current SOTA models that are mostly equivalent/better than the average engineer - another generation or two they'll be waaaaay better than the average engineer. How much more car do you need, and how much are you prepared to pay for it?
At least for me, I don't like using Opus or Fable because I don't like the way it talks to me (talking down to me, refusals, moral high-grounding, overly pomposity etc). 3.8 Flash is currently my favourite model due to how it communicates and works with me and the speed is much better suited for rapid iteration. So put me in the comfortable, reliable, use-it-every-day-to-get-shit-done-with-no-fuss Toyota Corolla camp. Its good enough for me and I don't get any benefits in my day to day life by using Opus 5.5 and it's sanctimonious personality.
Currently the difference in my work between the best models and okay models is pretty night and day to the point its not worth using the older models. Now I haven't used the newest top open source models. But if I give the same code base to say Qwen 3.6 and GPT 5.6, Qwen will analyze the source of the problems wrong, provide actually wrong solutions, write code that takes 100x as long to run. Its not even worth it outside of querying for snippets of code. GPT 5.6 will help actually root cause the problems.
Ah but if the price is 10x - as the IBM CEO suggests - your employer might just ask you to use that, and then expect you to make up the gap between the cheap and expensive models using your own brain instead of just pissing money up the wall on high token costs AND your salary. You know, like the old days when we actually had to do the work ourselves.
I think the point is that you are not everyone. Not everyone has your requirements, sounds like paying the premium is worth it for you, but others simply don't need it.
I love 3.8 Flash, if only it weren’t unusably flaky. Sometimes simple responses take minutes or I get rate limited after sending a single message (I guess it’s a shared pool rate limit). Have you not experienced this (would love to know if there’s a way to actually use it reliably!)
I've personally experienced three big jumps in LLM capability: o3, opus 4.6, and opus 5.5. I could go back to opus 4.6, but it did require way more involvement from my end.
Of course IBM - the guys that sell you mainframes - would be saying this.
This is a win though. But not for the reasons people might think it is.
It is a win because it's a massive player now countering the new (temporary) behemoths. With all the leverage, budget and connections.
And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people. Meaning that they're going to be far closer to the baseline of sane humans just from averaging alone.
The guy that sells mainframes also gives away the Granite models for free. I've found them to be quite capable in content retrieval and long-running workflows, even (or especially?) the small ones.
> And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people
IBM of today is a hallow corpse of what it was. The hardware business remains but they’ve mastered extracting rent from imprisoned customers long ago. No sane person should willing go into business with them, same as Oracle.
"Cheap" is up for debate, but the models themselves are great. Kimi K3 (and now 3.1 Pro) was the first open model I played with and didn't think to switch back to a frontier model. Now it's my daily driver and I have zero complaints.
The frontier labs are in a tough spot. It's hard to justify their cost when you can get something that's easily as good for cheaper (and anecdotally, more reliable/accessible). I do HITL so my perspective will be different than a fully-agentic setup, but as soon as I started using open stuff regularly, I realized that any moat the US labs had built is drying up.
That’s not a bad little model! Way more impressive than I expected. I am still messing about with it, but I think it’s a keeper for general discussions.
Though if you have the room for it, Muse Glimmer is better in practice on low reasoning strength, and while its token generation is slow, it actually gets to answers pretty quickly.
Probably mostly me and one or two others to be fair ;-)
"There are dozens of us!"
The thing you need to know about Muse is that you control its reasoning/thinking depth in the system prompt (low/medium/high/xhigh):
Reasoning strength: low
… is pretty responsive and I've found its time to first answer to be surprisingly good.
It is broadly comparable to Qwen 3.8 27B on "low" thinking, though it is weaker on very long context, one-shot stuff, but I dunno… I just like it more as a developer companion.
Considering the disclosed numbers for DeepInfer and the usefulness/popularity of DeepSeek 4.1 Flash, I expect it is inevitable that more companies are going to consider the self-hosting option seriously, and perhaps play with the idea of offering inference as a vendor.
Worth putting DeepSeek's own pricing rule next to that, because it sets the floor for self-hosting. In a closed-door investor meeting in May 2026, Liang Wenfeng said the API is priced so that "we go to the market and buy a batch of equipment, and recoup the cost in ten months" - that being, in his words, a reasonable profit. He adds that maximizing profit would mean charging more, but demand is inelastic in that band (he cut one model's price to a quarter and the team was delighted). I translated the excerpts, since the transcript only circulates in Chinese: https://telegra.ph/Liang-Wenfeng-on-DeepSeek-cost-restraint-...
If the vendor price is calibrated to a ten-month hardware payback, then at equal utilization there is no arbitrage in self-hosting: the crossover is roughly whether your sustained API spend exceeds the hardware plus that ten months. DeepSeek's own disclosed numbers give the benchmark to beat: 226.75 H800 nodes averaged over a day (~1,814 GPUs), 608B input tokens/day of which 56.3% hit the KV cache, 168B output tokens/day, ~73.7k tok/s input per node while prefilling and ~14.8k tok/s output while decoding.
That utilization is the hard part, and it is roughly what the IBM CEO is pointing at with "where are my kernel engineers". Self-hosting open weights on two or three racks is not the problem; amortizing them at a frontier lab's duty cycle is.
Everyone I know is either starting to or planning to switch to open weight models. Cost savings are just too good.
This is what will cook frontier AI and probably prevent the investment scale to happen for AGI. Humans + good enough cheap AI is just more economic and unbelievably productive. Like why we fly on jet technology from the 1960s still with incremental improvements.
It’s why Anthropic is desperate for capital and trying to get to a scale where AGI appears. But I’m afraid they won’t get there in time and the money to build the bigger model will have dried up.
What happens when your competitors invest in better and more expensive tools or materials in general? They (potentially) make better products, but those products also cost more. It's turtles all the way down.
I once overheard someone in a coffee shop saying that she needed GPU to run inference. But gaming laptops are too heavy with short battery life, while MacBooks are expensive.
I really wanted to tell her -- what she needed was compute, not hardware. Her best bet was to rent GPU which is easily available from many services. If she was paid to run inference, cost is not a concern. If she did it for her own work, she wouldn't need to shell out a ton of money initially and then be left with hardware that she no longer needs when the work is done. In any case, she wouldn't need to worry about purchasing hardware or battery life. ssh to a remote machine is simply way superior than running inference on a laptop she carries around by any means except in a few rare occasions (e.g. no Internet access).
Same thing for a company. Why would any company who has a business to run get themselves burned in acquiring GPUs, setting up racks, load balancing etc when Bedrock/Azure Foundry etc is right there with tons of both open and closed models and ZDR? And then see hardware depreciate in just a few years?
How good will be the business of dealing with 2nd hand GPUs for open weights model in the future? What will the AI labs do with their GPUs once they are beyond their EOL? Can somebody please advise?
He has a lot riding on this particular idea, and I think HN will generally be receptive.
I am not a believer in winner-take-all mechanics for models (even for AGI -- look no further than the natural world for many examples), and a large F50 company committing money/strategy to something outside the frontier labs is a great thing for the ecosystem.
If the focus is on models tailored for specific tasks, then I think that’s absolutely true. I’ve built a desktop app that uses Whisper and Demucs myself. Sure, it’s almost 2 GB and uses quite a bit of the device’s resources, but it’s free, runs fully offline and does what it’s supposed to do. I think that’s what really matters
Are there inference products focused on say, optimizing frequently used internal contexts, so that they are compact in the kv cache? Perhaps there rep could be optimized over hundreds+ of distinct real use cases?
Maybe your law firm doesn't want your data going outside your business. But you obviously want high quality analysis.
In the IBM-specific sense, the most unintentionally hilarious LLM interaction I have ever had was asking Granite’s 4.2 30B reasoning variant the car wash question.
Reams and reams of corporatese about why it wasn’t wrong when it was.
Any tips to get started with just one rig? A 4070 ti on a two year old Fedora workstation desktop?
What are the tools to look at first for usable models at that thin compute capacity and how do you go about exploring training for hobby and learning purposes?
Can we please stop giving weight to CEOs talking their book? Like of course IBM wants you to think this, it would be great for them if it’s true. And maybe it is true! But IBM’s CEO saying it’s true tells me just as much as if a frontier lab said it’s not: nothing.
It was a brilliant move at the time for IBM, and has helped them shift their business more into providing services and support.
I imagine they are panicking a little with LLMs reducing the reliance on external companies for software support, so they are really hoping to shift it to on-prem AI, which is where the bias in his statement comes from.
While he thinks most businesses can build useful AI infrastructure in just two or three racks, he feels many will struggle to get it running.
“The question becomes: how do you do that?” he said. “The question that the enterprise struggles with is: ‘Where are my kernel engineers? Where are the people who worry about the underlying network? Who can make sure that the model is clustered on GPUs that are close to each other in network proximity? Who can make sure that you’re doing the right caching of results so you get an incredible response?’”
Has he heard of “agentic AI”? You simply fire up Claude or ChatGPT and ask it how to do all of the above.
It's not a one time thing though - onprem inference requires continuous monitoring and tuning to get the most out of your GPUs. Looking at 25-50% performance benefits. Today, it does require dedicated Inference Engineers...
The optimal inference configuration varies according to the model, the specific traffic (which changes in time), and the specific hardware it's running to. The "knowledge" for it doesn't exist yet.
Usually in similar scenarios an agent will try to experiment with various configurations and get to a good result. However, you can't have an agent test out different inference serving configurations/architectures, since every experiment might involves 10-60 minutes of spinning up a model (on very expensive GPUs)...
Do not do that if you can't prompt it correctly. It is a potential disaster waiting to happen. LLMs are likely to pull things off the shelf without real defense in depth and that's a problem. Maybe slightly less if it's only in an internal network, but still.
It can be very dangerous to let people loose with such advice, FWIW. They don't necessarily know what "internal usage" actually means, or how important it is to keep it that way as long as you don't have a real expert to at least help direct the LLMs.
That's based on the idea that humans can learn and adapt and solve problems, and that's not what companies want to think about when they're seeking a profitable transaction such as hiring.
Qwen 3.8 Flash with Strata here, the problem with Strata is that you cannot easily share the server as the performance will collapse outside of a single user.
There was another comment in the IBM earnings call on enterprise AI spend [1]. Specifically, concerns around ROI for AI experiments in the enterprise and difficulty incorporating AI into enterprise workflows. This was surprising given the hype around AI, and it came across as a negative indicator.
I’ve been in tech for a long time… I have no idea what IBM does. Now, I get that I’m not the customer, that their customer is “business” but not my business… no… bigger business. The biggest of businesses I assume. I think it must either be a club I’m not in, or IBM is three half racks in a trenchcoat submitting invoices for millions to companies where people just pay it because “it’s IBM, it must be important”.
Review the specifications for the latest Telum II chip if you need a hint. 5.5ghz base clock.
If you absolutely need to scale something vertically, IBM Z is the final stop on the line.
X86 is great at hosting an entire farm of applications. You would never want to install salesforce or gitlab on a mainframe. That would completely defeat its purpose. If you've never encountered an appropriate workload, IBM might never make intuitive sense.
Ironically one of the smaller places I've worked at tried to use an IBM product once - I'm still not clear on what it was meant to do or how it was better than any cheaper alternative, so I assume someone in leadership was fooled by a very impressive pitch.
Lots of work went into integration and it never went live.
Transaction can be a nebulous term; think of credit card transactions, ach and wire transfers, mutual fund buy/sells, bank deposits and withdrawals, etc.
OpenAI most likely pushed Cerebras into a contractual agreement not to do business with Anthropic through 2030. ("Most likely" because the names of the companies Cerebras is contractually obligated not to sell to are redacted in public filings)
It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.
Gaming used to be done in arcades for 25c a play*.
Over time NeoGeo realized they could sell you the hardware to play at home (“on-prem”) at arcade quality albeit at a price out of reach to most people. Alongside them, Nintendo and Atari gave you 80% of the fun at 20% of the cost. History will view games arcades as a flash in the pan compared to gaming at home.
Namco, Capcom, and Konami went into selling software instead of hardware but it feels hard to imagine buying a home copy of Fable** for $X,000 any time soon.
⸻
* That’s for video games. In fairground arcades you’d literally pay for tokens then use tokens for play. The analogy almost writes itself.
** The model, not the video game.
If you think the idea of buying Fable like model is hard, you should really try some of the latest open models like Qwen 3.8. They’re getting there very fast.
Qwen 3.8 and GLM 5.3 are my workhorses right now. Qwen for simpler stuff because it is wicked fast using 'ninfer', GLM 5.3 because it is very good a more complex stuff and less likely to get stuck so I can let it run for much longer without intervention.
I have Qwen 3.8 running under ninfer or Strata sized to fill my 5090.
From a fair bit of coding with the agent, my experience with Qwen 3.8 (Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-0 on Strata, Qwen 3.8 27B on ninfer). In my particular case, I was trying to clone a recent HN post which allowed you to visualize how out of tune your singing was relative to a song, but as a web app or portable native app.
My experience is that both those models will generate an enormous amount of noise (literally: it was running audible unit tests on my machine to test the note-detection algorithms) while coming to a solution, if they do find a solution- while Claude (whatever the latest model is) or ChatGPT (ditto) will do 1/10th the amount of work to get the right solution. I suspect if I had the resources to run GLM 5.3, it would probably be significantly faster to get to a working solution than Qwen-3.8 27B, but likely still somewhat inferior to Claude or ChatGPT solutions, and also would generate a much larger energy bill on my part.
How do you run GLM 5.3?
Their parent comment claims they have 8 GPUs and 2 DGX sparks.
Even 2 DGX sparks are limited to Q2 at best. As for GPUs, it of course depends, but anything under 50k just can't run it, not even Q2.
And at Q2, I think you're better of with Qwen3-coder-next on one DGX spark (or much better: on any apple silicon max, we are currently in the ridiculous situation that a second hand m1 max 64G will beat a DGX spark that's at least triple the price)
Halogen flash server for strix halo runs qwen3.8 flash next in 40gb
This being /u/jacquesm would imply it is kinder to assume him having that setup than claiming to have that setup.
[dead]
I like this analogy a lot.
It reminds me that coin-op SF2 champion edition was like $6000 new, and you rented time with tokens. Eventually you could play a very good but more limited SNES version of SF2 at home for no tokens. And then PC hardware matured to the point where you can play the exact bits via emulator at better than original fidelity.
It will be very nice when AI gets to the emulator stage.
> It will be very nice when AI gets to the emulator stage.
Do you think we'll be longing for the nostalgia hit of LLM models from 2025 in 2050?
Maybe we will collectively miss being completely right and inappropriate em—dashes.
I'm already nostalgic for very early Dall-e LSD-like art. Ref: "Sleeping Horses - Nov 2021."
Those early GAN image generators were so weird that they produced art that felt genuinely creative. I don't know that whatever people are using to create modern Tung-Tung-Tung-Sahur instagram reel slop is even capable of producing similar things.
I'd say Nintendo gave you 10x the fun of neo geo. Yeah it looked cool, but mario and Zelda on the nes played like nothing else. And then we got street fighter 2 on snes which is better than any snk fighter I've played :)
(I'll run for cover now as I'm sure neogeo fans will disagree!)
I'll agree, since arcades could never give the experience a JRPG could. No shop would allow a player to occupy a machine all day unless they spent a lot of quarters.
2D action games peaked with Metal Slug.
/SOtN entered the chat.
(I also adore MS)
I'm stealing this analogy. This is good.
But also look at live service games and the shit modern game studios are doing to try to go back to the old world of rent seeking.
Daily reminder that this isn’t what rent seeking means.
> Rent-seeking is the act of growing one's existing wealth by manipulating public policy or economic conditions without creating new wealth.
Game industry has first manipulated the economic conditions by restricting access to game content (skins, modding capabilities, always online drm, etc.) that previously was included in the price (buy game, you own that game outright).
Manipulation of public policy was a more recent thing with the laws surrounding underage gambling via 'loot-boxes'.
So I think the onus is on you to show us why these behaviors, which increase the wealth the companies are able to extract per game, are not rent seeking behavior.
Changing how you sell your own products is not "manipulating economic conditions" whatsoever. If it was, that would mean that every business in the history of commerce has engaged in it.
As for the laws on underage gambling, countries all over the world are outlawing that, suing gaming companies, etc. The public is not in favor of underage gambling, and it's unclear to me how banning it increases the wealth of these companies? So I think the onus is on you to explain how this is an example of rent seeking.
Modern gaming companies can be considered rent seekers not because they charge money for their products, but because many increasingly rely on extracting additional revenue from consumers by exploiting market power, consumer lock-in, and artificial scarcity rather than creating proportionately greater value. In economics, rent seeking is the pursuit of economic gains through the exploitation of existing advantages rather than through productive activity. Developing a great game is productive; leveraging an established franchise or player community to extract additional money without providing equivalent value is rent seeking.
Consider the use of microtransactions, battle passes, paid progression, and downloadable content. These systems often exploit the time, money, and emotional investment players have already made in a game. By deliberately slowing progression, withholding content, or introducing artificial scarcity, companies can encourage consumers to spend more money not because they are receiving a better product, but because they want to avoid frustration or access something they already value. The company is monetizing the player's existing commitment rather than necessarily creating new value.
This does not mean every gaming company or microtransaction is inherently rent seeking. Selling meaningful new content is a legitimate business activity. The criticism is that parts of the modern gaming industry increasingly prioritize extracting revenue from an existing player base over developing better games. When companies use their control over popular franchises and digital ecosystems to extract additional money from consumers who have few alternatives, they exhibit the very characteristics of rent seeking that economists criticize in other industries.
In economics, rent seeking is the pursuit of economic gains through the exploitation of existing advantages rather than through productive activity
No, it’s not, you just made that up. Why doesn’t this match your previous definition?
The rest of your long AI-written comment has nothing to do with the principle of economic rent seeking.
Not sure why I’ve been wasting my time, but I’m out. Peace!
You've stumbled on exactly why the big providers are lobbying for regulation and big claims that Ai is unsafe. They don't want you running these at home.
Going to the arcade was about a lot more than just the games, though.
>Gaming used to be done in arcades for 25c a play.
in the early days, 2 plays for a quarter was not unusual (but you couldn't buy one)
Arcades mostly failed to adapt to the advances of home consoles to provide experiences you couldn't get at home. That's on the dimwitted game suits of the 1990s who cared little about novel gameplay, networked gameplay, or better NPC AI and physics. Then mobile happened and there was a brief glorious Cambrian explosion of new ideas until the Zynga playbook and DLC. Evidence: lived through it as a small game studio developer trying to do original things. The fun experiment now and then is to break out your 1980s video game consoles and play through their libraries. Most of the games are objectively terrible but gems like Yar's Revenge are still gems today IMO.
It will take a while till prices drop to a level where affording 256GB card (even if it would be shared between few coworkers) will have any sense over buying tokens.
And the AI debt bubble price spike will probably stay there for a while.
Intel Crescent Lake is dropping. 1.4TB/s of LPDDR5X, up to 480GB. It's not going to be cheap but it's much better than trying to buy gddr based cards.
Still (as bit over) 1/2 the ram bus width of a single HBM4 stack though! (1280b vs 2048b).
That's going to be pretty pricey though. And I really want to see some independent benchmarks, that all looks very good on paper but there are some complications (notably: chip count, run length of the traces).
Well, if thw ai bubble implodes, youll sudden be exposed to real costs. Thats when a real opinion will be formed.
Uber took a decade but eventually, the real cost of a taxi service went up for many use cases.
Your argument is only solid if we are headed to the singularity.
> Your argument is only solid if we are headed to the singularity.
Perfect encapsulation of the problem. I may steal the wording you used for future conversations.
How much did you shell out for this setup? What’s the TTFT and the output tokens/sec rate? How much electricity does it consume?
Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max.
My local model never says no. I never get banned. If I want to do security stuff, reverse engine, there it goes! Do I wanna chat about politics? No problem! Do I want it to analyze medical stuff? I don't get to hear that it's not a doctor and I should call 911. Do I want to explore legal docs, no problem! There are lots of things that cloud models will say no to! I read that don't even agree to help with advanced biology and chemistry topic much. I also never run out of tokens, no 5 hr limit. I can run it 24/7. I don't run out of tokens. I can run 16 sub agents at the same time, non stop! 1-4 billion tokens a month? No problem. Any agent harness I want! No restrictions. Privacy all day. My own true system prompt, not something someone has determined, no filters. Furthermore, there are many open weight cloud models that don't have equivalent online. For a while before cloud models got good at math, the best model at math was DeepSeekv2 Math and there was no API. The only way to use it was to run local. All sorts of audio models, image/video models, etc. No multiple subscription needed. Private UI for my family, kids included (I can guard what kids infer). Got a friend that really needs some tokens? I give em access through private VPN. Telling someone that they are paying for decades of Claude for local AI is like telling someone they are paying for decades of public transport when they buy a car. Sure, you are right, but ...
Also no political censorship, and it doesn’t report you to the police when it suspects you did something illegal. That really happened, by the way.
Now I want a local LLM server :( $25k is just so much money.
If you already have a machine, throwing an r9700 in it is one of the cheapest ways to get qwen3.8 27B Q6 with 100k+ context working.
Sadly, the r9700 has gone up in price, but it's still the cheapest route. I use It on an old x99 Linux desktop and it achieves 50 tps decode and 1000+ tps prefill.
Thanks for the tip! I've already got a home server set up. Maybe this is my next AI purchase. Still, US$2,600 here in Europe is not cheap :-/
I was hesitant, so I went with a hassle-free return route when I bought mine.
My stack uses a local llamacpp ROCM build, modified by https://github.com/stew675/llama-cpp-rdna-boosts
The build uses my machine's ROCM and I just link it instead of embedded all that in an image, like the published ones do.
Then deploying it via docker compose to my local machine. The models I want to run are stored in a docker volume, so it's easy to switch the model while keeping the same build.
The biggest negative for me is that I don't think it's enough to do parallelization through llamacpp (I'd prefer to work on one thing faster than two things slowly). I haven't gotten vllm working yet. Also, once you have one, you constantly start thinking about a second card...
That's true. Now let's reverse things a bit, we go back five years in time and I give you 'magic in a box' for $2600. Would you buy it?
No political censorhip? Try asking Chinese models about Taiwan.
Some censorhip and bias towards certain ideology is baked into those models.
While it is not a problem for programming tasks, I would never use it for certain other tasks.
> No political censorhip? Try asking Chinese models about Taiwan.
Self-hosting means being able to use abliterated models where the superficial instructions like not describing Taiwan as independent are burned off.
Censorship and bias are related but not synonymous. I agree that it's harder to remove political bias when the training data includes it. We know many of the frontier models are heavily trained on Reddit, for example, [which likely explains why we see such heavy left wing bias.](https://trakkr.ai/bias/findings)
More citations:
https://www.gsb.stanford.edu/faculty-research/working-papers... (article: https://news.stanford.edu/stories/2025/05/ai-models-llms-cha...)
https://www.tandfonline.com/doi/full/10.1080/29974100.2026.2...
https://ieeexplore.ieee.org/document/10817610
https://www.washingtonpost.com/technology/interactive/2026/0...
https://link.springer.com/article/10.1007/s11127-023-01097-2
https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...
It could, but it's an inexact science. A little bit like tampering with the brain. Even a surgical procedure removing a tiny tumor can interfere with critical pathways. Usually abliterated models end up almost exactly as capable in benchmarks, but it requires the person doing the abliteration to anticipate or know about the censorship. In the case of Chinese model censorship it is believed that it is post-training, so the underlying data input is not the issue. It's how the model has been trained to use that data. Specifically, abliteration usually targets the refusal behaviour. I would guess that this example could be significantly improved, but I have not seen any abliteration attempts re this specifically.
All good points, but I am not sure who I trust less. Chinese companies or random people on Hugging Face. :) It is still a black box. And I have a feeling I could not abliterate those models on my own hardware.
I am not saying I would not use any of them at all, but I would avoid doing certain tasks with them. And with the testing I have done with abliterated models in the past, it is not 100% successful and it can still hit the same walls as original model.
I agree. I think the future for privacy conscious people is going to be an adversarial open weights model, where competing models (ideally from opposing superpowers and labs) critique each other's work to find consensus. Or at least alert for suspicious behaviour.
So... the Super Intelligence is Democrat?
After years of trump any intelligence is a democrat.
The only winning move...
Is to deprive yourself of the slightest chance of winning?
> No political censorhip? Try asking Chinese models about Taiwan.
In open models you can "uncensor" them: https://arxiv.org/abs/2406.11717
I'm building a local LLM server myself. For about 8k€ you can have 170tps, ~6000 prefill.
A lot more with Haiku equivalents (Qwen3.8 27B) or gemini. It's not frontier, but not that far away.
I would love to hear more about your setup. Which hardware and models? How much context can it handle? How does it compare to frontier models?
I mostly run Qwen3.8 Flash Next on 2xR9700 (but Intel Arc Pro B70 may be better currently, or 4x3090 is even better but out of budget, 32GiB Nvidia GCs on ebay are also usable). I have 128GiB DDR4 (3200Hz), but 64GiB is enough, even 32GiB maybe, I just reuse it from an old project.
You may need a AMD EPYC processor and associated motherboard, don't forget a good PSU at least 1200W (300W processor + 400W by card to support spikes of consumption). On some setup (~15k€) you can use 4 CG to go up to 128G of VRAM.
I'm not using a EPYC processor for now, so I get 170tps on decode but very slow prefill, the bottleneck is the PCIe exchange between the cards, I need to setup P2P (it means what you think it means, for GCs).
My setup supports full context (2*18 bytes so 256K), I didn't setup YaRN to get 1M of context (2*20 bytes), but it's an option with 128GiB of RAM, the visual component is extracted from RAM so a little slower. I'm not using multiple agents but some people does, it's god like with 4 graphics cards, on 2 only, it's relatively quick but limited to single user and up to 3 to 4 concurrents requests only.
It's not frontier, but look AA of QFN, it's good enough. DeepSeek V4.1 can run on my setup, but it's slow. On dual B70, there is some PoCs to make it usable (>60tps). It improves every days.
If you need more information, look for local setup users on the Web. There is a lot of people sharing information about benchmarks and setups. L80 is my reference for now.
Edit: I translated the unity.
Why do you keep writing Gio for gigabytes? I guess maybe GiB for gibibyte, but Gio is new to me
I'm sorry. Yeah Gio is only in french, I forgot to translate the unity.
octet => byte, https://fr.wiktionary.org/wiki/Gio
You're Blue Book compliant ;)
oh neat, I didn't know that. makes sense, thanks
MacBooks with large RAM and optimized for local models should be Apple's medium term goal.
It's a good first step. They're not great but they kind of work for many tasks. I think the small models can be useful in context-specific tasks but the friction is high right now because the user is forced to determine whether Siri is good enough, or a better LLM is required per task. Ideally Siri would be smart enough to intelligently route. Apple claims that's how it works but even their cloud processing is far behind frontier models.
Worryingly, they're taking a walled garden approach, and they don't want to expose user data to competitors. After the monumental failure of Siri over more than a decade, I don't trust Apple to execute competently on AI.
>and it doesn’t report you to the police when it suspects you did something illegal
This is an alignment issue. OpenClaw with a local LLM with an email you setup for it could contact the police.
It will not do that unless you specifically teach it to do that, or you use an Anthropic model. The issue in this case was Anthropic's safety monitoring built into the model. They record everything and flag potential illegal activity for human review. This occurs even over the API. I do not know if OpenAI does this, but I would not be surprised. I think the Chinese models are far less likely to do this (unless one resides in China).
> It will not do that unless you specifically teach it to do that…
Sorry, you're claiming local LLMs are somehow deterministic and never show unexpected behavior?
I’m suggesting it’s highly unlikely it will call the FBI to dob you in. That affirmatively trained behaviour, not something one should expect to accidentally happen.
Why would you assume an AI model's training set doesn't have things like "you should report crimes to law enforcement" in it?
[flagged]
renting GPU off cloud is also an option
Exactly, that's the spirit. I pool the resources with some friends as well.
I think both of y’all glossed over the $25k of it all pretty quickly. This isn’t remotely an option for anyone.
You see it as money spent, others might see it as an investment. OP doesn't necessarily say exactly what they use it for, but if education is the goal, $25k and spending your evenings learning about AI theory and tooling is way cheaper than a university degree and can easily result in millions more dollars in lifetime earnings given the current early state of the technology.
Besides, I regularly see people here in HN spend a LOT more than that on hobbies and depreciating assets. A few years back, people with coastal salaries and tech jobs were tripping over themselves to spend $100k on a certain brand of trendy electric car.
I can’t “invest” $25k into an LLM. The vast majority of people can’t. And I’m not spending $5k to timeshare it with 4 other people or whatever lol I currently make do with a 9070 + 32gb of ram.
AnyONE. But if you got some friends, and they got friends, then that $25k becomes smaller and stomachable. My favorite rumor out of LA is groups of friends with a Lamborghini Urus that they timeshare. No idea if it's true, but if car people can get together to buy/lease a car that's MSRP is $345k, I think $25k should be an option for anyone with friends. (Hypothetically. I've read a lot about this friend thing on the Internet!)
If 10 people are sharing it the speed advantages suddenly drop drastically. That’s $2500 a head. 5 would be $5k. Then all the rules and such you need in place
This is all this stuff that dries people to pay for frontier models. Even running modest models requires a few thousand, unless you just want really lean ones. People don’t need frontier but it’s not cheap to effectively run, say, a 16gb Qwen 3.6 or 3.8 in the 30b range. Ram is like $15/gb and GPU’s with 16gb vram are hardly cheap.
In 1993-94, I upgraded the RAM on my home 486, which started with 4MB. I doubled that to 8MB for $250, then to 16MB for another $250, and finally to 32MB (the max for the motherboard) for another $250. At the time, I was a poor undergraduate and had to give up other pleasures for these expenses. If I estimate correctly it's about $30K/gigabyte (only high end servers at the time had a gigabyte, and they would often cost $100K or more).
The lesson it taught me is that swapping is slow and that if it happens during your software development, you're much less productive. When I finally maxxed out the RAM on my machine, I could run linux + emacs + X11 + bash + g++ all at the same time, making me about 10X more productive. It's an investment and lesson that has helped me build my career over decades.
In the 90s we'd pay $5k a head (inflation adjusted - about $2k then) for a 486 with 80MB of disk space
Yes 30-40 years ago when home computers were still a pretty new technology you paid a lot of money just to have them.
And all sorts of companies, from small businesses to multinationals, and a fair few people at home, paid it.
Businesses were quite happy to spend $5k on technology for a worker if it makes things better, why wouldn't they today.
Unless I read it wrong, it sounds like this guy did not get it paid for by his employer - he runs multiple companies and is an angel investor with clearly deep pockets - and I have people talking about it being an investment in their careers. He could be using it support his businesses, yes, but that needs to be clarified and is still not a situation most people find themselves in.
This is not a straightforward business expense conversation IMO. Otherwise yes I do agree with you. A company should invest in supporting local models for their team.
But I don’t need my LLMs part time? And we all live in the same time zone.
And what happens when the girl at the bar figures out that you TIMESHARE that Lambo?
You timeshare the girl?
Ok taking the :door:...
Given all of the discourse about what’s happened at Cornell, you really thought this was a worthwhile comment to make?
This sort of casual misogyny is at the root of the problem. Shame on you.
Its a drop in the bucket for many businesses though.
Anyone? I know many people in Silicon Valley, Boston, Seattle, New York, and elsewhere who would consider that a reasonable expense if it kept them viable in the job market for a few more years.
My entire career has been based on purchasing the most hardware I could afford at home and using it near its peak. I've given up other life pleasures (dinners out, nicer car/housing) several times to max out my computational capacity.
Also hardware like this, if chosen carefully, often has a viable lifetime over 5+ years
I’m not spending $25k to “stay in the job market.” I come from film production, I’m no stranger to owning my own hardware. But that’s ridiculous. And at least people rented my gear.
> My local model never says no
In case a future user wants to know what a 2026 model would refuse to do, I needed to rotate dozens of passwords last night both on the server and on the clients. When it was time to verify that the rotated credentials worked, I asked for a script that would rate limit the credential checks. Claude/opus5 (opus!!) refused and stopped in its tracks.
In another instance, Claude was making great headway bringing a legacy Windows app up in WINE until it’s saw “auth” somewhere and shut down. In the shutdown state, Claude asked me to reword the prompt to see if I could get it to act on less dangerous instructions. But when I reworded my request, narrowed the scope to be clearly unthreatening, or even redirected the model, it’s still rejected me. I suspect Anthropic uses the less suspicious rephrasing as training material to increase their rejection rate.
> My local model never says no. I never get banned.
So much this. Plus everything else you mentioned. The freedom and control is real.
I agree the local models have many nice properties, but I would not be surprised if people start to recognize that Gemini pointing you at medical services instead of attempting to diagnose you is actually a good thing in many cases (and Gemini often does do some diagnosis anyway).
> My local model never says no.
> I don't get to hear that it's not a doctor and I should call 911.
https://en.wikipedia.org/wiki/Darwin_Awards
It's pretty easy for a reasonable person to determine when to call 911 and when to ask an LLM. Typically when I ask an LLM, it is for something I am not willing to escalate further at the moment.
Anthropic has a significant interest in reducing liability, which in this case means mostly blocking this use case.
> It's pretty easy for a reasonable person to determine when to call 911…
Unfortunately, it's fairly tough for someone to determine if they are themselves a reasonable person.
A model that only ever says yes is a cognitohazard, IMO. Like being a billionaire without the money to insulate you from the consequences.
Saying yes is not equal to "provides the requested information." If you use models for affirmation instead of information, this is a more reasonable concern.
"I don't get to hear that it's not a doctor and I should call 911."
Sometimes, the requested information should be accompanied with a note like that.
Have you heard the cost of health insurance and seeing a doctor? Some people don't see going to hospital as an option. Better for them to have a local model than no options. But let me tell you how I use it for health, I take all my health records, lab results, every thing, I throw it in there for a second opinion, it explains things to me, it suggests questions I should ask on follow up, possible options, etc.
> Have you heard the cost of health insurance and seeing a doctor?
Yes. I've also heard the cost (death) of people falling into pseudosciency rabbit holes.
The security research is probably the biggest reason to use local models as it seems like the big service providers will ban you for life if you even hint at doing that.
>I don't get to hear that it's not a doctor and I should call 911.
Local models can do this and with tool use they can even call emergency services for you on your behalf.
A cloud model can also get neutered anytime and/or start using more tokens while charging you the same or more per token.
isn't it the same using the same model you host locally but through a cloud provider? does the $ math favour a local setup then?
Cloud models sometimes get degraded. You can trust your local model and know what you are running.
seems like a stretch, especially when you could choose from 10 of them if u feel they're dropping off.
Also, it's not like you're running any full models on local without shelling out 20k+
If I want to go out of business that is what I would do. Most of the data I have here could never be shipped to a service provider, I'd be in lawsuits for the next decade or so and die bankrupt.
That NDA case is the real argument, more than price. Token prices keep dropping anyway, but "this data can never leave the building" doesn't change with a price cut ;)
Okay man I've got to ask, what kind of data are we talking about? Legal? Medical?
I've heard that there are legal software platforms like Clio that offer AI products, not sure what just eavesdropping on some lawyers at the bar.
http://www.modularcompany.com/
Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients.
It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money.
On a tangent: this is a really good website. I understood what you do and how you do it by reading down one page. There's a lot of projects/companies I immediately tune out from because I can't actually figure out what they do amongst all the buzzwords, animations, and excess scrolling in every dimension.
Holy fuck you have a lot of karma
He's been here for a loooonnnggg time. I'm pretty sure I found HN through one of his blog posts.
You can estimate the amount of time by the domains he's been squatting all these years.
Isn't he the guy who posted about his LEGO-sorting computer vision project? That post alone taught me things that increased my career opportunities significantly.
The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.
They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.
I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.
> The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.
But, there is no need to make that bet - when prices increase, I'd look into renting a previous-gen GPU on a cloud provider. If the online GPU providers raise prices, then I would look into purchasing for on-site.
Until that happens, I'm not making any bets, because if all the token providers go bust GPU rentals are going to fall in price, and if they don't go bust after raising prices, I can still get years of GPUs for self-hosting at the price of a self-owned rig.
Well there are risks to being part of a crowd that's all compelled to make the same decision at the same time. Certain risks can only be transferred, not reduced in absolute terms.
It's not economical today to run my own hardware, but the insurance is worth it to me in the sense that if my bet is wrong, I'm still okay with paying extra for the protection.
LLM subscriptions are heavily subsidized. I’d be surprised if Claude 20x costed the same after a decade.
Inference prices are going only down. It could be that some top subscription costs more in a decade, but I fully believe a full array of competent models will be extremely cheap.
The inference is profitable. It’s the training which is really expensive and not being reflected in the subscription prices.
Doesn't deepseek make $38 per GPU-hour? As in they can actually serve profitably?
Deepseek doesn't have subscriptions, you can only pay per token / API prices.
Yes but the API prices are very low. So much that it is hard to spend over 20 dollars per month as an individual no matter how much you use it.
How are their API prices compared to others?
Keep in mind that the prices of the big AI companies are currently mostly loss makers. Their plan is to get sufficient market saturation and dependency of their clients and if they achieve that, they will hike up the prices quite a bit.
(Of course this all depends on if their plans work out, but if not then the smaller local models will probably play a big role in the failure of their plan)
People keep repeating this, but it is only remotely close to true for the literal free plans and the small subset of consumers maxing out the pro plans.
At API pricing (i.e. what all enterprise customers are paying), inference is profitable, even if they're paying usurious rates on cost of capital for capex. Where the frontier labs are getting screwed is that they're in a Red Queen's race with each other, spending tens of billions of dollars per year training the next frontier model, and not being able to recoup the investment on that model before it's obsolete.
That's why you see the CEOs calling for government intervention to "pace the frontier": it lets them reduce investment on training without risking that their competitors keep going and make them obsolete. Unfortunately for them, the Chinese labs exist, and would love nothing more than to take the frontier, so they're in a bit of a pickle.
The people I personally know who signed the "pacing the frontier" petition (including me), or who called for it in other ways, are sincere in their concerns that things are moving too fast for safety.
Put the liability on the AI companies, and they'll slow down.
However, that also means AI will be even more lobotomized since the cloud providers will panic over being liable
> inference is profitable
People keep repeating this, but it is only remotely close to true if you evaluate inference in a vacuum without any of the other supporting costs to facilitate that capability.
Just a few years ago people were saying the same thing about WeWork: "leasing is profitable, the unit economics of individual locations is net positive", and yet...
But what if you become unable to pay for it ? What if:
- they raise the prices 10x
- go bankrupt
- ban you or your country
- decide not to help you with what you want to do for opaque reasons
- you are in jail because the model halucinated something and the company promptly reported you to the police
Better avoid all off these and go local IMHO.
>you are in jail because the model halucinated something and the company promptly reported you to the police
Just note that your local model can 100% do this itself.
If you compare to API prices the value difference is not as big.
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It's hard to go with anyone else unless you're rich.
If you compare to API prices the value difference is not as big.
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It'd stick with it too
I was thinking the economics of it the same way.. if the AI companies are burning debt and VC money to give me cheaper tokens why not take them for it? I can always rent a GPU server for those personal, private AI needs.. heck even run a decision model to route them to the rented GPU servers either on demand or scheduled for few hours in a day…
It doesn’t have to be black or white…
For my toy projects, adventures, learning exercises, and throwaway MVPs I will prefer the subscription plans… they are cheap for what they offer today.
From this article alone it should be obvious that the long-repeated myth about frontier inference being subsidized is just that: a myth.
It's training and tuning new models that cost them money. And that's why they are trying to scare people into thinking AI will kill us all in the next 5 years; they want bans on the open-weight models because they know they won't be able to compete with them.
A few days ago we got the news that DeepSeek makes 38 dollars profit per card with their off peak pricing (in the inference server release discussion).
If that is true, either the American labs are not optimizing correctly or they make a ton of money with inference.
There are lots of reasons Chinese AI companies will always be cheaper than Western AI companies, but chief among them is that China is pouring ludicrous amounts of money into new energy generation projects.
Hydro, wind, solar, nuclear, coal, they are building it all, as we speak, at least 10x more than the US. They are aiming to not just be energy independent, but energy dominant.
You can't now (and certainly won't in the future) be able to run an AI provider in the West as profitably as you can in China.
DeepSeek designs models that require less hardware resources for inference.
Yeah, the way I see it short term is take the free/subsidized tokens while we can and the further out we go the cheaper compute will get for local setups.
Do you take into account the token allowance? For a heavy user this would be the equivalent of many subscriptions and they add up.
This trope minimizes so many downsides of "frontier" models. When you can't do what you need to with a model it really isn't valuable - which is why subscriptions still exist. As soon as you're forced to pay actual token costs people who keep saying this will likely wish they had invested in something else earlier.
It's also very clear that providers are running variants of models, especially at the SotA levels. I wouldn't be surprised to find out at some point that 1-2 months after release OAI and Anthropic just run "good enough" quants to save themselves on inference (since they're losing money hand over fist). Another upside of local models is that I don't need to worry about model degradation at any given point.
If you know you will pay Claude a few thousand bucks a year a decade adds up too, owning the hardware could just as well end up cheaper because you can sell it or as others have alluded the price to rent could get much worse.
Would you really expect to be using the same hardware over the course of that decade? One of the biggest drawbacks to using on-prem hardware is keeping up with the upgrades. Leasing upgrade cycles are a thing for businesses, but for someone at home it seems like later in the decade the local system will be showing its age.
The 3090's I have would sell for more than new today... for the next couple of years or so it will be pretty bad on the hardware front. Until there is more RAM manufacturing capacity and NV has a credible competitor I don't think it will get better.
Heh, the hardware I bought I could easily sell for 2x so you may actually end up making money. Though I hope that that won't last.
The benefit of the agent working firstly for you rather than for Anthropic or OpenAI will become increasingly apparent. We're already seeing signs: remember when Anthropic openly promised to have Claude sabotage people using it to create competitors? And OpenAI is looking for revenue opportunities from advertising, which can't end well for their users.
Just think of how user-hostile other online services have gotten, and imagine how acutely painful the proprietary LLM providers could make theirs. Thankfully, open models provide some counterbalance.
> It is not cheaper than using a frontier lab
Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever.
True, they're subsidized, I'm not. But I don't have their utilization so I figure it would always be more expensive to run this locally.
Would you use hosted models if they promised anonymity and no log retention and servers were physically located inside jurisdictions that respect privacy (Switzerland, Norway or similar, I'm not into law) if that meant better price/performance ratio than your current rig?
Thank you so much for sharing your insight - it's not much serious discussion out there on sane reasons for self hosting (that I've seen).
You say the big rig is 25k - and then the two sparks are about 1k each?
Electricity is for "just" the big rig?
Did you get insurance on all this stuff? I imagine home insurance would not qualify for a bespoke home-datacenter.
I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
V100S is a very good card for some workloads. And they were cheap as well.
I have 240G VRAM total which is a good reason why that rig cost as much as it does, and if I were to build it at today's prices it would be even more expensive.
I got mine just a few weeks ago for about $750 each. I am a decent programmer and have Opus optimizing the inference engine. I know enough to keep it on track, but GPU programming is something I have literally never done, before, but now I know enough about Volta and how to optimize it to be dangerous.
Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.
And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)
You got lucky :)
I tried getting some 170's when I first heard about that unlock but they were already unobtanium by then. That would have been a nice way to add a lot of VRAM.
GLM 5.3 runs pretty comfortably on this rig, even with large context.
[dead]
Not OP. But I’ve done some calculation here https://localaicost.com/
I envy you. If I only had the money for that sweet hardware. And the electricity bill.
You mean, the Chinese models, right? Like the ones that stole all the weights from ChatGPT and Claude, just like those companies stole all the IP from the rest of the world, right?
I’ve always wondered about those models. Are we just sending all of our data over to China, instead of ChatGPT or Claude? Can we trust them any more than we do ChatGPT or Claude?
They didn’t steal them, they copied them, similar to how Anthropic and OpenAI copied the materials they used to train their models to begin with.
No need to send data away. Local open models keep everything on your device.
At work two different departments have it on their radar to do - apparently there are GPUs waiting to be used for something like it.
Smart move to get them bought ahead of time.
Might have been bought 12 months ago? Not sure when exactly but the impression I got was that they weren't too old.
Would it make more sense to just rent the infra from a Cloud provider just for amortization purposes? While...and this depending on the the Cloud provider...getting the same guarantees of governance and personal sovereignty?
You may get the same guarantees on paper, but at this point there's absolutely no reason to trust any AI provider.
At best they get a fine, and it will not make a dent in their cash reserves.
Bedrock guarantees neither your inputs are shared with the LLM providers, neither your data will be used to train the next model. I dont think AWS loves AI that much...to break their business model with a violation of terms and conditions.
But I was not even suggesting that, just run your local LLM on Cloud infra. Do you really believe they are ready to risk their Cloud business just to spy on your prompts?
Antrophic works with an ex-deputy of the CIA (David S. Cohen). OpenAI has an ex-director of the NSA (Paul Nakasone) in the board. Literally.
The Bedrock models are hosted at AWS.
I'm sure these gentlemen will figure it out.
8 gpus is cheap now?
Well... that's a tricky one. I bought a lot of these second hand. I can sell them for more than they originally cost new, so indeed, it is not 'cheap'.
Then again, I run a business and for the business it is absolutely manageable, it is roughly the price of a single job. Privately this would be a different matter but I assume that if you would do this privately you'd simply get a single (but powerful) GPU second hand and offload to RAM. Not as fast but with llama.cpp, 'ninfer', DwarfStar and various other clones of those two you can get some pretty decent performance out of single GPUs.
I've been buying powerful hardware since the 90's, starting with SGI and then ever larger Linux boxes. I make my living with them so it's never been a huge decision to invest some of the proceeds back into the tools. But on an absolute scale, yes, it is a lot of money.
Whether this idea is economically largely depends on how much utilization you can get and how much you can get GPUs for.
What he does not say, but imply is that IBM's open models are also good. And they are.
It really depends on usecase i guess. I've tried a few open models, they don't hold a candle Argon/Opus/Sonnet when it comes to my current major use case which revolves around CAD and aero simulation development work.
What's your use case where the open models are working well?
Yup
I’m running local models more and more and starting to move to just tailscaling into everything
Depends on context of what people build.
I mean IBM already has a track record of building gigantic amounts of garbage and over charging large corps and govts for it coasting on their brand name and networks to get deals. So using some cheap ass model fits their MO. Having used their shit mountain called websphere, what an IBM CEO says is not something I take too seriously.
IBM has two faces. Their consultancy arm is the exact opposite of what I would call a good company, even so, they make a good bit of money. The hardware side of the business is still quite impressive, yes, it is too expensive but their power rack boxes are effectively a little mainframe (check the IO stats on those things) and the performance is - I have to admit - impressive. That said I would never buy it because I can get much better value elsewhere. Even so I think they could do this if they really wanted to.
They do have big advantanges because many large corps have become totally dependent on them having outsourced everything and lost internal knowledge. That internal ops knowledge is very crucial/useful to take full advatange of what AI offers. In the coming years much of it will leak into frontier models. After which corps might not need IBM/TCS etc and just rebuild internal teams to handle ops like they once did.
Another benefit is that you're not helping the AI labs get better at what you do.
> Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct)
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
You need to reread what was written. No one is proposing to take away your right to leak proprietary edge to the rapacious frontier labs that need to eat most of the economy in order to survive.
You can try open models via OpenRouter and omp without having to buy or rent any GPU. I routinely try open models to see where they are. GLM 5.3 with DSFV4.1 as advisor via omp feels Opus 4.6 level, i.e. totally able to ship stuff.
I do use Anthropic and OpenAI mostly because it costs me 1/40 and 1/50 of the api prices, which is way cheaper than Chinese subscriptions, but I’m ready to jump ship when they pull the plug. Fortunately they have nothing that forces me to keep paying them, and I don’t pay the crazy enterprise subscription costs, just the personal account prices.
I also die to be able to run the OSS setup locally to stop being spied on all the time.
For context, I averaged 10 hours a day prompting LLMs from when ChatGPT came out. At the start it was just function generation, then single file modules, then modules, and now I reason, design, implement and test with them.
An interesting thing is that with omp you can plan with a smart model and then implement with a dumber one. So for example, plan with Kimi 3.1 via OpenRouter and then implement with DSFV4 with zero data retention active, sorted by cheapest, and minimal tps accepted.
I get the people who are pissed about their cushy jobs evaporating, but if you know how to use these technologies it’s very freeing because it’s like having a company of people working for you, except they are not human so they’re not wasting their life working for you instead of using it for themselves.
You can make them do pretty much everything that is fuzzy-verifiable, you don’t have to worry about their feelings or that they won’t do what you asked them to do or stop, and you can activate them when you want at scale in parallel or not use them at all for days.
For me it’s a dream honestly. I never felt so much empowered in my life while being enabled to unlock my creativity with almost no boundaries.
I fall into the people pissed about losing jobs camp, can you explain how you keep a cheery attitude like you do?
I geniuinely don't see what I could make with this new tech, my brain immediately goes to "this app idea is worthless, anybody can make it now".
As the OP said really. It is very very freeing to just be able to give a task that in ye olden days you knew was going to be a pain in the arse to do, and it will do it without you. Amazing.
It's like having extra copies of yourself that you can just fork and get stuff done. E.g. the other day I was in a meeting and there were some heavy discussions about a few gnarly issues. I fired off a few prompts in parallel while the AI researches options, approaches, made a prototype etc all while I was present and paying attention still in the face to face meeting. I am so much so much more efficient as an engineer now - getting better feedback, more bonuses etc. It's never been better - I handle thinking about the requirements, the computer science and software engineering big picture etc, and the AI handles the syntax trivia for implementation.
The models are really really good now.
The only downside is that as a result of more stuff getting done, we need to have more meetings to discuss what actually needs to be worked on.
You didn't really answer the question though, how are you not nervous about your job security and how do you still see any value in software that you make with almost 0 effort?
People at work are cutting and pasting Claude output for use in routine messages on Teams, troubleshooting suggestions, and proposed architecture designs, and honestly it's some of the dumbest shit I've ever seen. I have to tell people to reword it in their own words, and I believe they often can't.
I've seen developers use Claude to make suggestions for infrastructure design that made absolutely no sense and did not consider our environment. I've seen people write English that grammatically made sense but logically did not.
I don't see enough complaints about this and it can't be just my company.
Is anyone else seeing really severe Dunning-Kruger?
I always check what AI says to me, and I use my own words. I think the average engineer is not.
> I don't see enough complaints about this and it can't be just my company.
They call this "slop" - I think it is well known at this point?
That it exists? Sure.
That it's a degradation in some environments? There's not a lot being said about this. The laments are usually developers sad that they're prompting instead of coding. My lament isn't about the art of it, it's the human reasoning that seems to have gone away in my workplace when it comes to engineers.
Think bigger. Don't think in features. Think in business workflows.
You are a consumer, so you naturally think like the consumer you are. But consumers want cheap stuff and only big big businesses are able to provide cheap stuff at massive scale. So stop thinking in b2c. For us little businesses the money is in b2b. Businesses pay lots of money to have problems solved. They want to focus on their core business and outsource the rest.
Consumers hate paying subscriptions. Businesses love paying subscriptions. Because then they can make their problems someone else's problem and focus on what their actual business is.
Making an app has always been approximately 1% of the work involved in turning an idea into a business.
That’s not true in my opinion.
Maybe it’s 1% or 10% today, but a decade ago, having/assembling the technical chops to know how to build something was a huge part of the meat / barrier to entry.
It can still be true though, because entry is nothing without sales and a continued search for product market fit. I don’t know about 1% but if you say ‘for every 100 apps launched in 2015-2020 only 1 turned into a sustainable business’ I’d say that’s in the ballpark.
If code becomes cheaper that may change the business fundamentals but it also means capturing customers becomes way harder.
In a former life I was a freelancer who built first version mobile apps for seed and pre-seed startups. I’m a decent dev, so what I built was always good enough. Virtually none of these startups survived, and there were dozens of them.
What did you move into next?
Even a decade ago marketing/growth was the massive lion's share. IMO just building something has been near meaningless since at least post-Ruby on Rails. The app stores have been inundated since the late 2010s.
I’d challenge that and say that the opportunity cost was massive. So you needed to be super precise and confident in what to spend the time doing.
Now that opportunity cost is much lower so experimentation is more practical than before.
I've hit a weird point myself. I'm not overly interested in building software for other people now. What I'm able to do is build all my stupid hobby project ideas for my very specific use cases and not have to worry about making them profitable in order to justify the time spent on them. I just built my own point of sale system because nothing on the market meets my needs, and my needs are extremely niche.
Very interesting! What kind of business is your POS for?
I sell vintage cameras in an antiques centre, but only 3 days a week. I’d like to sell online too but I have no ability to put aside stock for that, and similarly I only have access to the stock a few days a week, which makes shipping awkward. Pretty much every item is unique and needs listing separately, which is a pain. This is probably doable on some existing POS but it’s almost certainly much more expensive than just building my own (I think Shopify starts around £40 per month for what I’d need, then still require a lot of customisation). A major aspect of my business is buying stuff in the stall and tracking its eventual sale price, which tends to be a pain on other systems when it’s all one-off unique items.
I also do repairs and so I recently added a job management system into the POS, which has been really useful so far. Next step is to sort out the online store/eBay integration aspect!
Take your time to set up the tools and just build something small. Create a personal website, or a calendar that helps to track friends' birthdays, or something else of that sort. It's a good way to calm yourself down, because it will give you a good picture of both the capabilities and the limitations of AI.
I've found it very liberating that I can handle some of the tedious parts of a project to AI. In the end, I spend perhaps the same amount of time on a project, but more of it goes to important work and less gets wasted on plumbing.
Thanks for the advice. I will try to repeat this, as I have already used the new tech and found it useful.
I am however worried that for the kind of work that the majority of the worlds tech workers do, this is a direct replacement for.
I guess we'll have to live to see.
Ideas have always been worthless though, the alpha is still in making something good + useful
Upvote for what sounds like genuine curiosity!
In OMP just setting the "task" role (which controls the default subagent model choice) to some cheap Flash model like Deepseek or Gemini goes a long way towards making better use of your quota.
Sol and Opus both are quite eager to use subagents and since the subagent tasks are often well specified and then verified by the main model there's hardly any advantage to using anything better for the subagent anyway, so IMO it's a no-brainer.
For me switching the main model to Kimi K3.1 Pro would put me towards 100% open, but I don't know if it uses subagents effectively. Any other open models I noticed just don't utilize them very much.
People are underestimating the lock-in of Claude Code and Codex. Most companies are not buying a model, they are buying the ecosystem.
But the lock-in is not _that_ big that a big co could not pull off a migration, but it would be like migrating from Teams to Slack, not like migrating away from Adobe.
Small nitpick, but you can use any model you want with Claude Code and Codex - doesn't have to come from Anthropic or OpenAI
This is exactly how I got people at work onboarded to our local models. Everyone was comfortable with Claude or codex CLI so i gave them prebaked config files to use them with our local models. Ezpz
People regularly switch
Companies don't switch lightly.
They do when they see the cost savings and ROI. It’s the one reliable thing that makes a company switch from any tech to another one.
Looking at our AI spend today vs 1 year ago we have massive changes. Anything that can bring this under control and potentially save millions a year is being explored.
We will probably end up with a combo of open weight models for most work and frontier models for key projects. For now. Who knows what the landscape looks like in 1 year.
Yup. Hard to stop individual devs as well… I can plug chatgpt into claude code and vice versa or any configuration I want, there’s no lock in
The lock-in is that the subscriptions plans only allow you to use their own harness.
That'll stop em
[dead]
> which is way cheaper than Chinese subscriptions
Cheaper than the API prices you pay on openrouter. But not cheaper than this:
https://www.alibabacloud.com/help/en/model-studio/token-plan...
But hardware to run them is conveniently something IBM would be happy to sell you.
I didn't think that IBM had gear that mortals would want to run. If you're not running a mainframe, what do they have to sell you? They still sell Power servers and it looks like they have an AI card that can be used on both. But, if we're talking about the IBM CEO talking about on-premises, what are they really talking about hardware-wise? It's not something they would sell, is it?
This is more of a statement for their consulting customers, right?
My guess is if you want to run AI on prem and do it with IBM gear it would be via their x86 servers + OpenShift AI via IBM Fusion HCI.
I didn’t think they still sold x86 servers.
I can picture that they’d have the software side sorted out with RedHat. But I don’t know what hardware they’d push to support local on-premises models. It looks like when you buy Fusion HCI, you can choose from Dell or Lenovo as the underlying hardware supplier.
If IBM would ship a piece of affordable hardware that can run large LLMs efficiently I'd be lining up.
"Affordable" and "IBM" don't belong in the same sentence together, but they're happy to sell you their inference-capable hardware if you've got the wallet for it.
I don't agree with that. They made the PC revolution possible by making hardware that was at least affordable enough to be bought by a large number of businesses and individuals. It also set the stage for the clone market. I'm pretty sure that if IBM really set their mind to it that they could do this.
That doesn't detract from your overall point that they tend to make overpriced stuff, but I don't think they have to do that. Besides, what's overpriced? NVidia's GB300 workstation sells for 95K euros, a single RTX 6000 is now 20K. Surely even IBM can compete at that level.
That was completely accidental and they themselves tried to close the barn door with the PS/2 line in 1987 but at that point it was too late.
True, even so: They still make some pretty impressive hardware. They have access to fab capacity, they know how to make chips. They have the capital. If they really wanted they could do it.
As someone who worked for the big blue not that long ago I would agree that they have the capability. The top heavy C-suite might have to get over themselves and their buzzwords and actually try to create something besides an arbitrary EPS though. I wouldn't recommend holding one's breath.
In what way is the IBM Spyre not what you're asking for?
In a way it is, but I would see that as a starting point, not as the ready solution. They'd need to do what NV has done for their eco-system: fantastic libraries, a credible future upgrade path, easy ordering, manufacturing partnerships, the works. I figure besides the performance gap that they are at least 5 to 10 years behind NV with their present offering. But NVidia is dropping stitches and IBM should take advantage of that.
I think IBM sees the Spyre as a reason to sell more Power hardware but that's just not where the industry is right now and I doubt it will be enough to create sufficient demand. Who knows, I don't have any insight in what they're doing there, but there is massive gap between A GB300 workstation and what IBM currently offers. At the same time: the RAM crunch hits all parties and that's a major obstacle as well.
Yeah. For all the jokes we make about paperclip maximizer, we've figured it out. It's not paperclips, it's RAM.
That's the thing though. Software is cheap with AI, but Nvidia has Cuda. If software doesn't matter anymore (it does) because it's so cheap now, then Nvidia's moat evaporates to IBM's Spyre. In reality "just a little bit of software" has inertia and cannot be dislodged quickly, quietly, or easily. cf Microsoft.
That. And not starting every sale with 'call us' where they first want you to tell them how much you can afford to spend. I buy a fair amount of gear and if the sales process starts with some sleazeball sales guy instead of a mouseclick on a local webshop then I'm out.
As for CUDA: I think CUDA is a formidable moat but I also think that that is precisely the kind of thing where the likes of IBM could (if they really wanted to) make things happen. They're already in the space with Redhat and they have ample knowledge about drivers and low level code. Putting together a team that either makes a drop-in replacement or alternatively, that starts supporting their custom hardware for the big players in the inference space (VLLM, llama.cpp and Torch) would make all the difference. They'd have to keep doing it though, even if the initial results were lower than expected because that market is not going to turn on a dime.
Respectfully, those people don't work at IBM anymore. The organizational memory doesn't last that long.
You'd think that with a 250K workforce and an active line of hardware including an AI accelerator that those people are still working there. I could be wrong, but someone has to be doing that work unless IBM in turn outsources it.
They haven’t been _that_ IBM for decades
I know. But: they have gobs of money, they still have some talent, they still have a hardware department that I'm pretty sure would jump at a chance to give NV a run for their money. This would require a massive long term investment and a commitment that I'm not sure they would have the stomach for.
"they have gobs of money, "
They do? They are a spec in the ocean relative to big players offering inference and training their own models. Pretty sure one thing they are struggling with is the money to compete successfully in the ai game this time around.
10 billion give or take, market cap 250 billion, 250K employees. That's not a 'spec in the ocean' by normal standards, and IBM is a mature company, not an inflated balloon.
It's the "PC-compatible" hardware make those affordable, not PC.
> They made the PC revolution possible
Yeah they were only half a decade late to the PC revolution back in the day...
They weren't really late. In many people's minds IBM validated the concept of the PC.
When that happened I'd been working at Tandy (EU version of Radio Shack) for a while and even though we sold computers they were not taken serious until IBM came along and then suddenly we couldn't supply them fast enough.
Was Tandy only EU? In the TRS-80 days the name came up a lot, although maybe just because of the acronym ... 6809E was a good time
Yes, that was the first time I had a 'proper' processor to play with. A Dragon 32 (CoCo clone from the UK).
I've never seen Tandy used outside of Europe but then again, I had not traveled as much then as now so it's definitely possible they used the brand on other continents as well. I wonder how many trees were cut for those catalogs :)
It was only a difference in the store names. They were sold at Radio Shack stores in US. Tandy, originally a leather company, bought Radio Shack in the 60s).
And AU. Dick Smith started out selling Tandy's, before switching to their own clones.
Yes but unironically? IBM should be what NVIDIA would be competing with their NVL72 and DGX products. It doesn't make sense that IBM is just watching NVIDIA take its lunch.
> It doesn't make sense that IBM is just watching NVIDIA take its lunch.
If you have ever worked at IBM, it makes total sense. Completely dysfunctional middle management and on up.
I think they will need a far more aggressive management before such a thing could happen. But I can dream about it.
Not till Jensen buys his next leather jacket.
I just can't believe the price of hardware these days, a 3090, a graphics card released in September 2020, can still be worth £1500 or more. Honestly everywhere is sold out below around £1800. Can you really trust them not to have been rinsed on eBay and have an extremely short usage - they are still around £1150 used there!
We need a step change in the amount of silicon chips produced. This situation has been going on now for over 2 decades, initially bitcoin but ever since various high compute scenarios keep causing GPU shortages and now RAM and storage as well. This one isn't likely to go away anytime soon so we need a drastic increase in production to bring pricing down to reasonable levels.
It is not going away any time soon, but when it does it will be sudden and have a massive drop in demand.
It was pure coincidence that the AI-craze neatly followed the crypto-craze (many neo-clouds in the AI space were originally crypto miners).
Not exactly a coincidence. You don’t get an AI craze without wide availability of GPUs for training. Not for a much longer time.
I dunno, I always thought the AI craze started from ChatGPT which, in turn, was mostly driven by Google research papers.
Yes it would have costed them more to train the first few versions of ChatGPT if it wasn't for the crypto mayham, but it would still happen. The hardware was always available and still improving regardless of crypto or not.
Yeah of course it would have eventually happened but the GPU demand in crypto speeded it up, that's from Sergey Brin himself: https://www.trustnodes.com/2018/04/29/googles-founder-says-t...
It's going to boil down to cost of the hardware vs. pay per use cloud models. I'm on the 100$/month chatgpt plan (~88euro). With all the resets, credits, and all the rest I typically don't run out of tokens. This is fine for me. I might go to the 200$/month plan at some point.
In comparison, my highend mac book pro costs 105 euro per month (leasing cost). That's not quite enough to run the bigger models and besides running the GPU on a laptop constantly makes it a bit hostile for my lap. And the model I would need a lot more memory for the bigger models. It's nice for experimenting though.
If you get your own GPUs in some kind of server form, you could run those locally or in a data center. Or you just get hardware resources from your favorite cloud provider. Hetzner (one of the more affordable ones) GPU servers start at around 200 euro per month. I know some people that use that with open weight models. Works fine, especially if you can share the resources between multiple users.
At which point of course you are doing the same as cloud based providers are doing. But probably with a bit worse economics. It all boils down to whether you are actually saving money or paying for a lot of underutilized hardware that's running 24x7 when your people actually only show up about 8x5.
I don't think most companies should be doing that. It sounds like they'd end up with a lot of idling over capacity at a premium. The only valid reasons to do this would be securing access to GPUs or some very urgent need to keep things on premise (which a lot of companies insist is true for them until they realize how much they are going to pay).
> It's going to boil down to cost of the hardware vs. pay per use cloud models. I'm on the 100$/month chatgpt plan (~88euro). With all the resets, credits, and all the rest I typically don't run out of tokens. This is fine for me. I might go to the 200$/month plan at some point.
It's also about "insurance protection" against the AI company raising the prices or changing their terms (e.g. becoming more frugal with respect to token drops).
If they raise their prices you can switch pretty quickly, insurance isn’t worth much.
The speed at which you can switch is probably proportional to your dependence. But the thing with insuring against risk is that you need to cover all of them and not just cherry pick them.
If you are running your own models on your own hardware you take a bunch of risk as well. Is your hardware capable enough to run your models? Do you have enough of it and access to more when you need to? And is the stuff that you picked still going to be good enough in six months? And can you trust those model providers that you picked to keep up with the rest of the industry or are they going to fall behind and leave you stuck with obsolete but still very expensive technology.
Of course, this is an article about IBM who are in the business of selling consulting and services at a premium. They would probably love companies to self host their models on infrastructure that IBM supplies and consults on (for obvious reasons). The question is, should you buy what they are selling or can you do better? My simple observation here is that if companies like IBM are getting excited it's probably because they are looking to separate a lot of companies from some pretty significant cash. So, I'd be wary to buy what they are selling. But that's just me.
In a rapidly evolving market with extreme levels of uncertainty around tech and what will happen next, being reluctant to get pinned down on the wrong solutions is probably a good instinct. The biggest risk here is making the wrong bets. And you need to look at what it's going to cost you if you get it wrong and what it's going to cost you to fix it.
This depends a lot on how deeply integrated you and your workflows are into the idiosyncrasies of some specific LLM or class of LLMs.
[dead]
In this context yes, but for example at my job we are using LLMs to evaluate some outputs of cheaper/simpler models and to add context to them. This gets expensive very quickly with per-token pricing and would pin our entire business on the whims of one or two providers. With self-hosted open-weight LLMs we can just operate like we always have, it's a different kind of processing but otherwise we are free to do as we please wrt compute.
So it matters a lot that these open-weight models are good enough; we would likely not be rolling out this new processing if it was fully dependent on Anthropic and OpenAI.
I think, despite the desperate attempt at IBM to remain relevant (and in business), their CEO has a point here that do you really need SOTA models for everything?
I had a really nice time with GLM5.2 for a month or two - it was, in my experience, approx Opus 4.5ish level of capability when I think Opus 4.8 was the new hotness
Even today, I use Gemini 3.8 Flash as my daily driver for most coding. Its fast and good, even if it is not the absolute SOTA at coding benchmarks it really is very good. Yes I have tried Opus 5.5 and 4 Argon etc and yes the are better, but do I need that?
I think that we are going to be at the stage soon-enough where models are "good enough" for 99% of people's needs. The SOTA will keep on marching ever-onwards I am sure, but I would not be surprised if people stick with previous-gen/previous-previous-gen/etc models because why upgrade?
If this does play out like this, I do wonder how that works out for the frontier labs (specifically OpenAI and Anthropic, less so Meta & Google) - if open models or even just cheaper Sonnet-level models from a few generations ago are good enough for basically everyone's daily work, where is the revenue? how can you justify pumping billions and billions into new generation models and datacenters etc when there is no driving need to use them?
>do you really need SOTA models for everything?
For work in which you are in the loop, absolutely. Do people not value their own time or something? Why in the world would you use less than the best LLM available to you? You're not going to run out of the $20 plan even if you use Opus 5.5 at xhigh and code 8 hours/day (assuming you're doing it piecewise using a Claude product) so what exactly are you conserving by using something inferior?
For large continuous volumes of programmatically submitted LLM tasks, sure.
I have single sentence prompts in one of my legacy codebases that it takes Opus 5.5 over $1000 in API spend to return an answer correctly on. I am not getting by with a $20 plan.
That's ~41 million tokens. Your codebase better be the size of the entire MySQL codebase.
Similar size.
That's my point - do you need the best available to you? What if tomorrow your employer said no more 5.5, or Anthropic IPOs and suddenly investors want to see a profit and that 20USD a month for Opus 5.5 becomes 20USD per-day or even per-hour? We're already seeing corps restrict their employees token usage.
Could your survive on say Opus 4.8? Or GLM 5.3?
I bet you could if you had to.
I think it's the same argument about cars. Why wouldn't you want a Ferrari? Do you not value your time? Why not go as fast as physically possible?! You need the absolute fastest, most-capable, best-handling car for every single journey you take! Or... Back in reality not every journey you take is trying to lap the track as quickly as possible and just any reasonable car will get you where you need to go.
This is my point. Will Anthropic and OpenAI continue to exist selling Ferraris when actually people realise they can get by with a Toyota Corolla 99% of the time? Ferrari exists in the car world so I guess there is an argument for "yes". I am not sure myself - Ferraris are after all rare and exotic cars ... I suspect that even if we are not there now, we are already seeing current SOTA models that are mostly equivalent/better than the average engineer - another generation or two they'll be waaaaay better than the average engineer. How much more car do you need, and how much are you prepared to pay for it?
At least for me, I don't like using Opus or Fable because I don't like the way it talks to me (talking down to me, refusals, moral high-grounding, overly pomposity etc). 3.8 Flash is currently my favourite model due to how it communicates and works with me and the speed is much better suited for rapid iteration. So put me in the comfortable, reliable, use-it-every-day-to-get-shit-done-with-no-fuss Toyota Corolla camp. Its good enough for me and I don't get any benefits in my day to day life by using Opus 5.5 and it's sanctimonious personality.
Currently the difference in my work between the best models and okay models is pretty night and day to the point its not worth using the older models. Now I haven't used the newest top open source models. But if I give the same code base to say Qwen 3.6 and GPT 5.6, Qwen will analyze the source of the problems wrong, provide actually wrong solutions, write code that takes 100x as long to run. Its not even worth it outside of querying for snippets of code. GPT 5.6 will help actually root cause the problems.
Ah but if the price is 10x - as the IBM CEO suggests - your employer might just ask you to use that, and then expect you to make up the gap between the cheap and expensive models using your own brain instead of just pissing money up the wall on high token costs AND your salary. You know, like the old days when we actually had to do the work ourselves.
I think this is a very high likelihood scenario.
I think the point is that you are not everyone. Not everyone has your requirements, sounds like paying the premium is worth it for you, but others simply don't need it.
I love 3.8 Flash, if only it weren’t unusably flaky. Sometimes simple responses take minutes or I get rate limited after sending a single message (I guess it’s a shared pool rate limit). Have you not experienced this (would love to know if there’s a way to actually use it reliably!)
I've personally experienced three big jumps in LLM capability: o3, opus 4.6, and opus 5.5. I could go back to opus 4.6, but it did require way more involvement from my end.
What subscription do you use for Gemini 3.8 flash and how much usage do you get?
Of course IBM - the guys that sell you mainframes - would be saying this.
This is a win though. But not for the reasons people might think it is.
It is a win because it's a massive player now countering the new (temporary) behemoths. With all the leverage, budget and connections.
And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people. Meaning that they're going to be far closer to the baseline of sane humans just from averaging alone.
The guy that sells mainframes also gives away the Granite models for free. I've found them to be quite capable in content retrieval and long-running workflows, even (or especially?) the small ones.
> And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people
IBM of today is a hallow corpse of what it was. The hardware business remains but they’ve mastered extracting rent from imprisoned customers long ago. No sane person should willing go into business with them, same as Oracle.
Yes, but that was not the point I was making.
They are many people and none of them are in some sort of AI death cult like the EA polycule.
They're not suddenly good guys. They're just sane
"Cheap" is up for debate, but the models themselves are great. Kimi K3 (and now 3.1 Pro) was the first open model I played with and didn't think to switch back to a frontier model. Now it's my daily driver and I have zero complaints.
The frontier labs are in a tough spot. It's hard to justify their cost when you can get something that's easily as good for cheaper (and anecdotally, more reliable/accessible). I do HITL so my perspective will be different than a fully-agentic setup, but as soon as I started using open stuff regularly, I realized that any moat the US labs had built is drying up.
been playing with LFM2.5-24B-A2B on my m4 mac, 700-900 prefill and 60 t/s in generation, its doing pretty well writing some python scripts for me.
That’s not a bad little model! Way more impressive than I expected. I am still messing about with it, but I think it’s a keeper for general discussions.
Though if you have the room for it, Muse Glimmer is better in practice on low reasoning strength, and while its token generation is slow, it actually gets to answers pretty quickly.
I downloaded Muse today actually to try too. Seem it said a few times on here that’s it’s good.
Probably mostly me and one or two others to be fair ;-)
"There are dozens of us!"
The thing you need to know about Muse is that you control its reasoning/thinking depth in the system prompt (low/medium/high/xhigh):
… is pretty responsive and I've found its time to first answer to be surprisingly good.It is broadly comparable to Qwen 3.8 27B on "low" thinking, though it is weaker on very long context, one-shot stuff, but I dunno… I just like it more as a developer companion.
(I'm using Unsloth's version)
Considering the disclosed numbers for DeepInfer and the usefulness/popularity of DeepSeek 4.1 Flash, I expect it is inevitable that more companies are going to consider the self-hosting option seriously, and perhaps play with the idea of offering inference as a vendor.
Worth putting DeepSeek's own pricing rule next to that, because it sets the floor for self-hosting. In a closed-door investor meeting in May 2026, Liang Wenfeng said the API is priced so that "we go to the market and buy a batch of equipment, and recoup the cost in ten months" - that being, in his words, a reasonable profit. He adds that maximizing profit would mean charging more, but demand is inelastic in that band (he cut one model's price to a quarter and the team was delighted). I translated the excerpts, since the transcript only circulates in Chinese: https://telegra.ph/Liang-Wenfeng-on-DeepSeek-cost-restraint-...
If the vendor price is calibrated to a ten-month hardware payback, then at equal utilization there is no arbitrage in self-hosting: the crossover is roughly whether your sustained API spend exceeds the hardware plus that ten months. DeepSeek's own disclosed numbers give the benchmark to beat: 226.75 H800 nodes averaged over a day (~1,814 GPUs), 608B input tokens/day of which 56.3% hit the KV cache, 168B output tokens/day, ~73.7k tok/s input per node while prefilling and ~14.8k tok/s output while decoding.
That utilization is the hard part, and it is roughly what the IBM CEO is pointing at with "where are my kernel engineers". Self-hosting open weights on two or three racks is not the problem; amortizing them at a frontier lab's duty cycle is.
Everyone I know is either starting to or planning to switch to open weight models. Cost savings are just too good.
This is what will cook frontier AI and probably prevent the investment scale to happen for AGI. Humans + good enough cheap AI is just more economic and unbelievably productive. Like why we fly on jet technology from the 1960s still with incremental improvements.
It’s why Anthropic is desperate for capital and trying to get to a scale where AGI appears. But I’m afraid they won’t get there in time and the money to build the bigger model will have dried up.
RIP frontier AI, the top customers can easily self host
What happens when their competitors decide to pay for a more intelligent frontier model?
What happens when your competitors invest in better and more expensive tools or materials in general? They (potentially) make better products, but those products also cost more. It's turtles all the way down.
Cool! That's when you pay the difference in marketing and auto-win your sector! People don't actually buy the best, they buy what they think is.
Ford vs. Ferrari, except that the Ford is actually a few months old Ferrari.
Does Ferrari have any moat if it can't convince the government to ban Ford?
The target audience is likely those who can't or won't use frontier models unless they are on-premise.
You buy the competitor and kill their culture from the inside, ruin them and take a tax loss on the company.
depends on the economics and much more, the best Ai cannot do a lot of things, 90% of human tasks by some reports
we technologists tend to extend what we see and think to other realms where it doesn't make sense
I once overheard someone in a coffee shop saying that she needed GPU to run inference. But gaming laptops are too heavy with short battery life, while MacBooks are expensive.
I really wanted to tell her -- what she needed was compute, not hardware. Her best bet was to rent GPU which is easily available from many services. If she was paid to run inference, cost is not a concern. If she did it for her own work, she wouldn't need to shell out a ton of money initially and then be left with hardware that she no longer needs when the work is done. In any case, she wouldn't need to worry about purchasing hardware or battery life. ssh to a remote machine is simply way superior than running inference on a laptop she carries around by any means except in a few rare occasions (e.g. no Internet access).
Same thing for a company. Why would any company who has a business to run get themselves burned in acquiring GPUs, setting up racks, load balancing etc when Bedrock/Azure Foundry etc is right there with tons of both open and closed models and ZDR? And then see hardware depreciate in just a few years?
Is anyone using onprem open weigh models for programming? If so what models, gpus and feedback?
How good will be the business of dealing with 2nd hand GPUs for open weights model in the future? What will the AI labs do with their GPUs once they are beyond their EOL? Can somebody please advise?
I was able to get this related article
https://rcrtech.com/semiconductor-news/what-happens-to-old-a...
He has a lot riding on this particular idea, and I think HN will generally be receptive.
I am not a believer in winner-take-all mechanics for models (even for AGI -- look no further than the natural world for many examples), and a large F50 company committing money/strategy to something outside the frontier labs is a great thing for the ecosystem.
If the focus is on models tailored for specific tasks, then I think that’s absolutely true. I’ve built a desktop app that uses Whisper and Demucs myself. Sure, it’s almost 2 GB and uses quite a bit of the device’s resources, but it’s free, runs fully offline and does what it’s supposed to do. I think that’s what really matters
Are there inference products focused on say, optimizing frequently used internal contexts, so that they are compact in the kv cache? Perhaps there rep could be optimized over hundreds+ of distinct real use cases?
Maybe your law firm doesn't want your data going outside your business. But you obviously want high quality analysis.
I think this is correct in the general sense.
In the IBM-specific sense, the most unintentionally hilarious LLM interaction I have ever had was asking Granite’s 4.2 30B reasoning variant the car wash question.
Reams and reams of corporatese about why it wasn’t wrong when it was.
> something that’s 10 percent of the cost
Is it though? You’re likely paying more on prem per unit of electricity and the racks of NVIDIA cards also aren’t exactly cheap.
I don’t see how that leads to 10%. I can see it being a good pitch if you’re in the biz of selling on prem servers though
So you'd like your Windows IIS sever instead of the Linux one?
Happy to sell you some Oracle too.
Any tips to get started with just one rig? A 4070 ti on a two year old Fedora workstation desktop?
What are the tools to look at first for usable models at that thin compute capacity and how do you go about exploring training for hobby and learning purposes?
Check out https://www.reddit.com/r/localLLaMA/
This is true, but also, beware the shovel sellers telling you how great shovels are.
Can we please stop giving weight to CEOs talking their book? Like of course IBM wants you to think this, it would be great for them if it’s true. And maybe it is true! But IBM’s CEO saying it’s true tells me just as much as if a frontier lab said it’s not: nothing.
"CEO Said A Thing!" Journalism
https://karlbode.com/ceo-said-a-thing-journalism/
Great term. Plus the related "Politician Said a Thing!" journalism
thank you, I was trying to re-find this the other day
But you know - it would be funny if somehow Open Weights are the thing that completely save IBM
Linux already served as a life-preserver for them also.
It was a brilliant move at the time for IBM, and has helped them shift their business more into providing services and support.
I imagine they are panicking a little with LLMs reducing the reliance on external companies for software support, so they are really hoping to shift it to on-prem AI, which is where the bias in his statement comes from.
I think AI could very well support a return to on-prem computing.
It’s a conversation point off which a discussion can be had
the only credible "conversation" is "well he would say that, wouldn't he?"
"we are delighted..." said a spokesman
It's not a one time thing though - onprem inference requires continuous monitoring and tuning to get the most out of your GPUs. Looking at 25-50% performance benefits. Today, it does require dedicated Inference Engineers...
That sounds like a task that can be automated.
It's not that easy.
The optimal inference configuration varies according to the model, the specific traffic (which changes in time), and the specific hardware it's running to. The "knowledge" for it doesn't exist yet.
Usually in similar scenarios an agent will try to experiment with various configurations and get to a good result. However, you can't have an agent test out different inference serving configurations/architectures, since every experiment might involves 10-60 minutes of spinning up a model (on very expensive GPUs)...
That sounds like a task LLMs will soon be able to take over.
Do not do that if you can't prompt it correctly. It is a potential disaster waiting to happen. LLMs are likely to pull things off the shelf without real defense in depth and that's a problem. Maybe slightly less if it's only in an internal network, but still.
The whole point of this was for internal usage.
It can be very dangerous to let people loose with such advice, FWIW. They don't necessarily know what "internal usage" actually means, or how important it is to keep it that way as long as you don't have a real expert to at least help direct the LLMs.
That's based on the idea that humans can learn and adapt and solve problems, and that's not what companies want to think about when they're seeking a profitable transaction such as hiring.
Anyone in disagreement hasn't tried open weight models for sure.
DeepSeek v4.1 Flash or GLM 3 are already very very capable models.
Only bottleneck at the moment is the hardware prices.
Qwen 3.8 Flash with Strata here, the problem with Strata is that you cannot easily share the server as the performance will collapse outside of a single user.
Easy to say when you have your own hardware.
This on the Front Page the same time as Anthropics S-1, fun times!
Good and cheap if you have the budget to buy IBM?
There was another comment in the IBM earnings call on enterprise AI spend [1]. Specifically, concerns around ROI for AI experiments in the enterprise and difficulty incorporating AI into enterprise workflows. This was surprising given the hype around AI, and it came across as a negative indicator.
[1] - https://www.ibm.com/downloads/documents/us-en/17f55bd310d5fd...
Buying your own car works out cheaper than taxis - according to the car dealer.
IBM fucks sam and dario. Priceless.
I’ve been in tech for a long time… I have no idea what IBM does. Now, I get that I’m not the customer, that their customer is “business” but not my business… no… bigger business. The biggest of businesses I assume. I think it must either be a club I’m not in, or IBM is three half racks in a trenchcoat submitting invoices for millions to companies where people just pay it because “it’s IBM, it must be important”.
Review the specifications for the latest Telum II chip if you need a hint. 5.5ghz base clock.
If you absolutely need to scale something vertically, IBM Z is the final stop on the line.
X86 is great at hosting an entire farm of applications. You would never want to install salesforce or gitlab on a mainframe. That would completely defeat its purpose. If you've never encountered an appropriate workload, IBM might never make intuitive sense.
Also, the roadmap:
https://newsroom.ibm.com/2026-08-24-ibm-unveils-next-generat...
> an appropriate workload
Honest question: What would such a workload look like? Why would Salesforce or GitLab not make sense on a mainframe?
Ironically one of the smaller places I've worked at tried to use an IBM product once - I'm still not clear on what it was meant to do or how it was better than any cheaper alternative, so I assume someone in leadership was fooled by a very impressive pitch.
Lots of work went into integration and it never went live.
> I have no idea what IBM does
For one thing, amongst many, IBM mainframes process 70% of global transactions on a value basis
Transaction can be a nebulous term; think of credit card transactions, ach and wire transfers, mutual fund buy/sells, bank deposits and withdrawals, etc.
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
Good enough for the old world, but I'm not so sure it's good enough for what's coming.
What do you think is coming that isn't already in the old world?
Cerebras with Claude 8?
OpenAI most likely pushed Cerebras into a contractual agreement not to do business with Anthropic through 2030. ("Most likely" because the names of the companies Cerebras is contractually obligated not to sell to are redacted in public filings)
Imagine a regulatory framework so weak that a company feels comfortable to do something this anti-competitive
Maybe bribes is the framework? It's hard to tell these days.