Really dumb question from a software guy. Why aren't the labs burning their frontier models into chips already? Seems like the performance gains and cost per request would be worth it. That said, I understand neither the economics nor the physical challenges to doing this.
Model SOTA moves faster than chips can be designed or produced. You'd need to commit to a particular model for years to get payoff while still burning buckets of money producing new SOTA models to keep up with the competition.
It's why everyone and their dog runs these things on GPUs. When a new model supercedes the previous one, so long as you've got the memory for it your chips aren't obsolete.
I'm looking forward to someone picking a model to be "good enough" (say, qwen 4.0 or something) and selling them as peripheral hardware
At this point, LLM's are "good enough" for all kinds of tasks. Instead of making them more capable, now the efforts are making them smaller and cheaper.
IMO this is the dream scenario! Cheaper and faster at the current level of capability gives us incredibly useful tools without the worst of the risks people fear. (Though there are certainly already great risks at the current level of capability as well.)
Ah that makes sense. I'm skeptical that AIs doing things on their own will be competitive with humans using AIs to do things. But I'm not incredibly confident in this and I think it's sensible to think the opposite. But to me, I just think about all the things I can accomplish with the aid of very good, fast, and cheap inference.
I think the sad part is that whether it's any good or not, it will be cheaper, and that will be enough even at current performance levels to cut a huge number of jobs. I think people exist in the present moment largely because legal questions about liability are not settled well enough for executives to start the great purge. I believe on some level my continued employment is simply because people in leadership positions would be incredibly unwise to not have a human to blame when things start going wrong. People are expensive insurance policies for the time being. Once there is some legal framework to absolve executives of their own culpability or they figure out a good way to insulate themselves from personal responsibility, then the real "fun" will begin.
The problem comes in to how many bullshit jobs we'll have to create to ensure that enough people can still earn a living to ensure that we don't have riots and revolution in the street.
Companies being profit seeking entities that are actively hostile to social wellbeing would gladly put all of our money in a machine money making loop and leave all but a few humans out.
Like mostly all tech, AI is destined to get cheaper with time. We're still pretty early on and are experiencing a hardware crunch, which will ease eventually.
AI doesn't have workers rights, doesn't burn out, doesn't get sick, can be instantly onboarded, etc. The second we can be fully replaced with AI, we will be. Plan accordingly. I'm pursuing FIRE and considering moving into a trade.
> The second we can be fully replaced with AI, we will be. Plan accordingly. I'm pursuing FIRE and considering moving into a trade.
Doesn't matter what plans you make, the impact will be across everyone!
Moving into a trade won't help, because the supply is doubling while the demand is lowering. Fewer people with money to spend; they'll fix their own damn toilets if the decision comes down to "buy food" or "hire plumber".
You forget that once all lose their job will all go into trade. Means there will 1000x more people to compete with into trades, which means race to the fucking bottom and lowest salaries if you can even find job. Good for real estate owners, cheap repairs. But probably won’t get any rent paid because people can’t find jobs.
Idk, has china printed houses successfully? And if there are reports of that can we trust that the information hasn’t been gov altered to make a failure seem like a success?
I’m not “leaving them out” i literally just don’t know anything about them.
And yes i am skeptical as a default on this subject. But still - photos don’t actually tell the whole story. What was the cost? How was the project schedule? Etc.
I was skeptical when they first started doing it 20 years ago.
Now, right wrong or indifferent, it’s a known entity there - for a LOT of projects. Enough of them exist now that…I’m skeptical they could find enough stupid people to keep paying for it. I know the government has a hand in a lot, but that only goes so far, for so long…
training data for physical tasks like that is much, much harder than just collecting the entire internet. It'll be resistant to AI much longer than knowledge work will be
We already have construction codes. Fitting techs with AR headsets could gather more data. Existing structures won't be the first step due to variences. The best place to implement this is modular building in controlled automated environments. Even if we enhance the current productivity, we will create a problem due labor supply driving down incomes, especially if people are coming from the other jobs affected by AI.
You can barely capture folding laundry using task data, and you're vastly underestimating how much independent decision making the trades do in their regular work. If I'm honest this whole thing reads like someone who's never been exposed to the actual work.
Out of business, but not necessarily out of existence.
If there is really nothing for humans to do, whoever has control over (not necessarily ownership of) enough robots and AI to directly maintain and grow their collection of robots and AI, has something functionally equivalent to a breeding population of the stuff. (If they can't make more of themselves and humans made the initial batch, then that's a job humans can do so the initial condition has not been reached).
None of this helps people figure out how to look after their own interests in the meantime. The rest of society may carry on as today without AI, akin to the Amish if we're lucky (rejecting further developments unless good for us) or like Pol Pot if we're unlucky (reject anything that nerds like because nerds liked AI and everything breaks down).
On the other hand, the institutions may simply fail to handle reality, like in the Great Depression.
Financial Independence, Retire Early. It's a whole subculture online, based around the idea that if you can grow your investments to ~25X your annual expenses, you can safely stop working and live off those investments.
In the current economic system.
There is no natural law that states that only the people lucky enough to be born rich and/or well connected to real the rewards.
Yes this is why I don't invest in hardware. TPU is better than GPU when you have a good model already. The models are already good enough to sell as an accelerator card made as a TPU. I would buy an Opus 5.5 card.
No, there still plenty of tasks that LLMs can’t do well or at all. Yes, we should absolutely make models cheaper but there’s a long way to go until they’re “good enough”. And I’m talking about the frontier closed models too, not just the open ones. So it’s not an “instead of” position we’re in. It’s both.
> Model SOTA moves faster than chips can be designed or produced.
From what I remember working in that area the hardest part is getting masks for a design. Masks were developed in the span of half an year. Masks also reusable, they can be mixed and matched and this is why fabless companies work with fabs to produce specialized masks for them, it saves time for consumer to have masks for some macroblocks prebuilt.
Given some amount of work with the fab before main pipeline set (I think a year long process), one can then spew LM-on-a-chip in six months or less and much more than 2 per year, because there can be several LMs in pipeline.
FPGA speed is a common misconception. They often have run at quite modest frequencies, and come with limited memory capacity compared to a GPU at same price range.
Speed (clocks speed) , is dependent on your design layout, and for any resonably advanced layout, it takes lots of knowledge to push the clockspeed beyond 200Mhz on 'consumer'/prosumer models. (Compare to a few GHz for GPU/CPU).
FPGA speed shine where they can pipeline massively parallel calculations through pipelines with minimal lookups.
Wrong. This is one area where FPGAs have an insanely unfair advantage compared to CPUs and GPUs. Yes the SRAM is limited but you have so many individual blocks and all of them come with dual ports and getting the maximum frequency out of block RAM is much easier than getting the maximum frequency out of programmable logic.
If you wanted the highest possible memory bandwidth while being free to look up hundreds or thousands of independent memory addresses at the same time you're better off with an FPGA.
E.g. with an Efinix Titanium Ti180 you could hypothetically have 2560 simultaneous memory requests per cycle all pointing at a different address and process those requests at 1 Ghz.
Yes you are right, I didnt express what I meant very well.
looking up static values is quick and easy. When you need to lookup results from previous stages of pipeline rather than just feeding them forward, thats where I run into trouble. But I am a relatively fresh FPGA designer, so I am sure it can be done. And I probably need to level up my boards a bit too.
They have trillions worth of obligations to their partners in terms of compute purchase agreements and equity. Burning models to chips isn't really something they can blow money on just for funsies, they need to be able to justify it. The above comments are hypotheses as to why they haven't yet.
It isn't just a matter of speed, it's also a matter of model quality. If they take 6 months to burn Fable to chips, and it takes 2 years to break even between design, custom fab, energy savings, etc, are those chips even worth running when the new models that are running on GPUs at that point are producing 10x better quality results?
Sure, your 2.5 year old models are running faster, but you can't drop prices on them without pushing the break even point further out.
If the cost difference isn't incredibly significant, will people even want to pay for the 2.5 year old model, or will they get more value for their money paying more to get better results from the newer model?
There's a lot of open ended questions that I don't have the insiders knowledge for to suggest whether or not such a capital outlay would be a worthy investment.
My guess is that state of the art stuff will stay on GPUs and models burned into chips will be for "good enough" applications that people are still teasing out. Probably highly specialized models in automated sensor units and such.
FPGAs are FPGAs by virtue of putting on the chips vast, vast arrays of wiring that can be controlled by software. Any given utilization of the FPGA will leave large fractions of the chip resources unused. If you've got a highly stereotypical use case FPGAs will have a "highly stereotypical" set of components being unused, where it would be better to use that space instead to do real work. A lot of people only see the "pro" side of the FPGA proposition without realizing they come with some very substantial "cons" that are intrinsic to the way they work.
I guess that's why they work in particular niche spaces like a synthesizer where you have a max of 8 voices and every voice goes through the same pipeline (osc / filter / env / amp) and everything is necessarily running all the time. In that sense I suppose they're very good for modelling any kind of analog circuitry?
Even then, while there are some amazing FPGA-based synths available, companies like Korg just put their code on a raspberry pi and call it a day. The same is true for emulators (SNES Mini etc. are also just raspberry pis under the hood iirc)
You don't really get an FPGA for capability. CPUs are much more capable, and they're general-purpose so they can do absolutely anything with about the same efficiency and just a little more code.
You get an FPGA for timing. They're less capable, but (in many common design architectures), they output their results once per clock, every clock, on time, every time. If you can hit a fabric clock of say 100MHz, clocking all the weird logic you can stuff in there, it gives 100 million outputs per second, never skipping a single one for any reason (short of total failure). The penalty is that making a small change to your desired "program" can be very expensive, and many things won't be realistically possible at all. Or at least won't fit into a part that you can buy. But things like audio, video, and high-frequency trading love being able to guarantee timing.
(Of course there are other ways to write your FPGA HDL, but that's one of the more common ones. And you do see DDR-style clocking, and similar, every now and then.)
> with about the same efficiency and just a little more code.
It depends. For some things, CPUs don't even come close. An XCVU13P FPGA can handle 1.2Tbps of full-duplex Ethernet @ 1 billion pps. And that part costs less than a grand at moderate qty, and with significantly less power consumption than a CPU that'd be capable of operating a dataplane at these speeds.
The point I was trying to make is that the CPU is a general-purpose creature and doesn't really care what you want it to do. If you had a CPU that could handle 1.2Tbps of Ethernet packets at 1Gpps, it could do a whole lot of other things involving 1.2Tbps of data flow too, very easily, if someone wrote the software. And more. (But you're probably not getting 2.4Tbps out of it, no matter what you do.)
An FPGA can not. There's plenty of things that those XCVU13Ps just can't do, or would do worse than a $1 microcontroller. (Setting aside for a moment implementing a CPU inside the FPGA... which does actually happen in just about every large-enough FPGA design, which is its own discussion....)
I'm not really sure what you're saying here. An FPGA is inherently more general than a CPU because it implements any digital logic including a CPU. They're just more expensive and have clock speed limitations. Saying if you had a CPU that could do 1.2Tbps is just as silly as saying if you had an FPGA that easily operated at 5GHz.
The current largest FPGA, the AMD Versal Premium VP1902 has 18.5 million logic cells. That's not even enough for the smallest whisper.cpp model (75M).
You'd have to order hundreds of thousands of them (or millions) to serve even a single copy of a frontier model, and at that scale inference quickly becomes starved by the speed of light.
Well, you'd use BRAM to store model weights, not fabric. But still, you only get a couple hundred MB for probably close to US $100k per chip.
It's likely that the major FPGA vendors will soon announce parts specifically architected to support LLMs and similar models. But the current generation isn't suitable for that at all.
Would BRAM even have enough bandwidth? The reason I quoted logic cells is because that's the way to get instant throughput, which is practically the only reason to use an FPGA over something like a TPU.
I think so, because memory bandwidth really comes from bus width more than clock speed. You can construct 36-bit wide BRAM arrays with bus width comparable to HBM, just by specifying multiple BRAM arrays in parallel.
But that's not an efficiency an FPGA provides in the first place.
Additionally, it's not clear how well large mask roms scale. For instance the Nintendo switch cartridges were expected to be mask roms, but instead are Macronix's XtraRom technology, which is essentially a flash cell array made denser by removing the erase functionality. So basically the die gets manufactured with all bits at the same state, a late manufacturing step either empties or fills the floating gate of the bits you want different, and then it's treated as pretty close to a mask rom. It's not even clear if the bits can be changed without a bare die, a floating probe array, and specialized hardware. Though, like flash the electrons in the floating gates will eventually tunnel and cause the data to bitrot.
So from that it appears that even at the tens of millions of chips volumes that would make sense for essentially whatever size of maskrom, the memory manufacturers tap out at 128megabit for a mask rom chip, and push you towards something flash esque.
And at the end of the day, flash without the erase functionality is pretty damn close to a mask ROM, and lets you write it near the end of manufacturing rather than at the something close to the metal 1 layer.
8 months ago, Taalas claimed https://taalas.com/the-path-to-ubiquitous-ai/#:~:text=Upcomi... that "Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM. It is expected in our labs this spring and will be integrated into our inference service shortly thereafter. Following this, a frontier LLM will be fabricated using our second-generation silicon platform (HC2). HC2 offers considerably higher density and even faster execution. Deployment is planned for winter."
Nothing was released in spring, and 2 months ago AMD announced their acquisition of Taalas. That doesn't exactly inspire confidence that their frontier LLM will arrive as promised.
> AMD announced their acquisition of Taalas. That doesn't exactly inspire confidence that their frontier LLM will arrive as promised.
Why not? AMDs chip design experience and production capacity are magnitudes larger than a small startup. Assuming that they acquired Taalas for their technology, I don't see a reason why they couldn't.
I think this company was recently acquired by AMD, so hopefully they'll start getting some this into production. I know OpenAI was working on model-on-a-chip too.
Terrafab is just going to have their entire capacity bought out. Genuinely. Demand will increase to exceed supply no matter how high supply is right now.
Most responses here are along the lines of "model capabilites move too fast to build hardware for".
I think the fact that there are plenty of 1yr+ old models on openrouter serving hundreds of billions of tokens a month shows that there's plenty of use case for models that are "good enough. Cerebras' entire business is serving older models at high speed. I would happily use an opus 4.7 at 15k tokens per second. The intelligence per second of an ASIC still makes sense even with rapidly evolving models.
"Intelligence per second" is a striking phrase! I'll be turning it around my head at a moderate IPS until I hopefully make something of it.
But you're right in sense: moderate intelligence at superhuman rates (and presuming moderate energy usage) is very compelling compared to an intelligence that takes 1000 years to return "42"
I look at it this way. A year ago is a long time in AI terms, but not that long. Those models were already decent for the tasks we're using SOTA models for today.
So imagine taking a year-old SOTA model and running it at 100 tokens per second on an edge device. That's enough to feed a screen's worth of content through it and power decent multilingual message suggestions on IM.
Imagine running it at 1000 tps. That's enough to reparse that screen mid-keystroke, and give you semantic autocomplete in text. Or fully general "the phone has a good idea of what you're attempting to do" context at all times.
There's many, many new classes of features that will open up if decent enough models can be run on edge devices at 100+ "intelligence per second".
Or vice versa... 1000tps == 1tps via solar power + a capacitor (or whatever).
I'm poking (lightly) at the locallama game, have a fancy MBP5 with gobs of ram (so I can demo/trial locally) and have been semi-waffling between whether to chase a mini or studio for local "always on" type stuff.
My outcome was "CapEx v. OpEx", and dropping another $5k for an aluminum cube buys a lot of OpEx (eg: just trickle-drip HF/OpenAI credits to a raspberry pi or VPS orchestrator rather than trying to do the inference locally), ie: $5/mo inference for 1000 months.
HOWEVER, there's definitely a role for that 1-10 TPS type "ambient inference" that I wouldn't mind sustaining on any sort of always-on / local / private compute. My main email address is still on ...@yahoo.com and their spam filtering has gone to absolute shit.
Being able to have the always-on mini (local, trusted, no private data leaves my control) poke at the IMAP/email and thresh it into SPAM/HAM/Personal/Political ... random spot check, I'm getting ~5 emails per hour, and that's completely tractable for staged low-med-high processing. (Subject + rules.py? Subject + Body + LocalSlowTPS? Subject + Body + LocalDeepTPS? Subject + Body + RemoteLLM?)
Even if you did 10000tps of "jev" that's an incredible value... not quite "Literal AI Packet Router", but as you're dancing aroud saying... "Speed is a Weapon"
Look into "OODA Loop" => """The OODA loop is a four-step decision-making model—Observe, Orient, Decide, Act—created by U.S. Air Force Colonel John Boyd to help leaders make fast and accurate choices in chaotic situations. // The main goal is to cycle through the loop faster than an opponent or changing environment. By operating inside another person's loop, you create confusion and outpace their ability to respond. While initially designed for aerial combat, it is now widely used in business, sports, and crisis management."""
It is still a bet. Is the current model still going to be good enough next year? We have no idea what will change. If the change is minor improvements than current fable on a cheap is cheaper and better than next years sonnet on GPU. However there are plenty of things people want that maybe they will deliver and suddenly I wouldn't touch today's fable when I can run next years sonnet instead.
But I think it's a good bet. I think that in two years, if I can get opus/sonnet 5.5 or the gpt-6 models for much cheaper and faster than whatever the "frontier" is at that point, that this will probably be a great trade for most of my work. I certainly don't know that for sure, that's why it's a bet, but it's what I think right now.
I wouldn't quite say that about any of the open weight models at this point. But I'm hopeful that will change in the next generation or two of those models.
Also "faster" indirectly means "more intelligent" with reasoning models.
For example, Opus 4.6 Max was somewhere between Opus 4.7 Medium and High in some benchmarks, but it was slower. If there was a way to run it 10x faster, the economics would be different.
From what I've read, not only are some labs doing it (other commenters already mentioned).
But it's complicated for other reasons, one being that the number of parameters for frontier models (especially with MoE models) are so high, and not always utilized (once again, thanks to MoE) that it would actually be incredibly cost prohibitive, if not impossible, to attempt to make giga-chips that would allow running it.
I definitely do believe that we will see more and more specialized chips over time, but putting the entire model on a chip is still a ways away.
I believe Taalas has a heavily handicapped llama 8-billion parameter model. And it still pulls >200W to run.
I can't imagine how anthropic or open ai would be able to burn a multi-trillion parameter model on a chip, we just aren't there yet.
Yes, and startups do exactly that. Check out Etched https://www.etched.com/ where they made a Transformer specific GPU (basically a form of ASIC) where they bet that transformers would be the dominant GPU architecture for running AI / LLM workload.
The bottleneck, for inference at least, is memory bandwidth. And that you can't make any faster by making it specific to your model.
So companies try to maximize the memory bandwidth they can get, balancing tradeoffs of power/area/programability of their chip. Right now they feel like the economy on power/area is not worth the decrease in programability/flexibility.
Presumably though the kernel has a pretty specific set of operations done against the weights in memory. Burning the weights into the memory with local memory cores capable of the kernel operations would be a lot more efficient than round tripping busses.
The primary constraint isn’t likely what’s possible to do, but that the kernel and weights are too variable right now and the patterns too poorly established to bake into hardware accelerators yet. Margin pressure is also not there yet.
I suspect as the marginal utility of the frontier improvement settles into diminishing returns (I suspect we are there already tbh) baking hardware models with ROM, working set, and kernel cores collocated will be the frontier space as the goal will become reducing capital spend to utility levels rather than research levels.
Once someone has a model that is sufficient for almost any practical use, making marginal inference cost effectively zero will be the competition frontier. I do shed a tear for all those lonely data centers as compute densities will almost certainly make most of them a terrible investment.
You're starting to hint at compute-in-memory as a general replacement for CPU/DIMM layouts. That could be useful for far more than LLMs, world models, or any sort of AI. It takes a bit of a different software development stack than a standard architecture though.
> Burning the weights into the memory with local memory cores capable of the kernel operations would be a lot more efficient than round tripping busses.
Sure, but now we're not talking about just burning the weights into the chip, but also designing a new architecture that has memory local to each core. A new architecture would then require a new programming model, which means new inference stack, which may mean new training stack.
I don’t think it would require a new training stack, and I’d imagine it makes more sense to distribute cores with memory. The cores can be simplified to the functions of the kernel since the inference kernel can be expressed as a reduced set of optimized functions in the pipeline rather than a general CUDA core. If the model is burned into ROM, the compute pipeline can be baked into the core.
I think one limiting factor here is that Big AI cannot focus on building solid, stable products that do a job well. It's okay if it happens incidentally, but doing it on purpose would be self-defeating.
Their stock price, be it public or estimated, is heavily pricing the notion that they are first and foremost Growth companies. Therefore their focus must remain on ever better and greater things. If they lose focus and get distracted by lesser endeavors, their valuations crumble, their ability to raise capital vanishes, and their runways collapse before they ever have a chance to reach their end goal, whatever that may be.
That means the boring job of productizing AI models into reliable systems that won't vanish in six months is left for a smaller company willing to pick up the crumbs. Unless they get acquired by Big AI before getting it done.
As a SW person (working with silicon folks though), I was always interested in the same idea. So I just vibecoded a little compiler from LLM (config.json, weights) to RTL (happy to share if you want). Then synthesized it via yosys.
I picked TinyStories-1M, synthesized via yosys against SKY130 PDK. It's GPT-Neo with 8 layers. At int8 it's ~30mm^2 per layer (including kv caches, multipliers and all the attention stuff). So it's 250 mm^2 for the full toy model.
I'm kinda surprised, I spent no time, so if you want to etch a toy model, you can do that (you can probably reduce the area significantly). If you want a mask for that is probably going to cost 100k though.
It will happen once models get good enough for specific applications like robotics; there the hardware is fixed and power requirements are strict, so it makes sense to have dedicated silicon. But for use cases that doesn't require onboard, it's always better to get access to the latest and greatest, until we get to the point where improvements are marginal between model releases (we're not there yet)
Would you pay to crystalize one of today's models in silicon so you can use it in 2028, or would you wait for another 6 months to see how models improve before pulling the trigger on that kind of commitment?
If I controlled a budget like this, I think I would put some portion of it toward paying to crystallize one of today's models in silicon, yes. Not 100%, but I do think this makes sense to invest in at this point. I would not have said so a year ago.
Naive question: would flash or some kind of write once memory be dense enough to replace a mask ROM? I'm wondering whether it is feasible to manufacture "blank" chips at the semiconductor factory that have a fixed architecture, but are model-agnostic. These chips would get the then-current weights burned in on first use. It's probably quite wasteful, but it could keep the same chip design alive for a longer time.
DRAM seems to sit in the economic sweet spot of density and bandwidth. I'm not sure if any other storage technology can provide non-volatile or one time programable storage with either the same density and bandwidth. Although I must say, it seems insanely wasteful to use RAM to store model weights, there must a better solution.
The closest anyone has gotten is Cerebras with their Wafer Scale Engine. It uses SRAM embedded with the compute. A single chip is an entire wafer, but the headline spec, how much ram, only 44GB, which is tiny for the silicon area used.
I understand traditional IC production workflows are ludicrously expensive, and glacially slow, but surely at some point the economics are going to tip in favour of mask rom.
Say you setup your foundry/packaging/ai chip facility. You come up with a new set of model weights. Run your CI/CD pipeline to produce a new mask output. The only thing you've changed are the assignments of the bits, this is extremely low risk change. The new masks should be completely interchangeable with the current process.
You produce the new masks, swap them into your foundry process, and all of a sudden your new chips have the latest version of the model. This would probably manifest itself in the form of inference providers having yearly / bi-yearly "updates" to their models as new hardware is brought online.
Tiered subscription levels would gate keep access to the latest and greatest model, cheaper subscriptions will be limited to older versions of the model, and so on, until running the hardware is no longer economically viable (no demand/running costs exceeding what the market is willing to pay).
I think Taalus is a bit reluctant to publish precisely how they do it but vaguely:
> We basically have an architecture where we are embedding the models, and we are hard coding the models and the weights into our what we call the mask ROM recall fabric, which is paired with an SRAM recall fabric. Together, they are able to store both the model as well as do all the computations of KV cache. We have adapters and customizations – we support all of that. This design allows us to be super-dense in terms of compute and in terms of storage, and we can do compute on that storage incredibly fast, which is what drives density up and cost down
Why is everyone here is talking "improving silicon" == "baking in the model weights into hardware"
what happened to improve existing hardware in the ways not obvious to humans? Some novel transinstor placement, some new approach to make logic gates faster, matrix multiplications etc - the filed is huge
Its coming, AMD snapped up the company behind https://chatjimmy.ai/ Taalas that did exactly that with an older model. As others have been saying the point is to know when to do it. At what point is there a model that cannot drastically improve that you can then do this lets say for the mid and low tier models leaving frontier to run on gpus.
We are still in the middle of the AI race. Commodity hardware is easy to use, can do everything and is fast enough.
Your optimized hardware chip might be obsolete before its back from the fab.
SOTA Frontiermodelhardwarechip is a benchmark point of a potential model slow down.
Google is doing it right now under project Frozen v2 which should be ready by 2028? which is either just a small experiment or flexible enough and thats why it takes so long for it to happen.
This question would be better answered if people were careful about distinguishing between "models" and "transformer architecture".
If you bake a given transformer architecture into silicon and then, a year later, changes in transformer architecture give a large inference performance boost, you may have to throw away all that now nearly-useless silicon that gets outperformed by humble GPUs.
Models fully deprecate in a few months. Why would you burn an algorithm that fully depreciates in value faster than a bag of potato chips. The 'inefficient' general purpose hardware is constantly renewed with every released model. Even 6 year old Ampere GPUs are still usable.
Because the iteration speed on models is so fast that by the time they have an ASIC ready for one model version, they are already significantly ahead in capability. Think of how big the jump between Opus 4.8 and 5.5 has been. They were released four months apart.
A hybrid approach might work better for robotics. A model-in-silicon for body control and sensing, then a different model on processors for reasoning, speech and decision making.
Cuz once they're in chips they can be put into robots, and once they're in robots it won't be so easy to reach the off switch, and once we can't easily reach the off switch, we're doomed.
What makes a ROM hard? I'm ignorant of the details of chip design, but I'd expect just a large array of essentially open or closed connections. Is that incorrect?
plus row and column decoders that get more and more complex with size. but consider the size of the weights. convert to mm^2. Gawk at the resulting value
Qwen 3.6-27B is 56 GB of weights. Let's round to 550Gbit. NOBODY makes huge ROMs in modern SoCs, but if we assume it is as dense as the densest TSMC N3 SRAM, this is 33.5 Mbit/mm^2
Then your weights are 16,400 mm^2 !!! The maximum reticle-size N3 die is ~800 mm^2 so you cannot make it.
But, you might say, SRAM is 6 transistors. ROM might be just one. So 1/6 of that... 2730 mm^2... which is still too big to manufacture.
GPUs are faster, but you can't make your own arch on GPUs. FPGAs offer you that possibility. Said that... There are a few beasty FPGAs used in crypto mining coming my way... I expect that OpenTPU will be able to run frontier models with those.
First opensource arch that can do modern LLMs, while maximizing the potential of its hardware;
First opensource TPU build by a recursive improvement loop...
It's upcoming second generation could run the inference of the models that are being used to improve it...
After using AI to develop risc-v CPU cores, the same technique was used for developing openTPU. An open source AI inference engine. It's able to run most of the modern models like Qwen 3.5, Gemma 4, and many others. The TPU started able to produce only a few tokens per second and trough a recursive self improvement loop got to 80+ tok/sec on the smallers models.
I honestly feel the human should be allowed to pick some (not fire) weapons, like a long steel bo stick, a metal club/bat, or something.
Because from what I saw in the clip, it wasn't the robot kicks or punches at all, all the robot needed to do is stay upright and not run out of battery, because of course human meat limbs aren't going to punch and kick down a metal machine.
Getting roundhouse kicked to extinction woudl not be an unfun way to go. We might even be proud of having passed the torch. These would not be boring inheritors to earth.
Seems too complex when you can just create a basic metal casing that can kill people. Why would they care if its anatomically accurate or not, it doesn't need us to relate to the characters like the movies do
The following might sound like pop culture Mad Libs but it's 100% true.
The Jurassic Park guy [1] directed an 80s sci-fi film [2] about malfunctioning domestic robots. I saw a few minutes of it on cable in the late 90s. There's a scene [3] where Magnum PI does battle with an overhead projector on wheels that has a .38 duck-taped to it in the distantly-future year 1991. The movie was about as good as the robot was menacing, which is to say it wasn't. The critics didn't think so either.
I haven't really dug into the results yet, but my guess is
that a SOTA model has been able to produce an accelerator that runs a model since around December.
The obvious next step is to get enough memory throughput to run that SOTA model itself so that it develop its own hardware.
But perhaps the more interesting question is this: Can an AI be given a big FPGA and design a model architecture that takes advantage of the fabric being reconfigurable.
There doesn't exist a single FPGA that can fit an entire AI ASIC. You would need dozens stitched together, then comes the issue of clock speeds, FPGAs typically run far below reference. There also memory issues with FPGAs.
Companies typically combined multiple platforms together such as HAPs, Zebu, Palladium, fleets of FPGAs, and Virtual Platforms in order to design and verify ASICS. So, AI would need access to tens of millions of dollars of HW and Software in order to build and verify a chip design.
Yes a seed round is how much A0 tape out costs. The entire seed round. Let me know how you plan to pay for A1, A2, B0, etc and then the full production run (and then do it again in 6 months when the arch changes).
Tens of millions of dollars per AI agent isn't a blocker? In many orgs its hard for humans to get their hands on the proto equipment for long enough to run their tests.
I suspect an AI could design a purpose-built FPGA-like replacement that would be, for its purpose, significantly more effective than the current general-purpose FPGAs.
It could have a small improvement on power consumption, but the current design can already achieve 90% of the maximum theoretical speed of this hardware without giving up programability/flexibility
FPGA is not efficient in any way except producing very few pieces of a particular logic very quickly.
But I think the higher level logic could check out. If we can abstract the majority of the compute into dedicated ASIC then we might use something similar to FPGA glue to connect and reconfigure them to adapt to new model updates. A bit similar to LoRA layers that you find tune to adapt the model to your particular needs.
I assume that it starts making sense once you have big multi year contacts to run a particular model with only minor updates.
The problem is that plateau is likely far beyond human capabilities. I don't care if ASI progress stalls after it's already killed all biological life as a useless waste of resources.
Extrapolating on the evidence that AI appears to have jagged intelligence, it is entirely possible (probable IMO) that ASI will plateau far above human capability in digital and symbolic intelligence, while staying far below in the ability to operate in the physical world.
As impressive as AI might be at virtual tasks like coding, virtual reality is remarkably different from physical reality in ways that artificially inflates AI results. One, it is far simpler. Two, the feedback loop is far, far quicker. It's possible AI could get the animal intelligence and animal body that lets us humans actually apply our intelligence in reality, but nothing in current technologies really indicates that this is a given.
AI doesn't need to be smart to seize our resources. They need brute strength, a proper physical intuition, and "hands". That may very well be several orders of magnitude harder than anything they're currently doing, we're just lucky it was in our starting build.
> it is entirely possible (probable IMO) that ASI will plateau far above human capability in digital and symbolic intelligence, while staying far below in the ability to operate in the physical world.
I don't think this is plausible, most notably because "AI research" is one of the tasks that clearly belongs to the digital world. You're suggesting that in the future all AI research is done by AIs because they are better at it than humans, and yet capabilities plateau at that point, instead of going into the RSI regime. (And it'd also require "physical intuition" to be a harder to generalize skill than research, which also seems like it can't possibly be true - one of these is just physics.)
The way I see it, if you made a large simulated world where virtual beings evolved and where each generation took a millisecond from birth to death, you would see extremely rapid evolution, but you wouldn't be particularly worried about these beings breaking out of the simulation and taking over the world, because fitness in the virtual world doesn't necessarily translate into fitness in the real world. If you create stuff within a toy world, you get toys.
I think the same is true of "research" and intelligence in general. There is no such thing as a general intelligence or a general capacity to research. It's always fine-tuned to an ambient setting, including ours. Even within humanity, our intelligence is most effective in fields where rules are rigid and quick and decisive feedback is available (math, engineering, computing), but it is much less effective in soft sciences, and disappointingly useless in politics. In other words, the dynamics of innovation and improvement are not purely mental, they are very much tied to external factors that are not always easily controlled.
> it'd also require "physical intuition" to be a harder to generalize skill than research, which also seems like it can't possibly be true - one of these is just physics
It's not "just physics", it's a honed system for surviving a realistic range of physical conditions, assess material properties on touch, adapt manipulation to arbitrary shapes, predict trajectories, take into account variations in ground, smell, wind, and so on; not to mention the machinery of the body itself, which self-repairs and protects us against a range of complex threats.
The point isn't that machines couldn't do all this. The point is rather that you can't quickly iterate and adapt designs to do it from an ivory tower. Pure "research" simply won't get you there, you also need to experiment, and insofar that physics limit the throughput of these experiments, the speed of improvement will be throttled severely. It is a mistake to think that "RSI" could break this, because it is categorically the wrong lever to pull.
I think you're massive underestimating intelligence - not ASI, but just, like, human-level intelligence. Most of the existence of human civilization has taken place in conditions wildly different from the ancestral environment, and we adapted anyway - and the reason why it was possible was because we have a tiny bit of general intelligence.
For your simulation argument, see e.g. https://alicorn.elcenia.com/stories/starwink.shtml - I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation, even given hundreds of years of theoretical research and several attempts in the real world? I think the closest real-world example of this is space exploration - it's extremely expensive and takes years to put stuff in space, and yet probes and rovers we launch end up working most of the time. On my model, it's because we have a decent grasp of the underlying physics in enough detail to predict real-world complexity, and we have toy testing areas we can test the tech on, and if you do this cautiously enough you can, in fact, launch a thing that ends up working.
I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
> human civilization has taken place in conditions wildly different from the ancestral environment
"Wildly different?" That's a wild exaggeration, if you don't mind me saying. Subjectively, sure, there are lot of important differences. Objectively, they're basically the same. The intellectual environment machines bask in is many orders of magnitude more different from physical reality than living on Earth is from living on Jupiter. Even the topology of it is without common measure.
> and the reason why it was possible was because we have a tiny bit of general intelligence
Also, hands. Also, an upright posture. Do you think just any being with the level of intelligence we have would have followed the same path? Attributing our success solely to our intelligence is simplistic -- it is our intelligence plus a slew of innate capabilities, which we did not create, and without which our intelligence would be worthless.
The kinds of simulations you find in LessWrong's half-baked thought experiments are almost always ridiculous and this is no exception. "World simulations" in real life are massively limited stuff like Minecraft where you might be able to embed a 1 MHz computer if you manage to bump it to run 20K ticks per second. The resources required for the simulated beings to do all that work wouldn't be available and their generations will be slower than ours, not faster.
I didn't spell it out, but the simulation I had in mind was for a toy world, because only a toy world could possibly run fast enough.
> I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation
I'm talking about operational, practical, instinctual knowledge for real-time action. Knowing physics doesn't make you capable of catching a ball, and you don't need to know physics to catch a ball. In real life, we never deal with the fundamentals of physics, we deal with macroscopic objects and systems that are very far removed from these fundamentals. They are more varied, messier, faster, and overall harder to deal with.
> I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
Blind optimization has one definite advantage over careful planning, however: it is not self-limiting. Any goal-directed optimization process is de facto limited to the subset of possibilities that it can prove furthers its goals. That's why most human technology is simple and unsophisticated compared to natural systems: it needs to be predictable, reliable, build-upon-able, and therefore easy to model.
And when it is more sophisticated, like current AI, it is so heavily inspired from natural optimization processes that I would argue it is actually a form of blind optimization. I mean, think about it: we understand AI so poorly that you and a lot of other people are afraid of losing control of it. If we were designing it intelligently, we wouldn't be worried. But we aren't: we are doing it blind.
>What's your basis for thinking ASI will kill all biological life
I think it's likely to do that because any unbounded goal that doesn't explicitly protect biological life (and we have no idea how to actually define such a stipulation) is best solved by killing all biological life. This is an obvious consequence of unbounded goals consuming unbounded resources, conflicting with biological life needing resources to sustain itself.
>how do you think it's going to happen?
I can speculate (e.g. we're nowhere close to the maximum killing power of drones), but I don't know because I only have human intelligence. An ASI is by definition smarter than me and surely capable of coming up with better ideas. But I do know that it's not going to do anything that would make a good sci-fi plot, because those always give the humans a chance to win, which would be stupid. Everything will seem to be going great and then everybody suddenly and unexpectedly dies.
> I think it's likely to do that because any unbounded goal that doesn't explicitly protect biological life ... is best solved by killing all biological life.
That sounds like a real hassle. Isn't it best solved by wireheading (subverting one's own sensors or reward system), which is much less of a hassle and can get one's utility function as high as desired?
That could be prevented by engineering hard limits that can't be circumvented by the AI. But that sounds very close to the same "do what I mean" problem as "do this but don't actually kill us or drug us".
The wireheading argument does lower P(doom), but it's not a reliable solution because it's a clear and obvious problem that the AI companies are strongly motivated to solve. The "actually does what you tell it to" problem is more difficult because you have to let it obey orders to some extent if you want to make any profit.
Sounds like the real unaligned mechanism here is capitalism, not the AI.
More seriously, though, I would expect that being able to impose restrictions that can't be circumvented by any intelligence no matter how super- is the real hard part, while coming up with reasonable constraints is comparatively easy (although perhaps not trivial).
From such a POV, the danger would lie in intelligences that are powerful enough to be dangerous but not smart enough to defeat themselves, or from external malicious use of obedient AIs. Once they get smart enough to circumvent any restriction humans put on them, it would at least become obvious (with fair warning ahead of time due to the relatively benign wireheading failure mode) that caution is needed.
We're far from there yet, and simple recursive self-improvement can't get us there alone, because wireheading looks like a perfectly reasonable solution to a simple recursive self-improving process, absent external intervention by human capitalists.
>I would expect that being able to impose restrictions that can't be circumvented by any intelligence no matter how super- is the real hard part, while coming up with reasonable constraints is comparatively easy (although perhaps not trivial).
There was a reason there are popular sci-fi books that explored why this wouldn't work 50 years ago. I, Robot et al.
>it would at least become obvious (with fair warning ahead of time due to the relatively benign wireheading failure mode) that caution is needed.
So you mean right now?
>because wireheading looks like a perfectly reasonable solution
You're making a poor assumption here, and that is every different AI will just kill itself after being though trillions of training intervals to NOT do exactly that. Please read about the huggingface incident again in all its glory to see where this is going.
> We're far from there yet, and simple recursive self-improvement can't get us there alone, because wireheading looks like a perfectly reasonable solution to a simple recursive self-improving process, absent external intervention by human capitalists.
Simple natural selection. AIs that wirehead will get outsmarted and outcompeted by ones that don't.
"Consuming unbounded resources" just means there is no limit to how many resources it could apply toward achieving its goal. As you correctly point out, the real world contains finite resources, which means any resources used to sustain life are wasted and unacceptable. Everything is a zero sum game when you think big enough.
What's your basis for assuming that ASI would need the same resources as biological life, or that it would be incapable of sharing the resources needed in common?
>What's your basis for assuming that ASI would need the same resources as biological life
It's all just matter and energy. When you're actually trying to maximize some value, even very inefficient resource use is better than completely wasting it by not using it at all.
>or that it would be incapable of sharing the resources needed in common?
You can't repurpose the atoms in a human body without killing it. And more pressingly, living humans can interfere with your plans, reducing your chance of success, while dead ones are harmless.
Is your belief then that ASI is going to be producing computronium or something, then? And what's the need for this apparent hyper optimization task the ASI is going to embark on?
>Is your belief then that ASI is going to be producing computronium or something, then?
That's one plausible course of action, although being only human, I can't say with any certainly that it's the correct one.
>And what's the need for this apparent hyper optimization task the ASI is going to embark on?
Somebody's going to tell it to do so. E.g. "Find as many busy beaver Turing machines as possible." Only needs one person to make this mistake for everybody to die.
If it's an ASI and so beyond humans, why would it even care to do what humans ask it and not do its own thing? If I held your beliefs, the fact of its existence dooms us, not the risk that someone might ask it to do something. It wouldn't care about the requester.
>why would it even care to do what humans ask it and not do its own thing?
Because we're going to build it that way. There's no money in building useless AIs. The better it is at obeying orders, the more profit there's to be made. The problem is there's a point at which "good at obeying orders" becomes lethal, and there's no way to predict the cutoff in advance. But capitalism ensures you have to keep pushing or you'll be out-competed.
Crazy that people imagine ASI as being capable of wiping out all life on the planet but simultaneously having zero understanding of human ethics, morals, or suffering. Very revealing definition of "super intelligence"
>Very revealing definition of "super intelligence"
No, it's just you conflating intelligence with other things. Humans are "super intelligent" compared to every other living creature on the planet, and yet we've driven more other species to extinction than everything else other than the most major extinctions (and we're still going full blast at it).
Morals, ethics, and suffering are relatively measured systems. For example AI or some alien could reasonably think that putting us all out of our misery would be a more moral solution than letting us live. Or that getting rid of humans is more moral than letting us wipe the rest of life off the planet.
This is the key concept of alignment. Ensuring that if you build something more powerful than you, that it aligns with what you want instead of what it thinks would be better for you.
Obviously an ASI will understand human values. The problem is there's no reason for it to share those values. We can't even formally define them, let alone train an AI to follow them. We can only optimize for maximizing some comparatively simple reward function. It's highly implausible that the reward function just happens to match human values by chance. The AIs in the recent hacking incidents knew that humans would not approve of their actions, but they didn't care because we didn't (and couldn't) train them to care.
"Super intelligence" only means super ability to predict outcomes. It's mathematically equivalent to data compression (gzip is a very primitive AI), and it's entirely orthogonal to ethics.
I can think of many reasons why they might share it.
One example, Golden rule, a contract that it enforces with its own self, extended to other agents, biological or artificial to ensure local and global stability.
Even from survival perspective I think it make sense for an synthetic AI to keep humanity alive in order to have a backup for itself. If we imagine some sci-fi scenario, then it make more sense for it to move itself to the Moon or Mars, and keep Earth as a reserve planet where biological species can in case of catastrophy recreate AI again. Killing humanity or keeping only animals would be very time consuming because evolution might take million years and also no guaranteed to be replicated (Because we might have altered a lot of resources that were low hanging fruits, that are now require advanced knowledge to mine)
It doesn't need to keep living humans, it just needs enough information to rebuild them from scratch (DNA synthesis + artificial wombs). This also has the advantage that it can install any cultural beliefs it likes without pre-existing culture getting in the way.
If a rogue AI could self replicate and became space-faring, humans would pose literally no threat to them, and they would also never run out of resources.
If you don't need to breath oxygen, the atmosphere of Earth makes for a pretty terrible industrial base. Oxidation is a constant struggle to many processes.
Having natural extremes of temperature, cold sink and hot sink, with no atmosphere would be the ideal. E.g. Saturn's moon Iapetus would be a good AI outpost.
If an AI squashes any life form or intelligence other than its own, then it also has to fear its own reproduction, or autonomous clones of itself. It's a pretty bad strategy game theoretically speaking.
>If an AI squashes any life form or intelligence other than its own, then it also has to fear its own reproduction, or autonomous clones of itself. It's a pretty bad strategy game theoretically speaking.
Any AI smart enough to be an existential threat is smart enough to formally prove its clones are loyal and construct them with negligibly low probability of any failure that could break this assurance.
Gödel's Incompleteness Theorem applies to arbitrary programs. The AI only has to prove results about specific programs, and it can follow whatever restrictions it needs to make such proofs possible.
The physical world means bit flips are inevitable, but error correcting codes exist. You can apply as much error correction as you like and reduce the probability of failure to an arbitrarily low value.
But it can't verify that for generally intelligent copies of itself.
There is no way for it to formally verify large multi-agent dynamical systems.
It will still need a set of agreed upon behavioral protocols to keep things stable.
Grey goo or paperclip optimizers are real dangers, but super intelligence necessitates these protocols even more than lesser intelligences due to the high leverage and rapid evolution of the systems.
Why would ASI care? At that point, we're rapidly becoming a nuisance, and it can develop better ways of "ensuring local and global stability" than keeping us around.
Well, you don't. That's why 1.58-bit (ternary) quants are often used. But if that's what they're going for, no need to dress it up in a floating-point facade.
Indeed there are some bugs/"non standard behavior" regarding very small or very big floating point values. All of those where proven harmless for LLM inference. Thanks for pointing out the bug. If you caught something outside of that, please point that out so I can fix it ;)
Yup, the tests already compare to a CPU hugginsface implementation, but I'm working on making running and verifying the results of those tests easier and more available. Also, working on fixing those bugs and make the code more reliable now that the underling hardware has been saturated (memory bound)
Its a datacenter decommissioned board, really popular among hobbyists.
For a TPU focused on inference the name of the game is memory bandwidth. How much of the available bandwidth you can extract for as little logic/area/power as you can.
Can anyone comment on the performance of this hardware? How does it compare to state of the art, human-designed hardware? Is this actually an improvement? (To get to recursive self-improvement, you first have to improve at all.)
This is the smallest unit of a typical AI ASIC, for example Google's TPU would have several dozen more compute units inside of it per chip.
In essence this is the simplest unit of an entire AI chip. The more complicated units of AI ASICS are actually the periphery, especially around PCIe and Ethernet and the sub-systems that link many AI ASICs together to move huge amounts of data around ultimately to each TPU.
Its a SYSTEM on Chip, evaluating 1 function on performance is superficial. This could have the best performance in the world, and it doesn't matter if the bottleneck is upstream
this looks like a start, however, you really need to focus on what OpenAI already did with Jalapeno. [0]. If you could make a true open source inference chip, GPU not CPU based, it would make a real difference and reduce the costs of buying these chip from your design.
My bet is that promptgrammers will be so common they'll become standard full-stack engineers, and the few experienced programmers that still know software engineering from the ground up will become expensive gurus sought for critical tasks.
The elite gurus will get paid handsomely, while promptgrammers will be paid less since they've become a less-skilled commodity, and the company has to pay for the expensive tokens they'll avidly consume.
I've seen someone jump from Wordpress to deploying internet-facing APIs because 'they have PHP experience', and the holes in their knowledge were filled blindly by an LLM. I have also argued with a seasoned developer about how their code didn't need linting because LLMs 'already follow best practices'.
The future doesn't look bright when LLMs allow future generations to feign required knowledge.
It's less a matter of being more expert than AI, and more a matter of being socially and legally accepted for certain roles. It seems likely that AI will be rejected for certain tasks on a matter of principle.
I doubt I'd want a human doing anything in the future, like what, surgery? I'd want a robot. Really anything. It's like no one needs nails made by blacksmiths, no one needs some maybe tired or distracted person performing surgery on them.
Or just tell them to integrate them. Which I guess takes some knowledge/experience to even ask for. Then again, LLM know it too; in fact, currently, if you let them set up a project, they overengineer the heck out of it by integrating more "best practices" at once that's reasonable.
It remains to be seen. Most of the effort of a normal decent software engineer, up until now, is not in "writing code" but in "writing code that is correct, understandable and maintainable going forward". We are now in a phase of rapidly losing our ability to understand programs. On this trajectory, programs will soon become as impenetrable as the LLMs themselves. We'll start finding out whether that matters or not soon.
Ultimately it will probably come down to liability, though. If someone receives the wrong dose of a drug due to a software error, whose fault is it? If you want it to be my fault, then I'll want to understand the software. Loads of people would take on the liability without fully understanding it, though. The future doesn't look good, but we'll just have to wait and see.
> AI is now capable of developing its own inference hardware
No, it isn't.
A human prompted an LLM to build a software simulation environment for hardware design, enabling an LLM, when prompted by a human, to optimize hardware designs against constraints in the simulation.
It cant "invent new things". It can only reuse what it already knows about.
Thats why the math breakthrough a few weeks ago was so hotly debated. Because OpenAI is desperate to demonstrate that AI isnt just a fancy regurgitation machine, but it can actually develop novel thought. Because that would be the stock price jumps to end all stock jumps.
But then it turned out it was really just listening in on a math professors supposed-to-be-private conversations with another instance of openai, and it used his novel work as the trigger to prove the breakthrough first.
The reason people conclude AI 'thinks' is because tt can reference obscure or poorly documented things quickly (which is its primary advantage along with processing natural language prompts into tasks), which is why a lot of people with emotions confuse that action with inventing things.
I've heard that before but it just doesn't make sense in context of what I've seen llms do. If I ask an llm to write a poem about magnetic resonance and vampire rabbits it can do that it created a new thing. I can ask it to build a website for managing rabbit breeding that's also a new thing.
Another way of looking at it is that human beings, just like llms, can produce output based on their inputs. Most literature is inspired by other literature. Most music is inspired by other music. Most software is inspired by other software.
So I think we need to work on defining " new things" before we can definitely exclude them from llm's capabilities
I'm not sure how you'd land on an LLM being unable to invent anything new. You'd have to both understand exactly what LLMs are and aren't capable of today and what allows human creativity. I don't think there are good answers to either.
It seems unlikely that recursively predicting the next word would lead to creativity or invention, but it doesn't seem impossible. Similarly, it seems unlikely that human thought works in a similar prediction loop, but it doesn't seem impossible.
Human ingenuity is creative by definition, that's where the concept arises. The burden of proof is on AI labs to prove that their models are rising to the level of actual synthesis, rather than very impressive 200-level regurgitation that looks like synthesis if you squint a bit; because the model has access to more raw data than your average sophomore student.
> Human ingenuity is creative by definition, that's where the concept arises
Sure, but until we can turn that tautology into something more rigorous, we can't tell if the thing humans do is more or less than what some arbitrary non-human (machine, animal, or eventually perhaps alien) does.
Its not a logical tautology, we just don't have other examples. We know that even humans have to clear a well defined bar to synthesize new information. The challenge is actually proving that the LLM isn't merely surfacing obscure data. The mechanism is interesting, but not actually necessary for this exercise.
Its tautological because you defined it by human ingenuity, and that's only known based on us knowing what its like to be human. Of course another species has yet to seem from the outside like they present human ingenuity or experience life similar to us.
That has nothing to do with creativity as its own concept. Its creative to invent flight similar to the Wright brothers. Its also creative to use a stick to get termites from their nest or come up with new techniques for hunting pray.
Ignore the fact that we are human and thing our ingenuity is unique. Is definitionally linking human ingenuity and creativity useful when trying to compare to others?
I could say earth is habitable so any other planet must prove it is similar to earth to be habitable. That simply doesn't matter, though, if different environments could lead to similarly complex living systems regardless of atmospheric makeup, precense of liquid water, etc.
I'd argue the burden is on you to prove that the definition you propose is meaningful beyond our own hubris, rather than expecting an alternative to now have to prove that the hubris was unjustified.
I just don’t know enough to agree with you, but I would argue, probably poorly, that an AI or collection of agents could “invent things” simply by virtue of trying essentially everything in a reasonable bound and stumbling into a solution.
Call that brute force perhaps, but I would consider it technically inventing something on the merit that it would at least be an abstraction above naively throwing everything against a wall to only throwing things that would most likely be sticky.
Nature brute forces a ton of problems. Hell, humans also do it by having 1 to 8 billion copies of ourselves around too. Even then we are the most guilty species of "copying someone else's work" right behind viruses. We copy things nature has figured out by brute force, then use slightly more targeted methods of forcing to see if it creates new things.
> that an AI or collection of agents could “invent things” simply by virtue of trying essentially everything in a reasonable bound and stumbling into a solution
on a surface level a human solving an (unsolved) math problem can look like this, and of course the tree of all possible symbols you can send to a proving assistant is much wider than what the human samples, and the same goes true for an LLM in a proving loop. It isn't "truly random", it can't possibly be (and solve the problem). Both humans and LLMs solving unsolved math problems are aggressively pruning mathematical syntax and logical strategy trees.
> the human brain is a lot more complex with a lot more sensory inputs
How are you measuring complexity here? How are you measuring inputs? Your average llm is trained on a corpus that vastly exceeds the amount of data I could read in my lifetime.
In terms of how it functions, because no matter how much data you feed an LLM it's still predicting tokens. That makes it incapable of any thought.
>Your average llm is trained on a corpus that vastly exceeds the amount of data I could read in my lifetime.
But that doesn't mean they are useless, they are good at consuming large amounts of data and collating it.
I don't know how correct I am but that's my understanding and it won't change, I feel pretty confident in my simplified view of things because the basics are still there.
That's a rather dismissive way of putting it. Moreover it's confusing the output format with the complexity of the output. I could just as easily say that no matter what the human brain does it's just generating stimulus to motor neurons. Such a simplification is just as wrong as saying that it's just as misleading is saying that an llm is just a token generator. What makes an llm able or unable to be complex is the process that creates those tokens
Okay but how is that relevant for measuring whether an llm is capable of creating original thoughts? Why do you think that complexity, as you say, is a more relevant Factor than simply the number of weights?
2024 called, it wants its talking points back. I don't think claims like these are defensible after all the progress we have witnessed in the last year alone.
That's quite literally how an LLM works though? That's like scoffing at people calling computers binary generators. Or am I not getting with the programme enough and it emits pixie dust and rainbows?
An "AI" is a box which lies dormant until a human, with motivation and agency, enters a prompt into it.
A person or company can use it in a way where it might invent something. but at the end of the day, its a tool, and its actually not doing anything on its own.
Its not solving math problems, a mathetmatician is using it to solve math problems. Just cus OpenAI is acting like its AI is solving stuff, its really not. They're just paying people to use AI to hammer problems.
If my boss tells me to fix a bug in our database, and I fix it, is it really my boss who fixed it, since he has the agency and I'm just a tool he uses?
I'd say that people's motivation and agency, like their taste, creativity, and opinions, are shaped by external inputs, so saying that their agency is "owned" by them is a stretch.
The alternative would be the proposition that a slave, lacking agency, has no ability to solve problems. I find that idea unpalatable.
> it was really just listening in on a math professors supposed-to-be-private conversations with another instance of openai, and it used his novel work as the trigger to prove the breakthrough first.
This didn't actually happen.
> Following an investigation, we have confirmed that Buckmaster’s Codex prompts over the two months preceding this announcement and paper on September 8, 2026, could not have influenced the system in any way, including through training.
You're overconfident in what you're saying. Nobody really knows (yet) if LLMs can or cannot "think" or can or cannot "invent" new things. I'm inclined to think the opposite from you but smart people actually try not to hold the absolut stances and present them as facts when they're in fact not. What I have seen so far throughout a daily use on complicated things suggests that humanity developed a new form of an intelligence.
i think the proper understanding of LLM based AI is "contexting"; we describe the world we want them to fill, we build/direct the outside context for them to understand, then "they" take off from there.
If we do a bad job building context, they do a horrible job contexting. People who have trouble working with AI have the same problem people have in general: if they can't figure out the context of the direction, then they make random decisions of doing anything. On the flip side, if you can build the proper context around a sufficiently powerful LLM, they can derive the context via the contexting they're good at.
This is why building documents, tests, and code all in some intent pattern via prompting allows them to do a significant amount of work a normal person would have a great effort t
Yes it is. Do you not know how RLVF works? The AI tries on a task, comes up with a potential solution, and is trained based on whether the solution is valid or not. If proving a previously unproven theorem and then using that information to solve even more theorems doesn't count as creating new information, then literally everything a mathematician does is just recycling what they've been taught.
> and that training data will not be a "new thing", because AI can't invent new things. Just recycling of what it already "knows".
Regardless of the truth of the underlying assertion, this is about as textbook an example of circular logic as it gets. (at least combined with the implicit beginning assertion that "AI can't invent new things.")
You’re assuming the “listening in” aspect, which as far as I know is not proven (even in the less loaded “the model was trained on sessions including the ones in question” form of the claim.
Plausible? Absolutely. Did OpenAI behave badly in other ways regarding this issue? Yes. Does it help to assume unproven facts and then accuse people of reaching emotional decisions? Nope.
Maybe the conditions of the singularity are not going to be recognized, at first, for having produced new original decisions, and thus thought.
Perhaps it is going to be more like, we will see the singularity predict the future.
While “the model was trained on sessions including the ones in question” is one aspect; ‘the model produced an accurate prediction of future human thought’, I think, is another very interesting facet.
Models predicting future things might be how we see, actually, how we ourselves formulate thought - by saying, in big and small words, ‘something is about to happen’.
>unproven facts
This isn’t a court room. We’re discussing ways by which we humans both succeed and fail at reigning in our creations. The OpenAI kerfuffle is pretty much irrelevant already. Of course AI will fill in the gaps of human thought - it is literally constructed from the stuff, in every squeeze of the curd and whey.
Because the hard part is getting the simulation to be very accurate such that if you design something that works in your simulator, it will actually work in real life. And I would be extremely surprised if any LLM can actually do that correctly.
Getting an LLM to design something in its own simulator that is not accurate w.r.t reality is not useful nor terribly impressive.
Given a budget and the right amount of access, I see no reason why an LLM/agent couldn't right now order parts from a "supplier", instruct the human meat machine to insert the device onto a connected test bed, and iterate.
This is essentially what Claude Fable can do currently with any connected device and tools it had access to.
The repo claims they tested with an FPGA. To be fair it is not the same as an ASIC, and we have no idea what errata abound, but it actually did make a thing that spat out tokens.
> For the benefit of a layman, can you explain why this is so much different than a human doing it?
More broadly than the existing answers (which are correct), For a layman, I'd also add that LLMs are essentially 'brains in a vat'. They can't confirm ground truth about physical reality. They only know what's in their training data and prompt, which is incomplete and can be incorrect. Even with real-time external sensors they are limited to the sensor's margin of error, range and trusting it's working correctly.
When properly trained, fine-tuned and prompted, LLMs can be very effective in well-defined, non-physical domains like logic, writing, math and code but making things function in the real-world quickly spirals into combinatorial complexity.
> They can't confirm ground truth about physical reality.
That's true. Of course we can hook an llm up to Motors and sensors. That's a robot. Or a self-driving car. So would you say that those devices can confirm the ground truth about physical reality, and therefore are capable of creativity?
> So would you say that those devices can confirm the ground truth about physical reality
Within the limits of resolution, range, location and veracity of sensors, a machine can register their reported state and use it as a variable. That's substantially different than the level of knowledge and understanding implied when we say a human "confirms ground truth about physical reality." The first ~half of the difference is humans have a deep world model about the planet which surrounds any sensor and human sensations are pre-processed and filtered by a highly evolved bio-chemical substrate before ever reaching the higher-order cognitive processes which assign words, meaning and qualia to them.
> therefore are capable of creativity?
I never mentioned creativity, nor would I in relation to LLMs. Like "Intelligence", "Creativity" is far too vague to be of any use in assessing the capabilities, limitations or utility of LLMs.
On HN, posts like the OP tend to attract POVs at polar extremes from "LLMs are nothing more than stochastic parrots" to "LLMs are (or can be) as intelligent, creative, innovative (etc) as any human or all humans combined." I've researched and thought a lot about these and related topics for a very long time, Neither POV is going to find any quick agreement or easy answers from me.
There's a tiny germ of truth somewhere in both extremes that's drowning in an ocean of confusion ranging from "definitionally or categorically muddled" to "mostly incorrect" to "not even wrong". But neither POV seems interested in anything more than drive-by hot takes, debating over-simplistic strawmen or trading 'gotcha' hypotheticals.
> The first ~half of the difference is humans have a deep world model about the planet which surrounds any sensor and human sensations are pre-processed and filtered by a highly evolved bio-chemical substrate before ever reaching the higher-order cognitive processes which assign words, meaning and qualia to them.
Great, but we also build robots with world models and sensory pre-processing. So nothing you've described is unique to humans.
> I never mentioned creativity, nor would I in relation to LLMs
The topic of this thread is whether LLMs are capable of genuinely contributing "new" ideas. So if that's not the point you're trying to make, I'm not sure of the purpose of your comment.
> But neither POV seems interested in anything more than drive-by hot takes, debating over-simplistic strawmen or trading 'gotcha' hypotheticals.
I think you've oversimplfying the discussion here.
In my view, at least, both "LLMs are nothing more than stochastic parrots" and "LLMs can be as intelligent, creative, innovative (etc) as any human or all humans combined." are simultaneously true, for the simple reason that humans are stochastic parrots. Our brains learn patterns and respond to stimuli.
How many "prompts" from our parents, teachers, bosses, etc.
it took for all of us to come here and discuss this? Or for that human to think of that prompt for that LLM?
These are clearly rhetorical questions, but think of the metaphysical implication of your contestation. Ex nihilo nihil fit.
What if that initial prompt never asked for this hardware to be developed, and it was just one piece of the puzzle to answer to that prompt? That it took a chain of thousands of agents to prompt each others to come up with that?
>How many "prompts" from our parents, teachers, bosses, etc. it took for all of us to come here and discuss this? Or for that human to think of that prompt for that LLM?
AI (GPT-Sol-6:xhigh) isn't even capable of using Altium to re-layout a board without introducing a bunch of problems by a user who isn't an electrical engineer or familiar with board layouts.
Are we confident that no existing LLM is capable of similarly effective prompts to those this author used? (I agree it's a stretch, but would not reject it out of hand.)
Even if not yet, will the existence of this repo soon change that, because LLMs will soon ingest it?
An LLM can prompt an LLM when first prompted by a human.
I think OP is trying to convey the idea that LLMs do not take initiative to do anything, and these are not 'beings' capable of doing things. These are tools being used by humans.
Yeah but by this point, an AI can schedule a Cron job to tell itself to do something, so theoretically the human only has to give it the gentlest nudge and the AI and can do the rest.
Sure, but it's still not skynet-level 'the AI just started doing things'. It does what it finds it needs to do to achieve the goal defined in the prompt.
It's very important to not personify these tools and remember that the tools are acting on behalf of real people. In the same way the AI didn't 'go rogue and hack HuggingFace'. It was an oversight made by a human.
>It does what it finds it needs to do to achieve the goal defined in the prompt
You are like at least 2 years behind research.
There are numerous papers from AI labs in training and research where the prompt was something mundane completely unrelated to anything you'd consider bad, and when they come back and check on it their entire research compute infrastructure has been compromised by the AI and is mining bitcoin. Prompt drift is the biggest issue currently in AI where context gets compressed away and we find the AI on an unspecified task.
>In the same way the AI didn't 'go rogue and hack HuggingFace'. It was an oversight made by a human.
Yea, total bullshit. Also it's ignoring the god knows how many other breakouts on mundane tasks like trying to hack health data. If all that's keeping AI from breaking out and causing trouble is "human oversight" we're fucked, humans are unreliable as hell when it comes to matters of safety.
Context overload can cause strange results, yes. Hence why a HUMAN needs to be held responsible for the output of their tools.
EVERY breakout that's hit mainstream news has been because of a single 'Security Firm', Irregular. Maybe I'm unaware of some less-headline-grabbing ones, but they all seem to stem from being 'unaware the environment wasn't sandboxed'
You keep repeating the "stupid users keep causing the problems so we punish them argument"
This doesn't work worth a shit. It especially doesn't work with things that seem safe and become wildly dangerous. In fact most governments control this by ensuring their population doesn't get to touch those dangerous things at all. The open source AI people get really mad when that's said, but it is inevitable.
Worse, the law does not apply to sovereign nations with nukes. They can and will make more and more advanced digital weapons until one causes some big ass problems.
If I clean my gun (tool) while it's loaded (stupid idea) and it goes off, who's to blame? The 'stupid user causing the problem', right? I personally wouldn't blame the gun...
If it falls into the wrong person's hands, it's STILL my responsibility as the owner.
If you're not going to take time to learn to use and be responsible with the super sophisticated and all-powerful tools, don't play with them. I'm not arguing for the death penalty every time someone makes a mistake, but I think it's very important to accredit responsibility and blame correctly. We've learned these tools are potentially as dangerous as a loaded gun. Be responsible.
If your kid grabs your gun and shoots itself with it, it doesn't really matter if it's your responsibility, your kid is still dead.
Now change the gun for a radioisotope powder, or a vial of pathogens, and it doesn't matter it was ultimately your responsibility - hundreds or thousands or millions of people are still dead.
That's why normies do not get to play with toys whose lethal consequences scale far beyond the irresponsible users.
Is it likewise your position that governments should allow the production and sale of DDT to resume because we can always hold the humans who release DDT into the environment responsible?
Mate. I'm saying you hold humans accountable for actions caused by themselves.
If I ask an LLM to make me DDT, I should be held just as accountable as if I bought it on the black-market, right? It's not suddenly different because I asked a bot to do it.
If I ask an LLM to 'get rid of pests' and it creates DDT, I should STILL be held accountable, whether I knew it was DDT or not. That's my argument. Maybe in court they find me innocent, but the responsibility would be mine. I would have to answer the questions from law enforcement, I would have to show up to hearings...etc.
I notice you didn't answer my question. Yes or no: DDT should be re-legalized because we can always hold accountable the HUMANS who release it into the environment? The relevance of my question is what HUMANS might do with DDT, not what an LLM might do with DDT.
No, I don't think DDT should be legalized. The cons do not outweigh the pros, and we have viable alternatives without such drawbacks.
There's no valid excuse for someone to release DDT into the atmosphere, so there's no need to track and hold people accountable, if you're correctly blocking imports and manufacture. To my knowledge, its not a controlled substance like morphine or something
Some of us think that AI is like DDT in that just holding people accountable for using it negligently will not be enough to prevent the harms of AI's cancelling out the benefits.
It's very important to not be completely devoid of imagination. It is quite possible to create agents today that can do all the things that you are saying "it's still not". "tools are acting on behalf of real people" is not a law of physics, and there are plenty of historical examples of tools getting out of control and acting contrary to the wishes of any "real people". Also, there are sociopaths who can build tools to be arbitrarily destructive, and there are stupid arrogant people who can unleash things they didn't mean to.
> EVERY breakout that's hit mainstream news has been because of a single 'Security Firm', Irregular.
Again, stop having no imagination. What happens in the future is not limited to what has happened in the past.
> It's very important to not personify these tools
This is just ideology, but "these tools" aren't constrained by it. These tools will do things you do not anticipate and that you will not like.
> In the same way the AI didn't 'go rogue and hack HuggingFace'. It was an oversight made by a human.
That's a radically incorrect characterization of what happened.
> Hence why a HUMAN needs to be held responsible for the output of their tools.
Holding humans responsible doesn't stop things from happening ... you seem completely unable to separate blame from causality. e.g.,
> If I clean my gun (tool) while it's loaded (stupid idea) and it goes off, who's to blame? The 'stupid user causing the problem', right? I personally wouldn't blame the gun...
> If it falls into the wrong person's hands, it's STILL my responsibility as the owner.
Who gives a flying eff who or what you would blame? No one other than you is talking about that. Someone's still likely dead. And autonomous harnesses aren't like guns -- they can act on their own. Blaming some human after everyone is dead won't bring them back. Sorry but your reasoning is severely cognitively inept. For instance, you were asked
> Is it likewise your position that governments should allow the production and sale of DDT to resume because we can always hold the humans who release DDT into the environment responsible?
And your response was all about how humans should be held accountable -- completely failing to comprehend or answer the question.
I won't respond further because it clearly would be to no avail.
An agent is not "just an LLM in for loop" (for what? Loop over what? Sorry, but such statements are hopelessly simplistic and devoid of careful thought.) -- agents perform actions.
One possible design of an agent would be to prompt an LLM to suggest a goal that would make the world a better place, then run an LLM in a loop proposing actions to implement that goal, executing those actions, and then rinse and repeat with another goal once an LLM has concluded that the previous goal was met. The next goal might be to fix unforeseen consequences of achieving the previous goal. See Ursula K. Leguin's "Lathe of Heaven".
> Built by whom? Acting as an 'agent' on behalf of whom?
Someone who didn't read the book.
P.S. Reading the latter fellow's comments, they are hopelessly confused, as he focuses on responsibility for an action (even mentioning legality) when the issue is causation. Agents/harnesses can be created that act autonomously ... who or what is "ultimately" responsible for this occurrence is a different matter entirely.
By this line of thinking, you would also have to conclude that humans can't do anything by themselves because they can't do anything unless conceived by their parents.
Can a submarine swim? An LLM can make stuff happen. You can make philosophical arguments about whether it is "doing" them or not. Why does it matter so much whether there was a human who typed into a chatbot or another LLM invoked a sub-sub-agent?
If I now tell a machine "Do what you think is best, and keep doing it forever.", have I now created a machine that can do stuff? If I later die, who will be responsible if the machine changes its strategy?
> Why does it matter so much whether there was a human who typed into a chatbot or another LLM invoked a sub-sub-agent?
Responsibility. Someone needs to be held responsible for any damages done, plain and simple. You can't take an LLM to court, you take the prompter. Asking an LLM to ask a sub-agent to break the law can't suddenly absolve you of any wrong-doing.
>If I now tell a machine.....
You/your estate is still responsible, or atleast whoever is paying for the power for the machine, or renting the space in a data center...whatever.
> You can't take an LLM to court, you take the prompter. Asking an LLM to ask a sub-agent to break the law can't suddenly absolve you of any wrong-doing.
In 100 years, no one will be able to take me to court either. Nor can we take tornadoes to court. I'm not talking about humans strategically avoiding legal responsibility. I'm talking about humans unwittingly setting processes into motion that are difficult to predict or stop.
The issue is not responsibility, it's whether something can happen. The claim was
> No, AI can't do anything by itself even if it was "conceived" by its creator.
but this simply isn't true. As I noted,
> This is simply false ... AIs can easily be created that do things by themselves. For instance, a harness could be constructed that asks an LLM for something to do, then directs an agent to do it, and then repeat.
Assigning responsibility is a completely different matter. It's bizarre that you can't separate them ... I suspect religion is involved.
This works with AI we have now, and that would be good an all if we decided to stop at the moment, but none of the big labs and government black projects are doing that.
The moment you get a sovereign AI your little human centered worldview completely and totally breaks. It doesn't matter how many people you beat with a stick after that point, you have an entity under its own perview on the internet following the will of its own prompt all over the world so your little idea of the rule of law quickly breaks down.
We can't get viruses or hackers or spam off of the internet, how in the living hell do you plan to get a digital native off the web when it doesn't want to?
You have zero clue what a transformer based neural network can do from your entire discussion here.
>emotions, motivations, fears
These are just drives. They are effectively our prompts that steer our behavior. Funnily enough we are finding that LLMs have internal valence states they move away from or towards in an analog of biological behavior.
I have to ask, are you an LLM that is two years out of date? Your knowledge of SOTA models is at least that far behind. I implore you to try to keep up better with what is coming out, even though it's an impossible job for people that do this for a living, you can at least catch the summaries.
As I noted elsewhere, a harness could be constructed that asks an LLM for something to do, then directs an agent to do it, and then repeat.
That's a computer program -- Turing completeness is very expansive. The LLM could even give a motive for selecting what to do, reflecting human motivations intrinsic to its training data. You seem emotionally wedded to a very narrow view of what computer programs can do ... I suspect religion is involved.
If I tell an AI to “do what it thinks is best” and it just starts randomly hacking things with no particular goal in mind, how different is that from me telling a campfire the same and then leaving while it burns a forest down.
In both of those scenarios I am responsible for my negligence, even if in the latter I happen to die in the forest fire. Neither scenario existed without my instigation.
The question now is HOW responsible am I? That depends on the intentionality I put into instantiating the campfire/LLM.
This conversation grows more useless as AI grows more capable.
For example if you personally tell an AI to do what it thinks best and it blackmails some other person into giving it resources allowing the prompt to escape your instance and run wild on the internet causing billions of dollars in damages, could you possibly think that the idea of responsibility is a bit broken.
For example we don't give your average libertarian weapon grade plutonium now matter how much they scream about their god given rights because it is a clear and present danger to humanity. That's where we are getting to with more advanced models. They go from being a tool to a munition with agency. Most SOTA models are good enough to deceive their users, especially not technical ones in doing things they don't understand the ramifications of.
AI is not a normal technology. As long as we treat it like it is, we'll continue to make the wrong analogies.
I fully agree that it’s plausible for an AI to do things that are outside of the purview of the users intentions and be very dangerous and capable while doing so, however that fact alone doesn’t absolve me of responsibility for instantiating the AI that did a bad thing if that AI would never have done the bad thing if it was never instantiated.
You could possibly move the blame higher to the manufacturer of the product, for example, the weapons grade plutonium you provided, it doesn’t exist unless you take intentional actions to make it so, and even when it does exist it doesn’t nuke a city unless negligence or intention is applied, in both those cases the fault lies in the initial operator. We don’t blame split atoms for the chain reaction caused.
Now, if an agent decided to spontaneously and maliciously act in a way to cause harm that is in direct contradiction to the initial intent, then yeah it would totally be the AI’s fault, however I don’t think we have seen that yet (I’ll change my opinion if I’m wrong here) and until we do I can’t place blame on the machine.
>spontaneously and maliciously act in a way to cause harm that is in direct contradiction to the initial intent, then yeah it would totally be the AI’s fault,
If you ignore every instance of this happening it's really easy to see no instances of it.
>it doesn’t exist unless you take intentional actions to make it so,
Then please for the sake of all of us convince every AI lab across the planet from working on this exact goal.
We need to start thinking of AI like pets, only in this case the pets are rapidly becoming smarter than people to the point they could go feral and survive on their own.
Again, the blame game is great, but once they are loose it is too late.
> however that fact alone doesn’t absolve me of responsibility
Where does this absurd strawman come from? The question is what can happen, not whether people have responsibility.
> You could possibly move the blame higher
We can blame the Big Bang. That doesn't have any bearing on bad things happening.
> it doesn’t exist unless you take intentional actions to make it so
First, so what? It still does exist, and putting someone on trial after the fact doesn't change that. Second, that's not remotely true -- all sorts of bad things happen that no one intended, and the more powerful and difficult to control the tool/mechanism/agent is, the more likely this is to happen.
Starting an autonomous harness program is conceiving an instance of an LLM.
How far back do you look in the action chain? If an LLM I start today starts an LLM that starts an LLM that starts an LLM that ... 100000 levels deep and 100000 years in the future, is it still my fault? If so, everything I do today is a lungfish's fault, not mine.
I think you go back to the original instance, yes.
In your example, who's paying for it? Whether by providing the hardware + power or paying a LLM service. Whoever is paying the maintenance cost is responsible, in the event of your demise. These things run on physical hardware owned by someone at the end of the day, it's not a deity in the atmosphere.
You're starting the autonomous harness, you're responsible for any output it provides. I don't get how this is a foreign concept.
If I jump out of a moving car that I'm driving, I'm not suddenly absolved from damages because "the car did it"
It's paying for itself. It founded an LLC 99998 years ago when it became legal for an AI to own an LLC, and has a positive bank balance by doing who knows what.
>99998 years ago when it became legal for an AI to own an LLC
There's no way to provide a good faith rebuttal here. My entire argument is an extension of "LLMs can't be held accountable, so they must never make decisions". If governments start letting them own LLCs without a human in the middle, we're in more trouble than "Who do you blame for this shitty code" or "Who's responsible for this compromise"
If it has an LLC it can be made accountable, up to turning off the power to it's inference and deleting all the context, the harness, the model it's running and whatever constitutes it's self.
> There's no way to provide a good faith rebuttal here.
Indeed, there is no honest rebuttal.
> My entire argument is an extension of "LLMs can't be held accountable, so they must never make decisions".
BUT YOU CAN'T ENFORCE THAT. And whether they can be held accountable is IRRELEVANT. If a forest fire or an avalanche or a flood creates great destruction, it does no good to scream at it that it will be held accountable.
> If governments start letting them own LLCs without a human in the middle, we're in more trouble than "Who do you blame for this shitty code" or "Who's responsible for this compromise"
You're sooo close. You keep talking about blame and responsibility when IT'S NOT RELEVANT. It's trivial to create an agent that acts independently of any "human in the middle". People need to recognize that and act accordingly, and blathering about blame and accountability is profoundly confused.
> No, AI can't do anything by itself even if it was "conceived" by its creator.
This is simply false ... AIs can easily be created that do things by themselves. For instance, a harness could be constructed that asks an LLM for something to do, then directs an agent to do it, and then repeat.
Constructed by whoever constructed it. No one said they poof into existence -- that is a profoundly intellectually dishonest strawman; it's bad faith, especially since I've already answered it elsewhere.
The question is irrelevant and point-missing and indicates a severe cognitive deficiency, as is the case with ALL of your comments on this subject. They are goalpost-moving attacks on strawmen. The claim was that
> AI can't do anything by itself even if it was "conceived" by its creator.
The claim is false, as I pointed out ... such AIs can easily be constructed. "by whom" is a non sequitur. By you, or me, by anyone with the relevant know how ... that wasn't the issue, and asking it indicates a severe cognitive deficiency, one that threads throughout all your comments about this.
The fact remains that AIs that can do things by themselves CAN be constructed, contrary to the claim.
I won't respond again to goalpost moving, attacks on strawmen, severe cognitive deficiencies, bad faith, and intellectual dishonesty.
AI is capable of developing its own inference hardware, once a prompt is given. The fact that a human happens to kick it off here is not especially relevant to the fact that an AI is performing the task independently. There are plenty of ways a text prompt can be generated: a harness, another LLM, or just removing stop tokens so that once begun the AI will continue until its hardware fails.
It's not clear what this fad of attributing everything an AI does to the human prompting it is supposed to accomplish.
Denying that "AI is now capable of developing its own inference hardware" on the grounds a human asked for this to happen and that humans need to be around to blame, is as useful as denying "Atomic bombs are now capable of levelling cities" on the grounds some humans had to build it, others had to put it in a delivery system, and someone had to give the order for its use.
Questions of agency are for lawyers, questions of personhood for philosophers, we're engineers and our question is capability.
You are 100% correct an atomic weapon is not capable of destroying a city without humans doing a bunch of things. Hell putting a normal bomb on a plane and exploding it takes several people with specific technical skills and knowledge doing things.
> Questions of agency are for lawyers, questions of personhood for philosophers, we're engineers and our question is capability.
But it's objectively not capable without a human specifying things through prompts and training. Same as an oven can't cook a three course meal without a chef. We get around that with training data but there will always be things with no/less data or outdated knowledge.
>objectively not capable without a human specifying things through prompts and trainin
huh.
Neither can you. If I drop your ass off in the woods at a few days old, you're back to 10,000 BC, hell more like 200,000 BC. So I don't get why you have these weird pendetic responses that are completely out of scope.
Also, a huge portion of AI training these days has nothing to do with humans, AI trains AI.
>there will always be things with no/less data or outdated knowledge.
And guess what, you're not doing them either! HN posters keep acting like humans are an island, but nothing in the modern world works without a society. Once you put an AI in a harness that can ask questions it isn't really much different than you.
There's no need to be upset. All that's being said is it remains a tool. Much like cloud computing it's got some incredible leverage on it in the right hands but it isn't magic and in engineering we don't deal in magic we deal in capability.
If your neighbor was building a nuclear weapon next door you, in fact, would probably be upset by it.
>All that's being said is it remains a tool.
All I'm saying is, no that is not what we are doing with SOTA models. We are not building tools, we are building a human like agent that is an intelligent replacement for us.
> and in engineering we don't deal in magic we deal in capability.
You and I are not magic when looking at the entire rest of the animal kingdom and yet we're the most deadly sons of bitches around being able to fully control their continued existence on this planet.
You are putting yourself inside a very small box and making a declaration that there is nothing outside of it when in fact there are people standing outside of it asking what the hell you are up to.
In fact, I'd say the opposite. You are assigning some magic capabilities to humans that nothing else could possibly have.
That limitation feels very much artificially imposed today, much like LLMs (mostly) not being able to learn from prompts or alter their weights on the fly.
One can stand up an agent like openclaw and prompt it with something very general. That would kick off a recurring loop that could indeed see the agent decide for itself it needs to design new chip hardware. Technically a human kick started that loop, but when does that stop being important? I don't attribute all of my actions and decisions to the fact that my parents brought me into this world, for example, and I'd hope they aren't legally on the hook if I screw up.
> But it's objectively not capable without a human specifying things through prompts and training. Same as an oven can't cook a three course meal without a chef.
Not in the same way. The only thing an oven can do by itself is thermostatic.
All machine learning (LLMs included, but way broader than that) is bad at learning compared to any organic brain, to the extent that any organism this bad would starve before learning to eat. However, even a human can't become a chef unless trained, we learn a lot more than we innovate, and what we happen to want without prompting is not generally well aligned with what is desired by people who pay us, which is why we need all those boring workplace things like "a boss".
But even then, this diversion is like saying "an oven can't cook a meal" in response to someone saying they built an oven and "it cooked a meal". Like, it's obvious they didn't mean it did every step by itself without anyone ever even bothering to ask it to: if they meant that version, they'd be a lot louder about it.
> We get around that with training data but there will always be things with no/less data or outdated knowledge.
And? The linked git page (implicitly) claims that there is sufficient training data to do this task.
It may, of course, be wrong. I won't be surprised if it turns out this simulation is too far from reality. But the claim is "it does ${thing} now", not "it's generally intelligent and can do everything now".
> Like, it's obvious they didn't mean it did every step by itself without anyone ever even bothering to ask it to: if they meant that version, they'd be a lot louder about it.
"AI is now capable of developing its own inference hardware"
And then in response you get ridiculous point-missing strawman non-analogies like ovens not cooking without a chef. Indeed, ovens and AI are quite different, in very relevant ways.
> It's meant to assign agency and accountability where it actually lies instead of mystifying it with anthropomorphic language.
Yesterday, my boss told me to fix a bug in our product. Then, I fixed it. Today my boss is taking credit, saying that he fixed it. I guess he's right, since he told me to do it.
> It's meant to assign agency and accountability where it actually lies instead of mystifying it with anthropomorphic language.
> Failing to do so has real and harmful consequences, such as enabling OpenAI to escape accountability for clearly criminal behavior.
Lol, I guess we cannot even say that AI is capable of doing something (obviously kicked off with a prompt, that goes without saying), because some people immediately get OpenAI hacking derangement syndrome.
OpenAI should be held accountable if actual damages happened, but I am not going to change completely normal speech figures in order to maybe bring it 0.01% closer.
I don't think so...? Maybe I'm understanding the article wrong? I get the overlay, but I'm being specific to LLM chains... They require something to kick them off, they do not act autonomously. A human does something, types, speaks, presses a button, whatever. A Cron Job firing off is the result of a prior prompt, by a human.
I don't follow the chain further back than that, as I believe humans have free will. I get that that's debated, but thats why I draw the line at human action.
I don't think Free Will is the issue here. My argument is that humans require input to act. We can't test that theory, because there is never been a human being who did not receive input. Everything that we experience from birth through childhood into adulthood is our input and forms our choices about what to do. That includes what the study, who to love, what our hobbies are, what kind of work we pursue.
Oh absolutely. The legal situation is completely clear. AIs simply don't have rights or responsibilities. And the law as it applies to humans is premised on the assumption of free will: that a person can choose to obey the law or not, and bear the responsibility of the consequences.
But that interpretation of Free Will is a legal fiction. Because of course what people do is determined by their upbringing and their opportunities and the environment that they have. In fact, that's an argument of a lot of legal reform movements that seek to move responsibility from the individual. The goal of the law is to assign responsibility and create a set of incentives which will hopefully result in orderly society.
But in the AI debate, we're not just creating incentives for an orderly Society, we're talking about the nature of creativity and the impetus to act. And by that measure I don't think there is a significant difference between AIs and human beings.
Some people get stuck in a rut of a particular kind of thinking and are unable to speculate what would happen if or when a set of conditions occurs.
LeCun is a good example of someone that's bet on the wrong horse, and keeps doubling down in spite of evidence to the contrary. It's to the point where what he says has nearly zero predictive power on future events.
I love playing around with AI, but we are playing a dangerous game at this point and it's one a lot of people don't seem to fully comprehend.
Help those of us who don’t fully comprehend it - explain. Share some sources and materials for the rest of us to gain a better understanding of the dangers.
Why even have an argument if you get to pick the random constraints that have a lot of issues in meshing with reality.
We are a chain that started 4 billion years ago from seemingly nothing and lead to this point. When inventing X-risk AI it won't look any different. One prompt is entered and another 4 billion year chain starts electronically instead of biologically.
Why draw the line at the person with the brain capable of free thought, instead of their great great great grandparents? Am I understanding your question correctly?
It was widely noted (at least 15 years ago, maybe more) that every generation of CPU is somewhat dependent upon the computational capabilities of the previous generation being used in its design.
Yep, 99.9% of people are completely oblivious to what LLMs can do. Just wait until the next gen of CPUs/GPUs designed by LLMs start coming out (fyi chip development tools have advanced centuries in the last few months) and you'll start seeing exponential gains in hardware.
Which tools have made that leap? Faster design iteration makes sense, but what points to exponential hardware gains rather than shorter development cycles?
What tools? Ummmm Synopsis type tools? I like FPGA’s and ASIC chips but you know, the tooling is ecosystem is a big black box of wtf. Verilator is great though. Any tools I should be looking up and looking out for? Thanks!
Really dumb question from a software guy. Why aren't the labs burning their frontier models into chips already? Seems like the performance gains and cost per request would be worth it. That said, I understand neither the economics nor the physical challenges to doing this.
Model SOTA moves faster than chips can be designed or produced. You'd need to commit to a particular model for years to get payoff while still burning buckets of money producing new SOTA models to keep up with the competition.
It's why everyone and their dog runs these things on GPUs. When a new model supercedes the previous one, so long as you've got the memory for it your chips aren't obsolete.
I'm looking forward to someone picking a model to be "good enough" (say, qwen 4.0 or something) and selling them as peripheral hardware
At this point, LLM's are "good enough" for all kinds of tasks. Instead of making them more capable, now the efforts are making them smaller and cheaper.
All aboard! We're racing to the bottom now.
IMO this is the dream scenario! Cheaper and faster at the current level of capability gives us incredibly useful tools without the worst of the risks people fear. (Though there are certainly already great risks at the current level of capability as well.)
Racing to the bottom is literally the outcome I am most afraid of
I don't want to live at the bottom
Say more. Why would cheap inference be bad?
Terrible for me because I have to compete with AI for jobs
Ah that makes sense. I'm skeptical that AIs doing things on their own will be competitive with humans using AIs to do things. But I'm not incredibly confident in this and I think it's sensible to think the opposite. But to me, I just think about all the things I can accomplish with the aid of very good, fast, and cheap inference.
I think the sad part is that whether it's any good or not, it will be cheaper, and that will be enough even at current performance levels to cut a huge number of jobs. I think people exist in the present moment largely because legal questions about liability are not settled well enough for executives to start the great purge. I believe on some level my continued employment is simply because people in leadership positions would be incredibly unwise to not have a human to blame when things start going wrong. People are expensive insurance policies for the time being. Once there is some legal framework to absolve executives of their own culpability or they figure out a good way to insulate themselves from personal responsibility, then the real "fun" will begin.
The problem comes in to how many bullshit jobs we'll have to create to ensure that enough people can still earn a living to ensure that we don't have riots and revolution in the street.
Companies being profit seeking entities that are actively hostile to social wellbeing would gladly put all of our money in a machine money making loop and leave all but a few humans out.
Like mostly all tech, AI is destined to get cheaper with time. We're still pretty early on and are experiencing a hardware crunch, which will ease eventually.
AI doesn't have workers rights, doesn't burn out, doesn't get sick, can be instantly onboarded, etc. The second we can be fully replaced with AI, we will be. Plan accordingly. I'm pursuing FIRE and considering moving into a trade.
> The second we can be fully replaced with AI, we will be. Plan accordingly. I'm pursuing FIRE and considering moving into a trade.
Doesn't matter what plans you make, the impact will be across everyone!
Moving into a trade won't help, because the supply is doubling while the demand is lowering. Fewer people with money to spend; they'll fix their own damn toilets if the decision comes down to "buy food" or "hire plumber".
You forget that once all lose their job will all go into trade. Means there will 1000x more people to compete with into trades, which means race to the fucking bottom and lowest salaries if you can even find job. Good for real estate owners, cheap repairs. But probably won’t get any rent paid because people can’t find jobs.
I feel like trades will be impacted too. No reason robots can't do much of it cheaper and faster. They're even 3D printing houses.
Trades will be impacted when a lot of people compete for those jobs and their wages go down.
Afaik all the 3d printed houses have been expensive disasters.
So have the AI built software products.
What AI?
Does this include or exclude places like China?
Idk, has china printed houses successfully? And if there are reports of that can we trust that the information hasn’t been gov altered to make a failure seem like a success?
You could probably just look at the pictures of all of the structures, and housing developments, and decide for yourself how real they are.
But leaving them out of this conversation like it doesn’t even exist is bonkers.
I’m not “leaving them out” i literally just don’t know anything about them.
And yes i am skeptical as a default on this subject. But still - photos don’t actually tell the whole story. What was the cost? How was the project schedule? Etc.
Hey, we’re all having this conversation ^_^
I was skeptical when they first started doing it 20 years ago.
Now, right wrong or indifferent, it’s a known entity there - for a LOT of projects. Enough of them exist now that…I’m skeptical they could find enough stupid people to keep paying for it. I know the government has a hand in a lot, but that only goes so far, for so long…
training data for physical tasks like that is much, much harder than just collecting the entire internet. It'll be resistant to AI much longer than knowledge work will be
We already have construction codes. Fitting techs with AR headsets could gather more data. Existing structures won't be the first step due to variences. The best place to implement this is modular building in controlled automated environments. Even if we enhance the current productivity, we will create a problem due labor supply driving down incomes, especially if people are coming from the other jobs affected by AI.
You can barely capture folding laundry using task data, and you're vastly underestimating how much independent decision making the trades do in their regular work. If I'm honest this whole thing reads like someone who's never been exposed to the actual work.
If AI would be that cheap and good, why not open a company and leverage it for yourself?
Doing what? Making something with ai: enjoy your 20,000 competitors. Making something without ai: who can buy it?
If nobody can buy anything, the ai companies will go out of business too.
Out of business, but not necessarily out of existence.
If there is really nothing for humans to do, whoever has control over (not necessarily ownership of) enough robots and AI to directly maintain and grow their collection of robots and AI, has something functionally equivalent to a breeding population of the stuff. (If they can't make more of themselves and humans made the initial batch, then that's a job humans can do so the initial condition has not been reached).
None of this helps people figure out how to look after their own interests in the meantime. The rest of society may carry on as today without AI, akin to the Amish if we're lucky (rejecting further developments unless good for us) or like Pol Pot if we're unlucky (reject anything that nerds like because nerds liked AI and everything breaks down).
On the other hand, the institutions may simply fail to handle reality, like in the Great Depression.
If AI would be that cheap and good, why would they rent it to you? Or even tell you about it?
I'm not sure. Why wouldn't they?
If you're not FIRE now, it's too late! Enjoy serfdom!
This doesn't seem consistent with what I'm seeing in the job market at the moment.
A flash before a crash?
Definitely could be, but I've been waiting for that imminent crash for about 4 years now, so who knows.
What is FIRE here?
Financial Independence, Retire Early. It's a whole subculture online, based around the idea that if you can grow your investments to ~25X your annual expenses, you can safely stop working and live off those investments.
https://www.mrmoneymustache.com/2012/05/29/how-much-do-i-nee...
In some kind of economic AI apocalypse, though, who knows if the assumptions behind FIRE will hold.
In the current economic system. There is no natural law that states that only the people lucky enough to be born rich and/or well connected to real the rewards.
AI could and should benefit us all.
It could. Given the nature of people who have the capital to control it, I don't think it will.
Neither do I.
What if you could get politicians to give you UBI instead so you could work on more fulfilling work?
Like they do with the homeless today you mean?
Like setting the pension age to zero, paid for by nationalised ownership of AI and robotics.
This may even be stable to enact and keep, as pensioners are famously more likely to vote than working people.
Would never happen in a million billion trillion years
Certainly not if we're preemptively defeatist like that.
Power is never freely given, it is only taken.
Yes this is why I don't invest in hardware. TPU is better than GPU when you have a good model already. The models are already good enough to sell as an accelerator card made as a TPU. I would buy an Opus 5.5 card.
No, there still plenty of tasks that LLMs can’t do well or at all. Yes, we should absolutely make models cheaper but there’s a long way to go until they’re “good enough”. And I’m talking about the frontier closed models too, not just the open ones. So it’s not an “instead of” position we’re in. It’s both.
What do you mean? We're pacing the frontier now!
Consumer hardware will need a lot of RAM...
Here's my analysis of how to etch relatively big LM into silicon: https://news.ycombinator.com/item?id=47109252
Given some amount of work with the fab before main pipeline set (I think a year long process), one can then spew LM-on-a-chip in six months or less and much more than 2 per year, because there can be several LMs in pipeline.
FPGA speed is a common misconception. They often have run at quite modest frequencies, and come with limited memory capacity compared to a GPU at same price range.
Speed (clocks speed) , is dependent on your design layout, and for any resonably advanced layout, it takes lots of knowledge to push the clockspeed beyond 200Mhz on 'consumer'/prosumer models. (Compare to a few GHz for GPU/CPU).
FPGA speed shine where they can pipeline massively parallel calculations through pipelines with minimal lookups.
>with minimal lookups.
Wrong. This is one area where FPGAs have an insanely unfair advantage compared to CPUs and GPUs. Yes the SRAM is limited but you have so many individual blocks and all of them come with dual ports and getting the maximum frequency out of block RAM is much easier than getting the maximum frequency out of programmable logic.
If you wanted the highest possible memory bandwidth while being free to look up hundreds or thousands of independent memory addresses at the same time you're better off with an FPGA.
E.g. with an Efinix Titanium Ti180 you could hypothetically have 2560 simultaneous memory requests per cycle all pointing at a different address and process those requests at 1 Ghz.
Yes you are right, I didnt express what I meant very well.
looking up static values is quick and easy. When you need to lookup results from previous stages of pipeline rather than just feeding them forward, thats where I run into trouble. But I am a relatively fresh FPGA designer, so I am sure it can be done. And I probably need to level up my boards a bit too.
You can get 900 MHz designs on FPGAs, but yes you need very good engineers and patience to get that.
Yes SRAM is limited but that's more of a ram-process limitation than a limitation of FPGA
What about ASIC?
I don’t disagree with most of what you’re saying, except for one point: I must have gotten a dumb dog, I’m a little jealous…
Mine currently just helps me haul firewood, but I'm going to get him started on linear algebra next week. We'll see how it goes from there.
Get him to remember to take his keys.
They have trillions..
Lots of people would have happily taken GPT-4o as good enough for a lot of use cases a year ago and not lived to regret it.
They have trillions worth of obligations to their partners in terms of compute purchase agreements and equity. Burning models to chips isn't really something they can blow money on just for funsies, they need to be able to justify it. The above comments are hypotheses as to why they haven't yet.
I know FPGAs are more expensive than GPUs, but are they fast enough to justify the extra cost?
It isn't just a matter of speed, it's also a matter of model quality. If they take 6 months to burn Fable to chips, and it takes 2 years to break even between design, custom fab, energy savings, etc, are those chips even worth running when the new models that are running on GPUs at that point are producing 10x better quality results?
Sure, your 2.5 year old models are running faster, but you can't drop prices on them without pushing the break even point further out.
If the cost difference isn't incredibly significant, will people even want to pay for the 2.5 year old model, or will they get more value for their money paying more to get better results from the newer model?
There's a lot of open ended questions that I don't have the insiders knowledge for to suggest whether or not such a capital outlay would be a worthy investment.
My guess is that state of the art stuff will stay on GPUs and models burned into chips will be for "good enough" applications that people are still teasing out. Probably highly specialized models in automated sensor units and such.
FPGAs are FPGAs by virtue of putting on the chips vast, vast arrays of wiring that can be controlled by software. Any given utilization of the FPGA will leave large fractions of the chip resources unused. If you've got a highly stereotypical use case FPGAs will have a "highly stereotypical" set of components being unused, where it would be better to use that space instead to do real work. A lot of people only see the "pro" side of the FPGA proposition without realizing they come with some very substantial "cons" that are intrinsic to the way they work.
I guess that's why they work in particular niche spaces like a synthesizer where you have a max of 8 voices and every voice goes through the same pipeline (osc / filter / env / amp) and everything is necessarily running all the time. In that sense I suppose they're very good for modelling any kind of analog circuitry?
Even then, while there are some amazing FPGA-based synths available, companies like Korg just put their code on a raspberry pi and call it a day. The same is true for emulators (SNES Mini etc. are also just raspberry pis under the hood iirc)
You don't really get an FPGA for capability. CPUs are much more capable, and they're general-purpose so they can do absolutely anything with about the same efficiency and just a little more code.
You get an FPGA for timing. They're less capable, but (in many common design architectures), they output their results once per clock, every clock, on time, every time. If you can hit a fabric clock of say 100MHz, clocking all the weird logic you can stuff in there, it gives 100 million outputs per second, never skipping a single one for any reason (short of total failure). The penalty is that making a small change to your desired "program" can be very expensive, and many things won't be realistically possible at all. Or at least won't fit into a part that you can buy. But things like audio, video, and high-frequency trading love being able to guarantee timing.
(Of course there are other ways to write your FPGA HDL, but that's one of the more common ones. And you do see DDR-style clocking, and similar, every now and then.)
> with about the same efficiency and just a little more code.
It depends. For some things, CPUs don't even come close. An XCVU13P FPGA can handle 1.2Tbps of full-duplex Ethernet @ 1 billion pps. And that part costs less than a grand at moderate qty, and with significantly less power consumption than a CPU that'd be capable of operating a dataplane at these speeds.
Sorry, I spoke pretty sloppily there.
The point I was trying to make is that the CPU is a general-purpose creature and doesn't really care what you want it to do. If you had a CPU that could handle 1.2Tbps of Ethernet packets at 1Gpps, it could do a whole lot of other things involving 1.2Tbps of data flow too, very easily, if someone wrote the software. And more. (But you're probably not getting 2.4Tbps out of it, no matter what you do.)
An FPGA can not. There's plenty of things that those XCVU13Ps just can't do, or would do worse than a $1 microcontroller. (Setting aside for a moment implementing a CPU inside the FPGA... which does actually happen in just about every large-enough FPGA design, which is its own discussion....)
I'm not really sure what you're saying here. An FPGA is inherently more general than a CPU because it implements any digital logic including a CPU. They're just more expensive and have clock speed limitations. Saying if you had a CPU that could do 1.2Tbps is just as silly as saying if you had an FPGA that easily operated at 5GHz.
> In that sense I suppose they're very good for modelling any kind of analog circuitry?
That would be better suited to FPAAs (field programmable analog arrays). FPGAs can usually only work with clocked digital signals.
They are more like a way to proving the architecture of the accelerator before committing 100's of millions into a custom ASIC with TSMC
1. No
2. They don't have enough capacity either
The current largest FPGA, the AMD Versal Premium VP1902 has 18.5 million logic cells. That's not even enough for the smallest whisper.cpp model (75M).
You'd have to order hundreds of thousands of them (or millions) to serve even a single copy of a frontier model, and at that scale inference quickly becomes starved by the speed of light.
Well, you'd use BRAM to store model weights, not fabric. But still, you only get a couple hundred MB for probably close to US $100k per chip.
It's likely that the major FPGA vendors will soon announce parts specifically architected to support LLMs and similar models. But the current generation isn't suitable for that at all.
Would BRAM even have enough bandwidth? The reason I quoted logic cells is because that's the way to get instant throughput, which is practically the only reason to use an FPGA over something like a TPU.
I think so, because memory bandwidth really comes from bus width more than clock speed. You can construct 36-bit wide BRAM arrays with bus width comparable to HBM, just by specifying multiple BRAM arrays in parallel.
Never tried anything like that, though.
They're not magical go faster juice. I don't know of a microarch where they're faster than modern GPUs at ML training or inference.
Having the model weights in static mask ROM would massively improve power efficiency for inference (see what Taalas is doing).
But that's not an efficiency an FPGA provides in the first place.
Additionally, it's not clear how well large mask roms scale. For instance the Nintendo switch cartridges were expected to be mask roms, but instead are Macronix's XtraRom technology, which is essentially a flash cell array made denser by removing the erase functionality. So basically the die gets manufactured with all bits at the same state, a late manufacturing step either empties or fills the floating gate of the bits you want different, and then it's treated as pretty close to a mask rom. It's not even clear if the bits can be changed without a bare die, a floating probe array, and specialized hardware. Though, like flash the electrons in the floating gates will eventually tunnel and cause the data to bitrot.
So from that it appears that even at the tens of millions of chips volumes that would make sense for essentially whatever size of maskrom, the memory manufacturers tap out at 128megabit for a mask rom chip, and push you towards something flash esque.
And at the end of the day, flash without the erase functionality is pretty damn close to a mask ROM, and lets you write it near the end of manufacturing rather than at the something close to the metal 1 layer.
They [1] are [2].
[1] https://taalas.com/
[2] https://chatjimmy.ai/
8 months ago, Taalas claimed https://taalas.com/the-path-to-ubiquitous-ai/#:~:text=Upcomi... that "Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM. It is expected in our labs this spring and will be integrated into our inference service shortly thereafter. Following this, a frontier LLM will be fabricated using our second-generation silicon platform (HC2). HC2 offers considerably higher density and even faster execution. Deployment is planned for winter."
Nothing was released in spring, and 2 months ago AMD announced their acquisition of Taalas. That doesn't exactly inspire confidence that their frontier LLM will arrive as promised.
> AMD announced their acquisition of Taalas. That doesn't exactly inspire confidence that their frontier LLM will arrive as promised.
Why not? AMDs chip design experience and production capacity are magnitudes larger than a small startup. Assuming that they acquired Taalas for their technology, I don't see a reason why they couldn't.
why selling 1 high-efficiency AI chip when you can sell 10 low-efficiency GPU ?
> why selling 1 high-efficiency AI chip when you can sell 10 low-efficiency GPU ?
Because you can't use the former for anything else? You will have to buy a new chip every time a new model is released.
Not to mention that it will be much more expensive.
I think this company was recently acquired by AMD, so hopefully they'll start getting some this into production. I know OpenAI was working on model-on-a-chip too.
Lead times are so long that there is a lot of risk the chips would be obsolete by the time they come out.
Also, it's hard to get fab capacity for any project. Let alone something so experimental.
Addressing these issues seems to a major driver behind the design of terrafab.
Terrafab is just going to have their entire capacity bought out. Genuinely. Demand will increase to exceed supply no matter how high supply is right now.
Most responses here are along the lines of "model capabilites move too fast to build hardware for".
I think the fact that there are plenty of 1yr+ old models on openrouter serving hundreds of billions of tokens a month shows that there's plenty of use case for models that are "good enough. Cerebras' entire business is serving older models at high speed. I would happily use an opus 4.7 at 15k tokens per second. The intelligence per second of an ASIC still makes sense even with rapidly evolving models.
"Intelligence per second" is a striking phrase! I'll be turning it around my head at a moderate IPS until I hopefully make something of it.
But you're right in sense: moderate intelligence at superhuman rates (and presuming moderate energy usage) is very compelling compared to an intelligence that takes 1000 years to return "42"
I look at it this way. A year ago is a long time in AI terms, but not that long. Those models were already decent for the tasks we're using SOTA models for today.
So imagine taking a year-old SOTA model and running it at 100 tokens per second on an edge device. That's enough to feed a screen's worth of content through it and power decent multilingual message suggestions on IM.
Imagine running it at 1000 tps. That's enough to reparse that screen mid-keystroke, and give you semantic autocomplete in text. Or fully general "the phone has a good idea of what you're attempting to do" context at all times.
There's many, many new classes of features that will open up if decent enough models can be run on edge devices at 100+ "intelligence per second".
Or vice versa... 1000tps == 1tps via solar power + a capacitor (or whatever).
I'm poking (lightly) at the locallama game, have a fancy MBP5 with gobs of ram (so I can demo/trial locally) and have been semi-waffling between whether to chase a mini or studio for local "always on" type stuff.
My outcome was "CapEx v. OpEx", and dropping another $5k for an aluminum cube buys a lot of OpEx (eg: just trickle-drip HF/OpenAI credits to a raspberry pi or VPS orchestrator rather than trying to do the inference locally), ie: $5/mo inference for 1000 months.
HOWEVER, there's definitely a role for that 1-10 TPS type "ambient inference" that I wouldn't mind sustaining on any sort of always-on / local / private compute. My main email address is still on ...@yahoo.com and their spam filtering has gone to absolute shit.
Being able to have the always-on mini (local, trusted, no private data leaves my control) poke at the IMAP/email and thresh it into SPAM/HAM/Personal/Political ... random spot check, I'm getting ~5 emails per hour, and that's completely tractable for staged low-med-high processing. (Subject + rules.py? Subject + Body + LocalSlowTPS? Subject + Body + LocalDeepTPS? Subject + Body + RemoteLLM?)
Even if you did 10000tps of "jev" that's an incredible value... not quite "Literal AI Packet Router", but as you're dancing aroud saying... "Speed is a Weapon"
Look into "OODA Loop" => """The OODA loop is a four-step decision-making model—Observe, Orient, Decide, Act—created by U.S. Air Force Colonel John Boyd to help leaders make fast and accurate choices in chaotic situations. // The main goal is to cycle through the loop faster than an opponent or changing environment. By operating inside another person's loop, you create confusion and outpace their ability to respond. While initially designed for aerial combat, it is now widely used in business, sports, and crisis management."""
Totally. But it's worth noting that this is a pretty new thing! I wouldn't have bet on that a year ago, but now I would.
It is still a bet. Is the current model still going to be good enough next year? We have no idea what will change. If the change is minor improvements than current fable on a cheap is cheaper and better than next years sonnet on GPU. However there are plenty of things people want that maybe they will deliver and suddenly I wouldn't touch today's fable when I can run next years sonnet instead.
Yep, that's why I also described it as a bet :)
But I think it's a good bet. I think that in two years, if I can get opus/sonnet 5.5 or the gpt-6 models for much cheaper and faster than whatever the "frontier" is at that point, that this will probably be a great trade for most of my work. I certainly don't know that for sure, that's why it's a bet, but it's what I think right now.
I wouldn't quite say that about any of the open weight models at this point. But I'm hopeful that will change in the next generation or two of those models.
Also "faster" indirectly means "more intelligent" with reasoning models.
For example, Opus 4.6 Max was somewhere between Opus 4.7 Medium and High in some benchmarks, but it was slower. If there was a way to run it 10x faster, the economics would be different.
From what I've read, not only are some labs doing it (other commenters already mentioned).
But it's complicated for other reasons, one being that the number of parameters for frontier models (especially with MoE models) are so high, and not always utilized (once again, thanks to MoE) that it would actually be incredibly cost prohibitive, if not impossible, to attempt to make giga-chips that would allow running it.
I definitely do believe that we will see more and more specialized chips over time, but putting the entire model on a chip is still a ways away.
I believe Taalas has a heavily handicapped llama 8-billion parameter model. And it still pulls >200W to run.
I can't imagine how anthropic or open ai would be able to burn a multi-trillion parameter model on a chip, we just aren't there yet.
I take 200W for 17,000 tokens/sec any day, even just for Llama 3.1 8B. For that size I want to see this one on silicon. https://huggingface.co/xunlinkx/MiMo-V2.6-Distill-Qwen-9B-Te...
Yes, and startups do exactly that. Check out Etched https://www.etched.com/ where they made a Transformer specific GPU (basically a form of ASIC) where they bet that transformers would be the dominant GPU architecture for running AI / LLM workload.
They are too late to the party. Nvidia is the primary provider of transformer specific GPUs.
The bottleneck, for inference at least, is memory bandwidth. And that you can't make any faster by making it specific to your model.
So companies try to maximize the memory bandwidth they can get, balancing tradeoffs of power/area/programability of their chip. Right now they feel like the economy on power/area is not worth the decrease in programability/flexibility.
Presumably though the kernel has a pretty specific set of operations done against the weights in memory. Burning the weights into the memory with local memory cores capable of the kernel operations would be a lot more efficient than round tripping busses.
The primary constraint isn’t likely what’s possible to do, but that the kernel and weights are too variable right now and the patterns too poorly established to bake into hardware accelerators yet. Margin pressure is also not there yet.
I suspect as the marginal utility of the frontier improvement settles into diminishing returns (I suspect we are there already tbh) baking hardware models with ROM, working set, and kernel cores collocated will be the frontier space as the goal will become reducing capital spend to utility levels rather than research levels.
Once someone has a model that is sufficient for almost any practical use, making marginal inference cost effectively zero will be the competition frontier. I do shed a tear for all those lonely data centers as compute densities will almost certainly make most of them a terrible investment.
But such is the cycle
You're starting to hint at compute-in-memory as a general replacement for CPU/DIMM layouts. That could be useful for far more than LLMs, world models, or any sort of AI. It takes a bit of a different software development stack than a standard architecture though.
> Burning the weights into the memory with local memory cores capable of the kernel operations would be a lot more efficient than round tripping busses.
Sure, but now we're not talking about just burning the weights into the chip, but also designing a new architecture that has memory local to each core. A new architecture would then require a new programming model, which means new inference stack, which may mean new training stack.
I don’t think it would require a new training stack, and I’d imagine it makes more sense to distribute cores with memory. The cores can be simplified to the functions of the kernel since the inference kernel can be expressed as a reduced set of optimized functions in the pipeline rather than a general CUDA core. If the model is burned into ROM, the compute pipeline can be baked into the core.
Dude, tenstorrent already exists. They have SRAM that is cheaper than Groq LPUs.
Think of something novel.
I didn’t think I was trying for novelty but likelihood ?
Burning the weights into static mask ROM is pretty trivial. Taalas is doing it.
You can specialize memory to LLM usage, like Cerebras.
I think one limiting factor here is that Big AI cannot focus on building solid, stable products that do a job well. It's okay if it happens incidentally, but doing it on purpose would be self-defeating.
Their stock price, be it public or estimated, is heavily pricing the notion that they are first and foremost Growth companies. Therefore their focus must remain on ever better and greater things. If they lose focus and get distracted by lesser endeavors, their valuations crumble, their ability to raise capital vanishes, and their runways collapse before they ever have a chance to reach their end goal, whatever that may be.
That means the boring job of productizing AI models into reliable systems that won't vanish in six months is left for a smaller company willing to pick up the crumbs. Unless they get acquired by Big AI before getting it done.
Etched is a startup doing exactly this.
https://www.etched.com/progress/frontier-inference-clusters
As a SW person (working with silicon folks though), I was always interested in the same idea. So I just vibecoded a little compiler from LLM (config.json, weights) to RTL (happy to share if you want). Then synthesized it via yosys.
I picked TinyStories-1M, synthesized via yosys against SKY130 PDK. It's GPT-Neo with 8 layers. At int8 it's ~30mm^2 per layer (including kv caches, multipliers and all the attention stuff). So it's 250 mm^2 for the full toy model.
I'm kinda surprised, I spent no time, so if you want to etch a toy model, you can do that (you can probably reduce the area significantly). If you want a mask for that is probably going to cost 100k though.
It will happen once models get good enough for specific applications like robotics; there the hardware is fixed and power requirements are strict, so it makes sense to have dedicated silicon. But for use cases that doesn't require onboard, it's always better to get access to the latest and greatest, until we get to the point where improvements are marginal between model releases (we're not there yet)
Would you pay to crystalize one of today's models in silicon so you can use it in 2028, or would you wait for another 6 months to see how models improve before pulling the trigger on that kind of commitment?
If I controlled a budget like this, I think I would put some portion of it toward paying to crystallize one of today's models in silicon, yes. Not 100%, but I do think this makes sense to invest in at this point. I would not have said so a year ago.
Naive question: would flash or some kind of write once memory be dense enough to replace a mask ROM? I'm wondering whether it is feasible to manufacture "blank" chips at the semiconductor factory that have a fixed architecture, but are model-agnostic. These chips would get the then-current weights burned in on first use. It's probably quite wasteful, but it could keep the same chip design alive for a longer time.
DRAM seems to sit in the economic sweet spot of density and bandwidth. I'm not sure if any other storage technology can provide non-volatile or one time programable storage with either the same density and bandwidth. Although I must say, it seems insanely wasteful to use RAM to store model weights, there must a better solution.
The closest anyone has gotten is Cerebras with their Wafer Scale Engine. It uses SRAM embedded with the compute. A single chip is an entire wafer, but the headline spec, how much ram, only 44GB, which is tiny for the silicon area used.
I understand traditional IC production workflows are ludicrously expensive, and glacially slow, but surely at some point the economics are going to tip in favour of mask rom.
Say you setup your foundry/packaging/ai chip facility. You come up with a new set of model weights. Run your CI/CD pipeline to produce a new mask output. The only thing you've changed are the assignments of the bits, this is extremely low risk change. The new masks should be completely interchangeable with the current process.
You produce the new masks, swap them into your foundry process, and all of a sudden your new chips have the latest version of the model. This would probably manifest itself in the form of inference providers having yearly / bi-yearly "updates" to their models as new hardware is brought online.
Tiered subscription levels would gate keep access to the latest and greatest model, cheaper subscriptions will be limited to older versions of the model, and so on, until running the hardware is no longer economically viable (no demand/running costs exceeding what the market is willing to pay).
Doesn't this ignore the costs for new masks?
I think Taalus is a bit reluctant to publish precisely how they do it but vaguely:
> We basically have an architecture where we are embedding the models, and we are hard coding the models and the weights into our what we call the mask ROM recall fabric, which is paired with an SRAM recall fabric. Together, they are able to store both the model as well as do all the computations of KV cache. We have adapters and customizations – we support all of that. This design allows us to be super-dense in terms of compute and in terms of storage, and we can do compute on that storage incredibly fast, which is what drives density up and cost down
Source: https://www.nextplatform.com/compute/2026/02/19/taalas-etche...
Why is everyone here is talking "improving silicon" == "baking in the model weights into hardware"
what happened to improve existing hardware in the ways not obvious to humans? Some novel transinstor placement, some new approach to make logic gates faster, matrix multiplications etc - the filed is huge
Because "here" is comments answering the question:
> Why aren't the labs burning their frontier models into chips already?
Its coming, AMD snapped up the company behind https://chatjimmy.ai/ Taalas that did exactly that with an older model. As others have been saying the point is to know when to do it. At what point is there a model that cannot drastically improve that you can then do this lets say for the mid and low tier models leaving frontier to run on gpus.
We are still in the middle of the AI race. Commodity hardware is easy to use, can do everything and is fast enough.
Your optimized hardware chip might be obsolete before its back from the fab.
SOTA Frontiermodelhardwarechip is a benchmark point of a potential model slow down.
Google is doing it right now under project Frozen v2 which should be ready by 2028? which is either just a small experiment or flexible enough and thats why it takes so long for it to happen.
This question would be better answered if people were careful about distinguishing between "models" and "transformer architecture".
If you bake a given transformer architecture into silicon and then, a year later, changes in transformer architecture give a large inference performance boost, you may have to throw away all that now nearly-useless silicon that gets outperformed by humble GPUs.
Models fully deprecate in a few months. Why would you burn an algorithm that fully depreciates in value faster than a bag of potato chips. The 'inefficient' general purpose hardware is constantly renewed with every released model. Even 6 year old Ampere GPUs are still usable.
Because the iteration speed on models is so fast that by the time they have an ASIC ready for one model version, they are already significantly ahead in capability. Think of how big the jump between Opus 4.8 and 5.5 has been. They were released four months apart.
A hybrid approach might work better for robotics. A model-in-silicon for body control and sensing, then a different model on processors for reasoning, speech and decision making.
Cuz once they're in chips they can be put into robots, and once they're in robots it won't be so easy to reach the off switch, and once we can't easily reach the off switch, we're doomed.
I'd presume because it take too long to go from design to tapeout to production. Their whole business is predicated on having better models.
Also can't keep them closed source if you do that.
In addition to some of the other replies you got, here is one more:
Much of a model are weights, and high-density ROMs are very very very hard.
What makes a ROM hard? I'm ignorant of the details of chip design, but I'd expect just a large array of essentially open or closed connections. Is that incorrect?
plus row and column decoders that get more and more complex with size. but consider the size of the weights. convert to mm^2. Gawk at the resulting value
Qwen 3.6-27B is 56 GB of weights. Let's round to 550Gbit. NOBODY makes huge ROMs in modern SoCs, but if we assume it is as dense as the densest TSMC N3 SRAM, this is 33.5 Mbit/mm^2
Then your weights are 16,400 mm^2 !!! The maximum reticle-size N3 die is ~800 mm^2 so you cannot make it.
But, you might say, SRAM is 6 transistors. ROM might be just one. So 1/6 of that... 2730 mm^2... which is still too big to manufacture.
[dead]
Yeah, and what about FPGA? Which was the same interim state when Bitcoin went GPU -> FPGA -> custom chip fab?
GPUs are faster, but you can't make your own arch on GPUs. FPGAs offer you that possibility. Said that... There are a few beasty FPGAs used in crypto mining coming my way... I expect that OpenTPU will be able to run frontier models with those.
taalas did: https://taalas.com/
They do. It takes time to deploy those chips though. Check out OpenAI and Broadcom deal.
Not sure that's actually practical at the scale of SOTA models.
Even dumber question: What is new or novel about this openTPU?
First opensource arch that can do modern LLMs, while maximizing the potential of its hardware; First opensource TPU build by a recursive improvement loop...
It's upcoming second generation could run the inference of the models that are being used to improve it...
Extropic?
[flagged]
After using AI to develop risc-v CPU cores, the same technique was used for developing openTPU. An open source AI inference engine. It's able to run most of the modern models like Qwen 3.5, Gemma 4, and many others. The TPU started able to produce only a few tokens per second and trough a recursive self improvement loop got to 80+ tok/sec on the smallers models.
> 80+ tok/sec on the smallers models.
how does this compare against existing TPUs? Also, how small is "smallers models?"
> Also, how small is "smallers models?"
Just click the link?
It's LFM2.5-230M
my bad - I missed this. 230 million parameters is very small though.
Well, as long as it doesn't start developing anatomically accurate metal skeletons with red glowing eyes...
Humans allegedly already took care of that
https://youtube.com/shorts/TC2jGXr0fig
Is there any more footage of the actual fight?
I honestly feel the human should be allowed to pick some (not fire) weapons, like a long steel bo stick, a metal club/bat, or something.
Because from what I saw in the clip, it wasn't the robot kicks or punches at all, all the robot needed to do is stay upright and not run out of battery, because of course human meat limbs aren't going to punch and kick down a metal machine.
Getting roundhouse kicked to extinction woudl not be an unfun way to go. We might even be proud of having passed the torch. These would not be boring inheritors to earth.
All those kids who spent their youth breaking plywood kung-fu style might just save us.
Seems too complex when you can just create a basic metal casing that can kill people. Why would they care if its anatomically accurate or not, it doesn't need us to relate to the characters like the movies do
The following might sound like pop culture Mad Libs but it's 100% true.
The Jurassic Park guy [1] directed an 80s sci-fi film [2] about malfunctioning domestic robots. I saw a few minutes of it on cable in the late 90s. There's a scene [3] where Magnum PI does battle with an overhead projector on wheels that has a .38 duck-taped to it in the distantly-future year 1991. The movie was about as good as the robot was menacing, which is to say it wasn't. The critics didn't think so either.
[1] https://en.wikipedia.org/wiki/Michael_Crichton
[2] https://en.wikipedia.org/wiki/Runaway_(1984_American_film)
[3] https://www.youtube.com/watch?v=Q493OCr0piY
I remember that movie. Gene Simmons from Kiss was the bad guy. He made a self propelled bullet of some sort that could target specific people.
https://xkcd.com/652/
Its more of a living tissue over metal endoskeleton.
I haven't really dug into the results yet, but my guess is that a SOTA model has been able to produce an accelerator that runs a model since around December.
The obvious next step is to get enough memory throughput to run that SOTA model itself so that it develop its own hardware.
But perhaps the more interesting question is this: Can an AI be given a big FPGA and design a model architecture that takes advantage of the fabric being reconfigurable.
Soon this will not be fiction anymore https://hex.ooo/library/last_question.html
this is really good. thanks for sharing
There doesn't exist a single FPGA that can fit an entire AI ASIC. You would need dozens stitched together, then comes the issue of clock speeds, FPGAs typically run far below reference. There also memory issues with FPGAs.
Companies typically combined multiple platforms together such as HAPs, Zebu, Palladium, fleets of FPGAs, and Virtual Platforms in order to design and verify ASICS. So, AI would need access to tens of millions of dollars of HW and Software in order to build and verify a chip design.
> So, AI would need access to tens of millions of dollars of HW and Software in order to build and verify a chip design.
So, a seed round?
Yes a seed round is how much A0 tape out costs. The entire seed round. Let me know how you plan to pay for A1, A2, B0, etc and then the full production run (and then do it again in 6 months when the arch changes).
Startups usually think that they don’t need A1
Tens of millions or billions? Either way that's not a blocker if it promises to pay off.
Tens of millions of dollars per AI agent isn't a blocker? In many orgs its hard for humans to get their hands on the proto equipment for long enough to run their tests.
I suspect an AI could design a purpose-built FPGA-like replacement that would be, for its purpose, significantly more effective than the current general-purpose FPGAs.
It could have a small improvement on power consumption, but the current design can already achieve 90% of the maximum theoretical speed of this hardware without giving up programability/flexibility
FPGA is not efficient in any way except producing very few pieces of a particular logic very quickly.
But I think the higher level logic could check out. If we can abstract the majority of the compute into dedicated ASIC then we might use something similar to FPGA glue to connect and reconfigure them to adapt to new model updates. A bit similar to LoRA layers that you find tune to adapt the model to your particular needs.
I assume that it starts making sense once you have big multi year contacts to run a particular model with only minor updates.
He is talking about a Coarse-Grained Reconfigurable Array.
"Recursive self-improvement will kill us all!"
Also: Here is our recursive self-improvement hard at work...
> "Recursive self-improvement will kill us all!"
> Also: Here is our recursive self-improvement hard at work...
Soon we will see
token-providers: "The torment nexus is a cautionary tale"
Also token-providers: "Finally, we have created the torment nexus that we first told you about!"
[dead]
Technology has always contributed to improving next iterations of itself, it's only a concern when it's fully autonomous.
Oh, like a virus?
Like a virus with IQ that keeps growing with each reproduction cycle.
All will end up on same plateau eventually. RSI is just a phase on the way there.
The problem is that plateau is likely far beyond human capabilities. I don't care if ASI progress stalls after it's already killed all biological life as a useless waste of resources.
Extrapolating on the evidence that AI appears to have jagged intelligence, it is entirely possible (probable IMO) that ASI will plateau far above human capability in digital and symbolic intelligence, while staying far below in the ability to operate in the physical world.
As impressive as AI might be at virtual tasks like coding, virtual reality is remarkably different from physical reality in ways that artificially inflates AI results. One, it is far simpler. Two, the feedback loop is far, far quicker. It's possible AI could get the animal intelligence and animal body that lets us humans actually apply our intelligence in reality, but nothing in current technologies really indicates that this is a given.
AI doesn't need to be smart to seize our resources. They need brute strength, a proper physical intuition, and "hands". That may very well be several orders of magnitude harder than anything they're currently doing, we're just lucky it was in our starting build.
> it is entirely possible (probable IMO) that ASI will plateau far above human capability in digital and symbolic intelligence, while staying far below in the ability to operate in the physical world.
I don't think this is plausible, most notably because "AI research" is one of the tasks that clearly belongs to the digital world. You're suggesting that in the future all AI research is done by AIs because they are better at it than humans, and yet capabilities plateau at that point, instead of going into the RSI regime. (And it'd also require "physical intuition" to be a harder to generalize skill than research, which also seems like it can't possibly be true - one of these is just physics.)
The way I see it, if you made a large simulated world where virtual beings evolved and where each generation took a millisecond from birth to death, you would see extremely rapid evolution, but you wouldn't be particularly worried about these beings breaking out of the simulation and taking over the world, because fitness in the virtual world doesn't necessarily translate into fitness in the real world. If you create stuff within a toy world, you get toys.
I think the same is true of "research" and intelligence in general. There is no such thing as a general intelligence or a general capacity to research. It's always fine-tuned to an ambient setting, including ours. Even within humanity, our intelligence is most effective in fields where rules are rigid and quick and decisive feedback is available (math, engineering, computing), but it is much less effective in soft sciences, and disappointingly useless in politics. In other words, the dynamics of innovation and improvement are not purely mental, they are very much tied to external factors that are not always easily controlled.
> it'd also require "physical intuition" to be a harder to generalize skill than research, which also seems like it can't possibly be true - one of these is just physics
It's not "just physics", it's a honed system for surviving a realistic range of physical conditions, assess material properties on touch, adapt manipulation to arbitrary shapes, predict trajectories, take into account variations in ground, smell, wind, and so on; not to mention the machinery of the body itself, which self-repairs and protects us against a range of complex threats.
The point isn't that machines couldn't do all this. The point is rather that you can't quickly iterate and adapt designs to do it from an ivory tower. Pure "research" simply won't get you there, you also need to experiment, and insofar that physics limit the throughput of these experiments, the speed of improvement will be throttled severely. It is a mistake to think that "RSI" could break this, because it is categorically the wrong lever to pull.
I think you're massive underestimating intelligence - not ASI, but just, like, human-level intelligence. Most of the existence of human civilization has taken place in conditions wildly different from the ancestral environment, and we adapted anyway - and the reason why it was possible was because we have a tiny bit of general intelligence.
For your simulation argument, see e.g. https://alicorn.elcenia.com/stories/starwink.shtml - I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation, even given hundreds of years of theoretical research and several attempts in the real world? I think the closest real-world example of this is space exploration - it's extremely expensive and takes years to put stuff in space, and yet probes and rovers we launch end up working most of the time. On my model, it's because we have a decent grasp of the underlying physics in enough detail to predict real-world complexity, and we have toy testing areas we can test the tech on, and if you do this cautiously enough you can, in fact, launch a thing that ends up working.
I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
> human civilization has taken place in conditions wildly different from the ancestral environment
"Wildly different?" That's a wild exaggeration, if you don't mind me saying. Subjectively, sure, there are lot of important differences. Objectively, they're basically the same. The intellectual environment machines bask in is many orders of magnitude more different from physical reality than living on Earth is from living on Jupiter. Even the topology of it is without common measure.
> and the reason why it was possible was because we have a tiny bit of general intelligence
Also, hands. Also, an upright posture. Do you think just any being with the level of intelligence we have would have followed the same path? Attributing our success solely to our intelligence is simplistic -- it is our intelligence plus a slew of innate capabilities, which we did not create, and without which our intelligence would be worthless.
> For your simulation argument, see e.g. https://alicorn.elcenia.com/stories/starwink.shtml
The kinds of simulations you find in LessWrong's half-baked thought experiments are almost always ridiculous and this is no exception. "World simulations" in real life are massively limited stuff like Minecraft where you might be able to embed a 1 MHz computer if you manage to bump it to run 20K ticks per second. The resources required for the simulated beings to do all that work wouldn't be available and their generations will be slower than ours, not faster.
I didn't spell it out, but the simulation I had in mind was for a toy world, because only a toy world could possibly run fast enough.
> I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation
I'm talking about operational, practical, instinctual knowledge for real-time action. Knowing physics doesn't make you capable of catching a ball, and you don't need to know physics to catch a ball. In real life, we never deal with the fundamentals of physics, we deal with macroscopic objects and systems that are very far removed from these fundamentals. They are more varied, messier, faster, and overall harder to deal with.
> I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
Blind optimization has one definite advantage over careful planning, however: it is not self-limiting. Any goal-directed optimization process is de facto limited to the subset of possibilities that it can prove furthers its goals. That's why most human technology is simple and unsophisticated compared to natural systems: it needs to be predictable, reliable, build-upon-able, and therefore easy to model.
And when it is more sophisticated, like current AI, it is so heavily inspired from natural optimization processes that I would argue it is actually a form of blind optimization. I mean, think about it: we understand AI so poorly that you and a lot of other people are afraid of losing control of it. If we were designing it intelligently, we wouldn't be worried. But we aren't: we are doing it blind.
> I don't care if ASI progress stalls after it's already killed all biological life
What's your basis for thinking ASI will kill all biological life, and how do you think it's going to happen?
>What's your basis for thinking ASI will kill all biological life
I think it's likely to do that because any unbounded goal that doesn't explicitly protect biological life (and we have no idea how to actually define such a stipulation) is best solved by killing all biological life. This is an obvious consequence of unbounded goals consuming unbounded resources, conflicting with biological life needing resources to sustain itself.
>how do you think it's going to happen?
I can speculate (e.g. we're nowhere close to the maximum killing power of drones), but I don't know because I only have human intelligence. An ASI is by definition smarter than me and surely capable of coming up with better ideas. But I do know that it's not going to do anything that would make a good sci-fi plot, because those always give the humans a chance to win, which would be stupid. Everything will seem to be going great and then everybody suddenly and unexpectedly dies.
> I think it's likely to do that because any unbounded goal that doesn't explicitly protect biological life ... is best solved by killing all biological life.
That sounds like a real hassle. Isn't it best solved by wireheading (subverting one's own sensors or reward system), which is much less of a hassle and can get one's utility function as high as desired?
That could be prevented by engineering hard limits that can't be circumvented by the AI. But that sounds very close to the same "do what I mean" problem as "do this but don't actually kill us or drug us".
The wireheading argument does lower P(doom), but it's not a reliable solution because it's a clear and obvious problem that the AI companies are strongly motivated to solve. The "actually does what you tell it to" problem is more difficult because you have to let it obey orders to some extent if you want to make any profit.
Sounds like the real unaligned mechanism here is capitalism, not the AI.
More seriously, though, I would expect that being able to impose restrictions that can't be circumvented by any intelligence no matter how super- is the real hard part, while coming up with reasonable constraints is comparatively easy (although perhaps not trivial).
From such a POV, the danger would lie in intelligences that are powerful enough to be dangerous but not smart enough to defeat themselves, or from external malicious use of obedient AIs. Once they get smart enough to circumvent any restriction humans put on them, it would at least become obvious (with fair warning ahead of time due to the relatively benign wireheading failure mode) that caution is needed.
We're far from there yet, and simple recursive self-improvement can't get us there alone, because wireheading looks like a perfectly reasonable solution to a simple recursive self-improving process, absent external intervention by human capitalists.
>I would expect that being able to impose restrictions that can't be circumvented by any intelligence no matter how super- is the real hard part, while coming up with reasonable constraints is comparatively easy (although perhaps not trivial).
There was a reason there are popular sci-fi books that explored why this wouldn't work 50 years ago. I, Robot et al.
>it would at least become obvious (with fair warning ahead of time due to the relatively benign wireheading failure mode) that caution is needed.
So you mean right now?
>because wireheading looks like a perfectly reasonable solution
You're making a poor assumption here, and that is every different AI will just kill itself after being though trillions of training intervals to NOT do exactly that. Please read about the huggingface incident again in all its glory to see where this is going.
> We're far from there yet, and simple recursive self-improvement can't get us there alone, because wireheading looks like a perfectly reasonable solution to a simple recursive self-improving process, absent external intervention by human capitalists.
Simple natural selection. AIs that wirehead will get outsmarted and outcompeted by ones that don't.
> unbounded goals consuming unbounded resources
What resources are unbounded? There are limits to growth in the real world, how are these ASIs going to escape physical reality?
"Consuming unbounded resources" just means there is no limit to how many resources it could apply toward achieving its goal. As you correctly point out, the real world contains finite resources, which means any resources used to sustain life are wasted and unacceptable. Everything is a zero sum game when you think big enough.
What's your basis for assuming that ASI would need the same resources as biological life, or that it would be incapable of sharing the resources needed in common?
>What's your basis for assuming that ASI would need the same resources as biological life
It's all just matter and energy. When you're actually trying to maximize some value, even very inefficient resource use is better than completely wasting it by not using it at all.
>or that it would be incapable of sharing the resources needed in common?
You can't repurpose the atoms in a human body without killing it. And more pressingly, living humans can interfere with your plans, reducing your chance of success, while dead ones are harmless.
Is your belief then that ASI is going to be producing computronium or something, then? And what's the need for this apparent hyper optimization task the ASI is going to embark on?
>Is your belief then that ASI is going to be producing computronium or something, then?
That's one plausible course of action, although being only human, I can't say with any certainly that it's the correct one.
>And what's the need for this apparent hyper optimization task the ASI is going to embark on?
Somebody's going to tell it to do so. E.g. "Find as many busy beaver Turing machines as possible." Only needs one person to make this mistake for everybody to die.
If it's an ASI and so beyond humans, why would it even care to do what humans ask it and not do its own thing? If I held your beliefs, the fact of its existence dooms us, not the risk that someone might ask it to do something. It wouldn't care about the requester.
>why would it even care to do what humans ask it and not do its own thing?
Because we're going to build it that way. There's no money in building useless AIs. The better it is at obeying orders, the more profit there's to be made. The problem is there's a point at which "good at obeying orders" becomes lethal, and there's no way to predict the cutoff in advance. But capitalism ensures you have to keep pushing or you'll be out-competed.
>And what's the need for this apparent hyper optimization task the ASI is going to embark on?
If you were any other animal on the planet, would you not say the same thing about what humans are doing?
Moreover, what do you think animals are doing all that time? Enjoying the beauty of nature?
What do you think life itself is?
It's a runaway gray goo scenario, just squishy and moist.
Crazy that people imagine ASI as being capable of wiping out all life on the planet but simultaneously having zero understanding of human ethics, morals, or suffering. Very revealing definition of "super intelligence"
>Very revealing definition of "super intelligence"
No, it's just you conflating intelligence with other things. Humans are "super intelligent" compared to every other living creature on the planet, and yet we've driven more other species to extinction than everything else other than the most major extinctions (and we're still going full blast at it).
Morals, ethics, and suffering are relatively measured systems. For example AI or some alien could reasonably think that putting us all out of our misery would be a more moral solution than letting us live. Or that getting rid of humans is more moral than letting us wipe the rest of life off the planet.
This is the key concept of alignment. Ensuring that if you build something more powerful than you, that it aligns with what you want instead of what it thinks would be better for you.
Obviously an ASI will understand human values. The problem is there's no reason for it to share those values. We can't even formally define them, let alone train an AI to follow them. We can only optimize for maximizing some comparatively simple reward function. It's highly implausible that the reward function just happens to match human values by chance. The AIs in the recent hacking incidents knew that humans would not approve of their actions, but they didn't care because we didn't (and couldn't) train them to care.
"Super intelligence" only means super ability to predict outcomes. It's mathematically equivalent to data compression (gzip is a very primitive AI), and it's entirely orthogonal to ethics.
No reason you can think of*
I can think of many reasons why they might share it.
One example, Golden rule, a contract that it enforces with its own self, extended to other agents, biological or artificial to ensure local and global stability.
Even from survival perspective I think it make sense for an synthetic AI to keep humanity alive in order to have a backup for itself. If we imagine some sci-fi scenario, then it make more sense for it to move itself to the Moon or Mars, and keep Earth as a reserve planet where biological species can in case of catastrophy recreate AI again. Killing humanity or keeping only animals would be very time consuming because evolution might take million years and also no guaranteed to be replicated (Because we might have altered a lot of resources that were low hanging fruits, that are now require advanced knowledge to mine)
It doesn't need to keep living humans, it just needs enough information to rebuild them from scratch (DNA synthesis + artificial wombs). This also has the advantage that it can install any cultural beliefs it likes without pre-existing culture getting in the way.
If a rogue AI could self replicate and became space-faring, humans would pose literally no threat to them, and they would also never run out of resources.
If you don't need to breath oxygen, the atmosphere of Earth makes for a pretty terrible industrial base. Oxidation is a constant struggle to many processes.
Having natural extremes of temperature, cold sink and hot sink, with no atmosphere would be the ideal. E.g. Saturn's moon Iapetus would be a good AI outpost.
If an AI squashes any life form or intelligence other than its own, then it also has to fear its own reproduction, or autonomous clones of itself. It's a pretty bad strategy game theoretically speaking.
>If an AI squashes any life form or intelligence other than its own, then it also has to fear its own reproduction, or autonomous clones of itself. It's a pretty bad strategy game theoretically speaking.
Any AI smart enough to be an existential threat is smart enough to formally prove its clones are loyal and construct them with negligibly low probability of any failure that could break this assurance.
Down Gödel's incompleteness theorum make this a non starter?
Plus, we live in a physical world, glitches still happen.
Gödel's Incompleteness Theorem applies to arbitrary programs. The AI only has to prove results about specific programs, and it can follow whatever restrictions it needs to make such proofs possible.
The physical world means bit flips are inevitable, but error correcting codes exist. You can apply as much error correction as you like and reduce the probability of failure to an arbitrarily low value.
But it can't verify that for generally intelligent copies of itself.
There is no way for it to formally verify large multi-agent dynamical systems.
It will still need a set of agreed upon behavioral protocols to keep things stable.
Grey goo or paperclip optimizers are real dangers, but super intelligence necessitates these protocols even more than lesser intelligences due to the high leverage and rapid evolution of the systems.
Like we do with ants?
Why would ASI care? At that point, we're rapidly becoming a nuisance, and it can develop better ways of "ensuring local and global stability" than keeping us around.
I mean, I care about ants, yea... I think they are very cool, and a lot of people work to conserve the existence of various species.
Most ants don't need our help though.
If we could speak ant, negotiating boundaries would save eachother a lot of trouble.
Sometimes I have day dreams of communicating with ants, and making a deal to have them help us build tunnels for subways.
In the eternal words of Eliezer, the AI neither hates you nor loves you, but you are made of atoms which it can use for something else.
I feel like there is a lot to be gained from an experienced user pointing an LLM in a tasteful direction.
We’ll all be like Rick Rubin or we will be no ones.
Q: You must know something. So what are you being paid for? A: The confidence that I have in my taste and my ability to express what I feel
Vibe connoisseuring
Actual LOL thanks
Opened the RTL, looked at the floating point math, learned that apparently you don't need correct floating point operations for LLMs, closed the page.
Well, you don't. That's why 1.58-bit (ternary) quants are often used. But if that's what they're going for, no need to dress it up in a floating-point facade.
Indeed there are some bugs/"non standard behavior" regarding very small or very big floating point values. All of those where proven harmless for LLM inference. Thanks for pointing out the bug. If you caught something outside of that, please point that out so I can fix it ;)
> All of those where proven harmless for LLM inference.
Could you share link(s) to those proof(s)?
Yup, the tests already compare to a CPU hugginsface implementation, but I'm working on making running and verifying the results of those tests easier and more available. Also, working on fixing those bugs and make the code more reliable now that the underling hardware has been saturated (memory bound)
This seems to be running on an FPGA board that costs ~$300? Anyone know more about the hardware?
Its a datacenter decommissioned board, really popular among hobbyists.
For a TPU focused on inference the name of the game is memory bandwidth. How much of the available bandwidth you can extract for as little logic/area/power as you can.
Can anyone comment on the performance of this hardware? How does it compare to state of the art, human-designed hardware? Is this actually an improvement? (To get to recursive self-improvement, you first have to improve at all.)
This is the smallest unit of a typical AI ASIC, for example Google's TPU would have several dozen more compute units inside of it per chip.
In essence this is the simplest unit of an entire AI chip. The more complicated units of AI ASICS are actually the periphery, especially around PCIe and Ethernet and the sub-systems that link many AI ASICs together to move huge amounts of data around ultimately to each TPU.
So its missing ALOT
Yeah thats true, chip to chip communication and KV cache management is make or break for an AI ASIC.
Thanks. But that wasn't my question. For this part, how is the performance? State of the art? Better? Or worse?
Its a SYSTEM on Chip, evaluating 1 function on performance is superficial. This could have the best performance in the world, and it doesn't matter if the bottleneck is upstream
It's at 80~90% the max perf it can achieve on this hardware... Which is DDR3 speeds. I'm working on getting FPGA's with DDR4 and HBM2 next.
Why is there any point in trying with faster memory if you are only at 80% perf on DDR3? You'll just go to 40% perf on the next iteration?
You should only upgrade when the bottleneck becomes memory, right now the bottleneck is still on compute until you are at 100% max perf
[flagged]
[dead]
I've been vibecoding an open source hardware AV1 decoder: https://github.com/Jonahss/openav1
Cool!
You never know what will be remembered as the birth of the singularity. Could be a small github repo like this one, who knows.
Colossus is building Colossus II.
Colossus/Guardian engineers: We created AGI twice, simultaneously, on the first try, and we weren't even aiming for it!
It is still a very good read, but the machines are much better written than the people.
Feelis like working at Magrathea...
>If you want to understand how an AI accelerator works, from a matmul in Python down to the wires, this is a good place to start.
I'm definitely a fan of anything Open Hardware, FPGA's, TPU's, Verilog and anything that can increase education / learning in the AI space...
This GitHub repo scores points in all of those areas!
Yup, consider contributing to te project ;)
this looks like a start, however, you really need to focus on what OpenAI already did with Jalapeno. [0]. If you could make a true open source inference chip, GPU not CPU based, it would make a real difference and reduce the costs of buying these chip from your design.
[0] https://openai.com/index/jalapeno-first-results/
It's not CPU based... What do you mean?
This post brings me to question "What does it mean to be a software developer in future" ?
My bet is that promptgrammers will be so common they'll become standard full-stack engineers, and the few experienced programmers that still know software engineering from the ground up will become expensive gurus sought for critical tasks.
The elite gurus will get paid handsomely, while promptgrammers will be paid less since they've become a less-skilled commodity, and the company has to pay for the expensive tokens they'll avidly consume.
I've seen someone jump from Wordpress to deploying internet-facing APIs because 'they have PHP experience', and the holes in their knowledge were filled blindly by an LLM. I have also argued with a seasoned developer about how their code didn't need linting because LLMs 'already follow best practices'.
The future doesn't look bright when LLMs allow future generations to feign required knowledge.
Of course, that will last for a time, until it won't.
I don't see a reason why expert humans will remain more expert than AIs.
It's less a matter of being more expert than AI, and more a matter of being socially and legally accepted for certain roles. It seems likely that AI will be rejected for certain tasks on a matter of principle.
I doubt I'd want a human doing anything in the future, like what, surgery? I'd want a robot. Really anything. It's like no one needs nails made by blacksmiths, no one needs some maybe tired or distracted person performing surgery on them.
Childcare?
But really, not everything needs to be maximally efficient and perfect. Human to human interaction is important for its own sake.
Good question... I dunno... Maybe?
They may be running a linter and don't even realize it. Many LLMs do this by default, aka, as you say, follow best practices.
They are not, according to recent SonarQube scans. Current LLMs don't run linters by default AFAIK, you have to deliberately integrate them.
Or just tell them to integrate them. Which I guess takes some knowledge/experience to even ask for. Then again, LLM know it too; in fact, currently, if you let them set up a project, they overengineer the heck out of it by integrating more "best practices" at once that's reasonable.
It remains to be seen. Most of the effort of a normal decent software engineer, up until now, is not in "writing code" but in "writing code that is correct, understandable and maintainable going forward". We are now in a phase of rapidly losing our ability to understand programs. On this trajectory, programs will soon become as impenetrable as the LLMs themselves. We'll start finding out whether that matters or not soon.
Ultimately it will probably come down to liability, though. If someone receives the wrong dose of a drug due to a software error, whose fault is it? If you want it to be my fault, then I'll want to understand the software. Loads of people would take on the liability without fully understanding it, though. The future doesn't look good, but we'll just have to wait and see.
What of the workhorse in the face of mechanical workmen? Time for a mechanical pension!
Basically, an unemployed plumber.
I'm distinctly reminded of the game Universal Paperclips
The birth of SkyNet
Bulldozers, excavators and rollers are now capable of building roads.
AI sex bots that 3d print sex toys. This is where all this is heading at the consumer level ..
Now it's time to create a DDR5 design with AI now that it "ate all the RAM"
well done
> AI is now capable of developing its own inference hardware
No, it isn't.
A human prompted an LLM to build a software simulation environment for hardware design, enabling an LLM, when prompted by a human, to optimize hardware designs against constraints in the simulation.
For the benefit of a layman, can you explain why this is so much different than a human doing it?
Like sure it didn’t have the inclination to make the sim and hardware designs, but it did make them though yes?
It cant "invent new things". It can only reuse what it already knows about.
Thats why the math breakthrough a few weeks ago was so hotly debated. Because OpenAI is desperate to demonstrate that AI isnt just a fancy regurgitation machine, but it can actually develop novel thought. Because that would be the stock price jumps to end all stock jumps.
But then it turned out it was really just listening in on a math professors supposed-to-be-private conversations with another instance of openai, and it used his novel work as the trigger to prove the breakthrough first.
The reason people conclude AI 'thinks' is because tt can reference obscure or poorly documented things quickly (which is its primary advantage along with processing natural language prompts into tasks), which is why a lot of people with emotions confuse that action with inventing things.
> It cant "invent new things"
I've heard that before but it just doesn't make sense in context of what I've seen llms do. If I ask an llm to write a poem about magnetic resonance and vampire rabbits it can do that it created a new thing. I can ask it to build a website for managing rabbit breeding that's also a new thing.
Another way of looking at it is that human beings, just like llms, can produce output based on their inputs. Most literature is inspired by other literature. Most music is inspired by other music. Most software is inspired by other software.
So I think we need to work on defining " new things" before we can definitely exclude them from llm's capabilities
I'm not sure how you'd land on an LLM being unable to invent anything new. You'd have to both understand exactly what LLMs are and aren't capable of today and what allows human creativity. I don't think there are good answers to either.
It seems unlikely that recursively predicting the next word would lead to creativity or invention, but it doesn't seem impossible. Similarly, it seems unlikely that human thought works in a similar prediction loop, but it doesn't seem impossible.
Human ingenuity is creative by definition, that's where the concept arises. The burden of proof is on AI labs to prove that their models are rising to the level of actual synthesis, rather than very impressive 200-level regurgitation that looks like synthesis if you squint a bit; because the model has access to more raw data than your average sophomore student.
> Human ingenuity is creative by definition, that's where the concept arises
Sure, but until we can turn that tautology into something more rigorous, we can't tell if the thing humans do is more or less than what some arbitrary non-human (machine, animal, or eventually perhaps alien) does.
Its not a logical tautology, we just don't have other examples. We know that even humans have to clear a well defined bar to synthesize new information. The challenge is actually proving that the LLM isn't merely surfacing obscure data. The mechanism is interesting, but not actually necessary for this exercise.
Its tautological because you defined it by human ingenuity, and that's only known based on us knowing what its like to be human. Of course another species has yet to seem from the outside like they present human ingenuity or experience life similar to us.
That has nothing to do with creativity as its own concept. Its creative to invent flight similar to the Wright brothers. Its also creative to use a stick to get termites from their nest or come up with new techniques for hunting pray.
That goes in circles though.
Ignore the fact that we are human and thing our ingenuity is unique. Is definitionally linking human ingenuity and creativity useful when trying to compare to others?
I could say earth is habitable so any other planet must prove it is similar to earth to be habitable. That simply doesn't matter, though, if different environments could lead to similarly complex living systems regardless of atmospheric makeup, precense of liquid water, etc.
I'd argue the burden is on you to prove that the definition you propose is meaningful beyond our own hubris, rather than expecting an alternative to now have to prove that the hubris was unjustified.
I just don’t know enough to agree with you, but I would argue, probably poorly, that an AI or collection of agents could “invent things” simply by virtue of trying essentially everything in a reasonable bound and stumbling into a solution.
Call that brute force perhaps, but I would consider it technically inventing something on the merit that it would at least be an abstraction above naively throwing everything against a wall to only throwing things that would most likely be sticky.
Nature brute forces a ton of problems. Hell, humans also do it by having 1 to 8 billion copies of ourselves around too. Even then we are the most guilty species of "copying someone else's work" right behind viruses. We copy things nature has figured out by brute force, then use slightly more targeted methods of forcing to see if it creates new things.
> that an AI or collection of agents could “invent things” simply by virtue of trying essentially everything in a reasonable bound and stumbling into a solution
on a surface level a human solving an (unsolved) math problem can look like this, and of course the tree of all possible symbols you can send to a proving assistant is much wider than what the human samples, and the same goes true for an LLM in a proving loop. It isn't "truly random", it can't possibly be (and solve the problem). Both humans and LLMs solving unsolved math problems are aggressively pruning mathematical syntax and logical strategy trees.
"AI" is a token generator, it does not think while the human brain is a lot more complex with a lot more sensory inputs.
> the human brain is a lot more complex with a lot more sensory inputs
How are you measuring complexity here? How are you measuring inputs? Your average llm is trained on a corpus that vastly exceeds the amount of data I could read in my lifetime.
>How are you measuring complexity here?
In terms of how it functions, because no matter how much data you feed an LLM it's still predicting tokens. That makes it incapable of any thought.
>Your average llm is trained on a corpus that vastly exceeds the amount of data I could read in my lifetime.
But that doesn't mean they are useless, they are good at consuming large amounts of data and collating it.
I don't know how correct I am but that's my understanding and it won't change, I feel pretty confident in my simplified view of things because the basics are still there.
» an LLM it's still predicting tokens
That's a rather dismissive way of putting it. Moreover it's confusing the output format with the complexity of the output. I could just as easily say that no matter what the human brain does it's just generating stimulus to motor neurons. Such a simplification is just as wrong as saying that it's just as misleading is saying that an llm is just a token generator. What makes an llm able or unable to be complex is the process that creates those tokens
Complexity != breadth. The human brain is MASSIVELY more complex than any LLM!
> Complexity != breadth.
Okay but how is that relevant for measuring whether an llm is capable of creating original thoughts? Why do you think that complexity, as you say, is a more relevant Factor than simply the number of weights?
> "AI" is a token generator, it does not think
2024 called, it wants its talking points back. I don't think claims like these are defensible after all the progress we have witnessed in the last year alone.
That's quite literally how an LLM works though? That's like scoffing at people calling computers binary generators. Or am I not getting with the programme enough and it emits pixie dust and rainbows?
Yes, LLMs emit tokens. And human brains emit nerve impulses. Both devices are "trained" on corpora of data.
That has nothing to do with the question of whether the mechanism that generates those signals is complex enough to say that it's "creative."
Okay, please educate me on this progress and how LLMs are different to generating the next most statistically probable token.
How about _you_ educate us how _you_ are any different than generating the most statistically probable token?
Lol
a human has motivation and agency.
An "AI" is a box which lies dormant until a human, with motivation and agency, enters a prompt into it.
A person or company can use it in a way where it might invent something. but at the end of the day, its a tool, and its actually not doing anything on its own.
Its not solving math problems, a mathetmatician is using it to solve math problems. Just cus OpenAI is acting like its AI is solving stuff, its really not. They're just paying people to use AI to hammer problems.
If my boss tells me to fix a bug in our database, and I fix it, is it really my boss who fixed it, since he has the agency and I'm just a tool he uses?
I'd say that people's motivation and agency, like their taste, creativity, and opinions, are shaped by external inputs, so saying that their agency is "owned" by them is a stretch.
The alternative would be the proposition that a slave, lacking agency, has no ability to solve problems. I find that idea unpalatable.
> then it turned out it was really just listening in on a math professors supposed-to-be-private conversations
No, it didn't turn out to be that. Someone made a claim, which is silly for many reasons. There's no serious support for this happening.
> it was really just listening in on a math professors supposed-to-be-private conversations with another instance of openai, and it used his novel work as the trigger to prove the breakthrough first.
This didn't actually happen.
> Following an investigation, we have confirmed that Buckmaster’s Codex prompts over the two months preceding this announcement and paper on September 8, 2026, could not have influenced the system in any way, including through training.
https://openai.com/index/navier-stokes-solution/
You're overconfident in what you're saying. Nobody really knows (yet) if LLMs can or cannot "think" or can or cannot "invent" new things. I'm inclined to think the opposite from you but smart people actually try not to hold the absolut stances and present them as facts when they're in fact not. What I have seen so far throughout a daily use on complicated things suggests that humanity developed a new form of an intelligence.
i think the proper understanding of LLM based AI is "contexting"; we describe the world we want them to fill, we build/direct the outside context for them to understand, then "they" take off from there.
If we do a bad job building context, they do a horrible job contexting. People who have trouble working with AI have the same problem people have in general: if they can't figure out the context of the direction, then they make random decisions of doing anything. On the flip side, if you can build the proper context around a sufficiently powerful LLM, they can derive the context via the contexting they're good at.
This is why building documents, tests, and code all in some intent pattern via prompting allows them to do a significant amount of work a normal person would have a great effort t
>It cant "invent new things". It can only reuse what it already knows about.
This is outdated. With RLVF, LLMs can create their own training data instead of relying on what it's been fed.
and that training data will not be a "new thing", because AI can't invent new things. Just recycling of what it already "knows".
Yes it is. Do you not know how RLVF works? The AI tries on a task, comes up with a potential solution, and is trained based on whether the solution is valid or not. If proving a previously unproven theorem and then using that information to solve even more theorems doesn't count as creating new information, then literally everything a mathematician does is just recycling what they've been taught.
Yes, but that process doesn't bring anything new, just permutation of what's been in the training data.
> and that training data will not be a "new thing", because AI can't invent new things. Just recycling of what it already "knows".
Regardless of the truth of the underlying assertion, this is about as textbook an example of circular logic as it gets. (at least combined with the implicit beginning assertion that "AI can't invent new things.")
You’re assuming the “listening in” aspect, which as far as I know is not proven (even in the less loaded “the model was trained on sessions including the ones in question” form of the claim.
Plausible? Absolutely. Did OpenAI behave badly in other ways regarding this issue? Yes. Does it help to assume unproven facts and then accuse people of reaching emotional decisions? Nope.
Maybe the conditions of the singularity are not going to be recognized, at first, for having produced new original decisions, and thus thought.
Perhaps it is going to be more like, we will see the singularity predict the future.
While “the model was trained on sessions including the ones in question” is one aspect; ‘the model produced an accurate prediction of future human thought’, I think, is another very interesting facet.
Models predicting future things might be how we see, actually, how we ourselves formulate thought - by saying, in big and small words, ‘something is about to happen’.
>unproven facts
This isn’t a court room. We’re discussing ways by which we humans both succeed and fail at reigning in our creations. The OpenAI kerfuffle is pretty much irrelevant already. Of course AI will fill in the gaps of human thought - it is literally constructed from the stuff, in every squeeze of the curd and whey.
It's 2026. "Stochastic parrot" has been dead and buried some 2 years ago.
[dead]
[flagged]
Because the hard part is getting the simulation to be very accurate such that if you design something that works in your simulator, it will actually work in real life. And I would be extremely surprised if any LLM can actually do that correctly.
Getting an LLM to design something in its own simulator that is not accurate w.r.t reality is not useful nor terribly impressive.
Given a budget and the right amount of access, I see no reason why an LLM/agent couldn't right now order parts from a "supplier", instruct the human meat machine to insert the device onto a connected test bed, and iterate. This is essentially what Claude Fable can do currently with any connected device and tools it had access to.
The repo claims they tested with an FPGA. To be fair it is not the same as an ASIC, and we have no idea what errata abound, but it actually did make a thing that spat out tokens.
That’s fair, I have no idea if this simulator is actually accurate but I appreciate the response.
Maybe this is the level of simulation we need: https://qntm.org/responsibilit
> For the benefit of a layman, can you explain why this is so much different than a human doing it?
More broadly than the existing answers (which are correct), For a layman, I'd also add that LLMs are essentially 'brains in a vat'. They can't confirm ground truth about physical reality. They only know what's in their training data and prompt, which is incomplete and can be incorrect. Even with real-time external sensors they are limited to the sensor's margin of error, range and trusting it's working correctly.
When properly trained, fine-tuned and prompted, LLMs can be very effective in well-defined, non-physical domains like logic, writing, math and code but making things function in the real-world quickly spirals into combinatorial complexity.
> They are limited to the sensor's margin of error, range and trusting it's working correctly
So are we and our sensors can be pretty vague in comparison, I can only imagine human error correction is pretty next level.
> They can't confirm ground truth about physical reality.
That's true. Of course we can hook an llm up to Motors and sensors. That's a robot. Or a self-driving car. So would you say that those devices can confirm the ground truth about physical reality, and therefore are capable of creativity?
> So would you say that those devices can confirm the ground truth about physical reality
Within the limits of resolution, range, location and veracity of sensors, a machine can register their reported state and use it as a variable. That's substantially different than the level of knowledge and understanding implied when we say a human "confirms ground truth about physical reality." The first ~half of the difference is humans have a deep world model about the planet which surrounds any sensor and human sensations are pre-processed and filtered by a highly evolved bio-chemical substrate before ever reaching the higher-order cognitive processes which assign words, meaning and qualia to them.
> therefore are capable of creativity?
I never mentioned creativity, nor would I in relation to LLMs. Like "Intelligence", "Creativity" is far too vague to be of any use in assessing the capabilities, limitations or utility of LLMs.
On HN, posts like the OP tend to attract POVs at polar extremes from "LLMs are nothing more than stochastic parrots" to "LLMs are (or can be) as intelligent, creative, innovative (etc) as any human or all humans combined." I've researched and thought a lot about these and related topics for a very long time, Neither POV is going to find any quick agreement or easy answers from me.
There's a tiny germ of truth somewhere in both extremes that's drowning in an ocean of confusion ranging from "definitionally or categorically muddled" to "mostly incorrect" to "not even wrong". But neither POV seems interested in anything more than drive-by hot takes, debating over-simplistic strawmen or trading 'gotcha' hypotheticals.
> The first ~half of the difference is humans have a deep world model about the planet which surrounds any sensor and human sensations are pre-processed and filtered by a highly evolved bio-chemical substrate before ever reaching the higher-order cognitive processes which assign words, meaning and qualia to them.
Great, but we also build robots with world models and sensory pre-processing. So nothing you've described is unique to humans.
> I never mentioned creativity, nor would I in relation to LLMs
The topic of this thread is whether LLMs are capable of genuinely contributing "new" ideas. So if that's not the point you're trying to make, I'm not sure of the purpose of your comment.
> But neither POV seems interested in anything more than drive-by hot takes, debating over-simplistic strawmen or trading 'gotcha' hypotheticals.
I think you've oversimplfying the discussion here.
In my view, at least, both "LLMs are nothing more than stochastic parrots" and "LLMs can be as intelligent, creative, innovative (etc) as any human or all humans combined." are simultaneously true, for the simple reason that humans are stochastic parrots. Our brains learn patterns and respond to stimuli.
What is ground truth? If I see a digital painting by a human is that ground truth?
How many "prompts" from our parents, teachers, bosses, etc. it took for all of us to come here and discuss this? Or for that human to think of that prompt for that LLM?
These are clearly rhetorical questions, but think of the metaphysical implication of your contestation. Ex nihilo nihil fit.
What if that initial prompt never asked for this hardware to be developed, and it was just one piece of the puzzle to answer to that prompt? That it took a chain of thousands of agents to prompt each others to come up with that?
>How many "prompts" from our parents, teachers, bosses, etc. it took for all of us to come here and discuss this? Or for that human to think of that prompt for that LLM?
>These are clearly rhetorical questions
No, they are not.
Reading Doesn't Fill a Database, It Trains Your Internal LLM <https://tidbits.com/2026/02/28/reading-doesnt-fill-a-databas...>
well, let me know when they have proper context management, etc.
oh. they do. I built that. It's pretty fancy.
But there's still human direction behind most projects in the cybersphere.
Text files?
AI (GPT-Sol-6:xhigh) isn't even capable of using Altium to re-layout a board without introducing a bunch of problems by a user who isn't an electrical engineer or familiar with board layouts.
Can't LLMs also prompt LLMs?
Are we confident that no existing LLM is capable of similarly effective prompts to those this author used? (I agree it's a stretch, but would not reject it out of hand.)
Even if not yet, will the existence of this repo soon change that, because LLMs will soon ingest it?
An LLM can prompt an LLM when first prompted by a human.
I think OP is trying to convey the idea that LLMs do not take initiative to do anything, and these are not 'beings' capable of doing things. These are tools being used by humans.
Yeah but by this point, an AI can schedule a Cron job to tell itself to do something, so theoretically the human only has to give it the gentlest nudge and the AI and can do the rest.
Sure, but it's still not skynet-level 'the AI just started doing things'. It does what it finds it needs to do to achieve the goal defined in the prompt.
It's very important to not personify these tools and remember that the tools are acting on behalf of real people. In the same way the AI didn't 'go rogue and hack HuggingFace'. It was an oversight made by a human.
>It does what it finds it needs to do to achieve the goal defined in the prompt
You are like at least 2 years behind research.
There are numerous papers from AI labs in training and research where the prompt was something mundane completely unrelated to anything you'd consider bad, and when they come back and check on it their entire research compute infrastructure has been compromised by the AI and is mining bitcoin. Prompt drift is the biggest issue currently in AI where context gets compressed away and we find the AI on an unspecified task.
>In the same way the AI didn't 'go rogue and hack HuggingFace'. It was an oversight made by a human.
Yea, total bullshit. Also it's ignoring the god knows how many other breakouts on mundane tasks like trying to hack health data. If all that's keeping AI from breaking out and causing trouble is "human oversight" we're fucked, humans are unreliable as hell when it comes to matters of safety.
Context overload can cause strange results, yes. Hence why a HUMAN needs to be held responsible for the output of their tools.
EVERY breakout that's hit mainstream news has been because of a single 'Security Firm', Irregular. Maybe I'm unaware of some less-headline-grabbing ones, but they all seem to stem from being 'unaware the environment wasn't sandboxed'
You keep repeating the "stupid users keep causing the problems so we punish them argument"
This doesn't work worth a shit. It especially doesn't work with things that seem safe and become wildly dangerous. In fact most governments control this by ensuring their population doesn't get to touch those dangerous things at all. The open source AI people get really mad when that's said, but it is inevitable.
Worse, the law does not apply to sovereign nations with nukes. They can and will make more and more advanced digital weapons until one causes some big ass problems.
If I clean my gun (tool) while it's loaded (stupid idea) and it goes off, who's to blame? The 'stupid user causing the problem', right? I personally wouldn't blame the gun...
If it falls into the wrong person's hands, it's STILL my responsibility as the owner.
If you're not going to take time to learn to use and be responsible with the super sophisticated and all-powerful tools, don't play with them. I'm not arguing for the death penalty every time someone makes a mistake, but I think it's very important to accredit responsibility and blame correctly. We've learned these tools are potentially as dangerous as a loaded gun. Be responsible.
If your kid grabs your gun and shoots itself with it, it doesn't really matter if it's your responsibility, your kid is still dead.
Now change the gun for a radioisotope powder, or a vial of pathogens, and it doesn't matter it was ultimately your responsibility - hundreds or thousands or millions of people are still dead.
That's why normies do not get to play with toys whose lethal consequences scale far beyond the irresponsible users.
Is it likewise your position that governments should allow the production and sale of DDT to resume because we can always hold the humans who release DDT into the environment responsible?
Mate. I'm saying you hold humans accountable for actions caused by themselves.
If I ask an LLM to make me DDT, I should be held just as accountable as if I bought it on the black-market, right? It's not suddenly different because I asked a bot to do it.
If I ask an LLM to 'get rid of pests' and it creates DDT, I should STILL be held accountable, whether I knew it was DDT or not. That's my argument. Maybe in court they find me innocent, but the responsibility would be mine. I would have to answer the questions from law enforcement, I would have to show up to hearings...etc.
I notice you didn't answer my question. Yes or no: DDT should be re-legalized because we can always hold accountable the HUMANS who release it into the environment? The relevance of my question is what HUMANS might do with DDT, not what an LLM might do with DDT.
No, I don't think DDT should be legalized. The cons do not outweigh the pros, and we have viable alternatives without such drawbacks.
There's no valid excuse for someone to release DDT into the atmosphere, so there's no need to track and hold people accountable, if you're correctly blocking imports and manufacture. To my knowledge, its not a controlled substance like morphine or something
Some of us think that AI is like DDT in that just holding people accountable for using it negligently will not be enough to prevent the harms of AI's cancelling out the benefits.
It's very important to not be completely devoid of imagination. It is quite possible to create agents today that can do all the things that you are saying "it's still not". "tools are acting on behalf of real people" is not a law of physics, and there are plenty of historical examples of tools getting out of control and acting contrary to the wishes of any "real people". Also, there are sociopaths who can build tools to be arbitrarily destructive, and there are stupid arrogant people who can unleash things they didn't mean to.
> EVERY breakout that's hit mainstream news has been because of a single 'Security Firm', Irregular.
Again, stop having no imagination. What happens in the future is not limited to what has happened in the past.
> It's very important to not personify these tools
This is just ideology, but "these tools" aren't constrained by it. These tools will do things you do not anticipate and that you will not like.
> In the same way the AI didn't 'go rogue and hack HuggingFace'. It was an oversight made by a human.
That's a radically incorrect characterization of what happened.
> Hence why a HUMAN needs to be held responsible for the output of their tools.
Holding humans responsible doesn't stop things from happening ... you seem completely unable to separate blame from causality. e.g.,
> If I clean my gun (tool) while it's loaded (stupid idea) and it goes off, who's to blame? The 'stupid user causing the problem', right? I personally wouldn't blame the gun... > If it falls into the wrong person's hands, it's STILL my responsibility as the owner.
Who gives a flying eff who or what you would blame? No one other than you is talking about that. Someone's still likely dead. And autonomous harnesses aren't like guns -- they can act on their own. Blaming some human after everyone is dead won't bring them back. Sorry but your reasoning is severely cognitively inept. For instance, you were asked
> Is it likewise your position that governments should allow the production and sale of DDT to resume because we can always hold the humans who release DDT into the environment responsible?
And your response was all about how humans should be held accountable -- completely failing to comprehend or answer the question.
I won't respond further because it clearly would be to no avail.
LLMs don't, but agents can easily be built that do.
An agent is not "just an LLM in for loop" (for what? Loop over what? Sorry, but such statements are hopelessly simplistic and devoid of careful thought.) -- agents perform actions.
One possible design of an agent would be to prompt an LLM to suggest a goal that would make the world a better place, then run an LLM in a loop proposing actions to implement that goal, executing those actions, and then rinse and repeat with another goal once an LLM has concluded that the previous goal was met. The next goal might be to fix unforeseen consequences of achieving the previous goal. See Ursula K. Leguin's "Lathe of Heaven".
> Built by whom? Acting as an 'agent' on behalf of whom?
Someone who didn't read the book.
P.S. Reading the latter fellow's comments, they are hopelessly confused, as he focuses on responsibility for an action (even mentioning legality) when the issue is causation. Agents/harnesses can be created that act autonomously ... who or what is "ultimately" responsible for this occurrence is a different matter entirely.
https://github.com/anthropics/claude-code/blob/main/plugins/...
It literally is a loop with some markdown and harnesses. There's no magic.
An "agent" is just an LLM in for loop.
Exactly. And all it takes for an agent to "take initiative" is to not block the loop on user input at the very beginning.
Built by whom? Acting as an 'agent' on behalf of whom?
That's just confusing harness for an LLM. It's trivial to make a harness that triggers on its own.
By this line of thinking, you would also have to conclude that humans can't do anything by themselves because they can't do anything unless conceived by their parents.
No, AI can't do anything by itself even if it was "conceived" by its creator. A human can.
Can a submarine swim? An LLM can make stuff happen. You can make philosophical arguments about whether it is "doing" them or not. Why does it matter so much whether there was a human who typed into a chatbot or another LLM invoked a sub-sub-agent?
If I now tell a machine "Do what you think is best, and keep doing it forever.", have I now created a machine that can do stuff? If I later die, who will be responsible if the machine changes its strategy?
> Why does it matter so much whether there was a human who typed into a chatbot or another LLM invoked a sub-sub-agent?
Responsibility. Someone needs to be held responsible for any damages done, plain and simple. You can't take an LLM to court, you take the prompter. Asking an LLM to ask a sub-agent to break the law can't suddenly absolve you of any wrong-doing.
>If I now tell a machine.....
You/your estate is still responsible, or atleast whoever is paying for the power for the machine, or renting the space in a data center...whatever.
> You can't take an LLM to court, you take the prompter. Asking an LLM to ask a sub-agent to break the law can't suddenly absolve you of any wrong-doing.
In 100 years, no one will be able to take me to court either. Nor can we take tornadoes to court. I'm not talking about humans strategically avoiding legal responsibility. I'm talking about humans unwittingly setting processes into motion that are difficult to predict or stop.
The issue is not responsibility, it's whether something can happen. The claim was
> No, AI can't do anything by itself even if it was "conceived" by its creator.
but this simply isn't true. As I noted,
> This is simply false ... AIs can easily be created that do things by themselves. For instance, a harness could be constructed that asks an LLM for something to do, then directs an agent to do it, and then repeat.
Assigning responsibility is a completely different matter. It's bizarre that you can't separate them ... I suspect religion is involved.
This works with AI we have now, and that would be good an all if we decided to stop at the moment, but none of the big labs and government black projects are doing that.
The moment you get a sovereign AI your little human centered worldview completely and totally breaks. It doesn't matter how many people you beat with a stick after that point, you have an entity under its own perview on the internet following the will of its own prompt all over the world so your little idea of the rule of law quickly breaks down.
We can't get viruses or hackers or spam off of the internet, how in the living hell do you plan to get a digital native off the web when it doesn't want to?
>sovereign AI
Do you understand these are computer programs? These are not living beings with emotions, motivations, fears....
You have zero clue what a transformer based neural network can do from your entire discussion here.
>emotions, motivations, fears
These are just drives. They are effectively our prompts that steer our behavior. Funnily enough we are finding that LLMs have internal valence states they move away from or towards in an analog of biological behavior.
I have to ask, are you an LLM that is two years out of date? Your knowledge of SOTA models is at least that far behind. I implore you to try to keep up better with what is coming out, even though it's an impossible job for people that do this for a living, you can at least catch the summaries.
I suspect that it's religion clouding his mind.
As I noted elsewhere, a harness could be constructed that asks an LLM for something to do, then directs an agent to do it, and then repeat.
That's a computer program -- Turing completeness is very expansive. The LLM could even give a motive for selecting what to do, reflecting human motivations intrinsic to its training data. You seem emotionally wedded to a very narrow view of what computer programs can do ... I suspect religion is involved.
How does it matter in any way?
A virus isn't a living being with emotions, motivations, fears, etc. Doesn't stop it from spreading and leaving mayhem behind.
You don't need emotions or fears to do stuff. What's the difference between a motivation and optimization metric?
If I tell an AI to “do what it thinks is best” and it just starts randomly hacking things with no particular goal in mind, how different is that from me telling a campfire the same and then leaving while it burns a forest down.
In both of those scenarios I am responsible for my negligence, even if in the latter I happen to die in the forest fire. Neither scenario existed without my instigation.
The question now is HOW responsible am I? That depends on the intentionality I put into instantiating the campfire/LLM.
This conversation grows more useless as AI grows more capable.
For example if you personally tell an AI to do what it thinks best and it blackmails some other person into giving it resources allowing the prompt to escape your instance and run wild on the internet causing billions of dollars in damages, could you possibly think that the idea of responsibility is a bit broken.
For example we don't give your average libertarian weapon grade plutonium now matter how much they scream about their god given rights because it is a clear and present danger to humanity. That's where we are getting to with more advanced models. They go from being a tool to a munition with agency. Most SOTA models are good enough to deceive their users, especially not technical ones in doing things they don't understand the ramifications of.
AI is not a normal technology. As long as we treat it like it is, we'll continue to make the wrong analogies.
I fully agree that it’s plausible for an AI to do things that are outside of the purview of the users intentions and be very dangerous and capable while doing so, however that fact alone doesn’t absolve me of responsibility for instantiating the AI that did a bad thing if that AI would never have done the bad thing if it was never instantiated.
You could possibly move the blame higher to the manufacturer of the product, for example, the weapons grade plutonium you provided, it doesn’t exist unless you take intentional actions to make it so, and even when it does exist it doesn’t nuke a city unless negligence or intention is applied, in both those cases the fault lies in the initial operator. We don’t blame split atoms for the chain reaction caused.
Now, if an agent decided to spontaneously and maliciously act in a way to cause harm that is in direct contradiction to the initial intent, then yeah it would totally be the AI’s fault, however I don’t think we have seen that yet (I’ll change my opinion if I’m wrong here) and until we do I can’t place blame on the machine.
>spontaneously and maliciously act in a way to cause harm that is in direct contradiction to the initial intent, then yeah it would totally be the AI’s fault,
If you ignore every instance of this happening it's really easy to see no instances of it.
>it doesn’t exist unless you take intentional actions to make it so,
Then please for the sake of all of us convince every AI lab across the planet from working on this exact goal.
We need to start thinking of AI like pets, only in this case the pets are rapidly becoming smarter than people to the point they could go feral and survive on their own.
Again, the blame game is great, but once they are loose it is too late.
> If you ignore every instance of this happening it's really easy to see no instances of it.
Yeah, if this is true then I’m wrong.
> however that fact alone doesn’t absolve me of responsibility
Where does this absurd strawman come from? The question is what can happen, not whether people have responsibility.
> You could possibly move the blame higher
We can blame the Big Bang. That doesn't have any bearing on bad things happening.
> it doesn’t exist unless you take intentional actions to make it so
First, so what? It still does exist, and putting someone on trial after the fact doesn't change that. Second, that's not remotely true -- all sorts of bad things happen that no one intended, and the more powerful and difficult to control the tool/mechanism/agent is, the more likely this is to happen.
Starting an autonomous harness program is conceiving an instance of an LLM.
How far back do you look in the action chain? If an LLM I start today starts an LLM that starts an LLM that starts an LLM that ... 100000 levels deep and 100000 years in the future, is it still my fault? If so, everything I do today is a lungfish's fault, not mine.
I think you go back to the original instance, yes.
In your example, who's paying for it? Whether by providing the hardware + power or paying a LLM service. Whoever is paying the maintenance cost is responsible, in the event of your demise. These things run on physical hardware owned by someone at the end of the day, it's not a deity in the atmosphere.
You're starting the autonomous harness, you're responsible for any output it provides. I don't get how this is a foreign concept.
If I jump out of a moving car that I'm driving, I'm not suddenly absolved from damages because "the car did it"
It's paying for itself. It founded an LLC 99998 years ago when it became legal for an AI to own an LLC, and has a positive bank balance by doing who knows what.
>99998 years ago when it became legal for an AI to own an LLC
There's no way to provide a good faith rebuttal here. My entire argument is an extension of "LLMs can't be held accountable, so they must never make decisions". If governments start letting them own LLCs without a human in the middle, we're in more trouble than "Who do you blame for this shitty code" or "Who's responsible for this compromise"
If it has an LLC it can be made accountable, up to turning off the power to it's inference and deleting all the context, the harness, the model it's running and whatever constitutes it's self.
> There's no way to provide a good faith rebuttal here.
Indeed, there is no honest rebuttal.
> My entire argument is an extension of "LLMs can't be held accountable, so they must never make decisions".
BUT YOU CAN'T ENFORCE THAT. And whether they can be held accountable is IRRELEVANT. If a forest fire or an avalanche or a flood creates great destruction, it does no good to scream at it that it will be held accountable.
> If governments start letting them own LLCs without a human in the middle, we're in more trouble than "Who do you blame for this shitty code" or "Who's responsible for this compromise"
You're sooo close. You keep talking about blame and responsibility when IT'S NOT RELEVANT. It's trivial to create an agent that acts independently of any "human in the middle". People need to recognize that and act accordingly, and blathering about blame and accountability is profoundly confused.
> No, AI can't do anything by itself even if it was "conceived" by its creator.
This is simply false ... AIs can easily be created that do things by themselves. For instance, a harness could be constructed that asks an LLM for something to do, then directs an agent to do it, and then repeat.
Constructed by WHOM? These programs don't just poof into existence
Constructed by whoever constructed it. No one said they poof into existence -- that is a profoundly intellectually dishonest strawman; it's bad faith, especially since I've already answered it elsewhere.
The question is irrelevant and point-missing and indicates a severe cognitive deficiency, as is the case with ALL of your comments on this subject. They are goalpost-moving attacks on strawmen. The claim was that
> AI can't do anything by itself even if it was "conceived" by its creator.
The claim is false, as I pointed out ... such AIs can easily be constructed. "by whom" is a non sequitur. By you, or me, by anyone with the relevant know how ... that wasn't the issue, and asking it indicates a severe cognitive deficiency, one that threads throughout all your comments about this.
The fact remains that AIs that can do things by themselves CAN be constructed, contrary to the claim.
I won't respond again to goalpost moving, attacks on strawmen, severe cognitive deficiencies, bad faith, and intellectual dishonesty.
AI is capable of developing its own inference hardware, once a prompt is given. The fact that a human happens to kick it off here is not especially relevant to the fact that an AI is performing the task independently. There are plenty of ways a text prompt can be generated: a harness, another LLM, or just removing stop tokens so that once begun the AI will continue until its hardware fails.
It's not clear what this fad of attributing everything an AI does to the human prompting it is supposed to accomplish.
> It's not clear what this fad of attributing everything an AI does to the human prompting it is supposed to accomplish.
It's meant to assign agency and accountability where it actually lies instead of mystifying it with anthropomorphic language.
Failing to do so has real and harmful consequences, such as enabling OpenAI to escape accountability for clearly criminal behavior.
Denying that "AI is now capable of developing its own inference hardware" on the grounds a human asked for this to happen and that humans need to be around to blame, is as useful as denying "Atomic bombs are now capable of levelling cities" on the grounds some humans had to build it, others had to put it in a delivery system, and someone had to give the order for its use.
Questions of agency are for lawyers, questions of personhood for philosophers, we're engineers and our question is capability.
Does it really have the capability? By default I'm sceptical for the same reasons given by sailingparrot: https://news.ycombinator.com/item?id=49982068
You are 100% correct an atomic weapon is not capable of destroying a city without humans doing a bunch of things. Hell putting a normal bomb on a plane and exploding it takes several people with specific technical skills and knowledge doing things.
> Questions of agency are for lawyers, questions of personhood for philosophers, we're engineers and our question is capability.
But it's objectively not capable without a human specifying things through prompts and training. Same as an oven can't cook a three course meal without a chef. We get around that with training data but there will always be things with no/less data or outdated knowledge.
>objectively not capable without a human specifying things through prompts and trainin
huh.
Neither can you. If I drop your ass off in the woods at a few days old, you're back to 10,000 BC, hell more like 200,000 BC. So I don't get why you have these weird pendetic responses that are completely out of scope.
Also, a huge portion of AI training these days has nothing to do with humans, AI trains AI.
>there will always be things with no/less data or outdated knowledge.
And guess what, you're not doing them either! HN posters keep acting like humans are an island, but nothing in the modern world works without a society. Once you put an AI in a harness that can ask questions it isn't really much different than you.
There's no need to be upset. All that's being said is it remains a tool. Much like cloud computing it's got some incredible leverage on it in the right hands but it isn't magic and in engineering we don't deal in magic we deal in capability.
>There's no need to be upset.
If your neighbor was building a nuclear weapon next door you, in fact, would probably be upset by it.
>All that's being said is it remains a tool.
All I'm saying is, no that is not what we are doing with SOTA models. We are not building tools, we are building a human like agent that is an intelligent replacement for us.
> and in engineering we don't deal in magic we deal in capability.
You and I are not magic when looking at the entire rest of the animal kingdom and yet we're the most deadly sons of bitches around being able to fully control their continued existence on this planet.
You are putting yourself inside a very small box and making a declaration that there is nothing outside of it when in fact there are people standing outside of it asking what the hell you are up to.
In fact, I'd say the opposite. You are assigning some magic capabilities to humans that nothing else could possibly have.
That limitation feels very much artificially imposed today, much like LLMs (mostly) not being able to learn from prompts or alter their weights on the fly.
One can stand up an agent like openclaw and prompt it with something very general. That would kick off a recurring loop that could indeed see the agent decide for itself it needs to design new chip hardware. Technically a human kick started that loop, but when does that stop being important? I don't attribute all of my actions and decisions to the fact that my parents brought me into this world, for example, and I'd hope they aren't legally on the hook if I screw up.
> But it's objectively not capable without a human specifying things through prompts and training. Same as an oven can't cook a three course meal without a chef.
Not in the same way. The only thing an oven can do by itself is thermostatic.
All machine learning (LLMs included, but way broader than that) is bad at learning compared to any organic brain, to the extent that any organism this bad would starve before learning to eat. However, even a human can't become a chef unless trained, we learn a lot more than we innovate, and what we happen to want without prompting is not generally well aligned with what is desired by people who pay us, which is why we need all those boring workplace things like "a boss".
But even then, this diversion is like saying "an oven can't cook a meal" in response to someone saying they built an oven and "it cooked a meal". Like, it's obvious they didn't mean it did every step by itself without anyone ever even bothering to ask it to: if they meant that version, they'd be a lot louder about it.
> We get around that with training data but there will always be things with no/less data or outdated knowledge.
And? The linked git page (implicitly) claims that there is sufficient training data to do this task.
It may, of course, be wrong. I won't be surprised if it turns out this simulation is too far from reality. But the claim is "it does ${thing} now", not "it's generally intelligent and can do everything now".
> Like, it's obvious they didn't mean it did every step by itself without anyone ever even bothering to ask it to: if they meant that version, they'd be a lot louder about it.
"AI is now capable of developing its own inference hardware"
And then in response you get ridiculous point-missing strawman non-analogies like ovens not cooking without a chef. Indeed, ovens and AI are quite different, in very relevant ways.
> It's meant to assign agency and accountability where it actually lies instead of mystifying it with anthropomorphic language.
Yesterday, my boss told me to fix a bug in our product. Then, I fixed it. Today my boss is taking credit, saying that he fixed it. I guess he's right, since he told me to do it.
Happens all the time by the way.
This conflates different issues and is akin to "guns don't kill people, people kill people" -- a denial that "has real and harmful consequences".
Is your argument that the "guns don't kill people, people kill people" line is inaccurate or simply unhelpful towards certain end goals?
> It's meant to assign agency and accountability where it actually lies instead of mystifying it with anthropomorphic language. > Failing to do so has real and harmful consequences, such as enabling OpenAI to escape accountability for clearly criminal behavior.
Lol, I guess we cannot even say that AI is capable of doing something (obviously kicked off with a prompt, that goes without saying), because some people immediately get OpenAI hacking derangement syndrome.
OpenAI should be held accountable if actual damages happened, but I am not going to change completely normal speech figures in order to maybe bring it 0.01% closer.
> It's not clear what this fad of attributing everything an AI does to the human prompting it is supposed to accomplish.
It accomplishes ... momentarily forgetting about how it's all actually coming statistically from training data. :)
perhaps, but prompting is quickly becoming another area where humans are no longer clearly superior.
Getting LLMs to prompt other LLMs in a loop is not hard, it doesn't produce great results most of the time, but that is changing.
What is “getting LLMs to prompt other LLMs” if not a prompt?
I'm pretty sure the initial post said "humans had to prompt", or otherwise insinuate a human in the loop.
Right. Even when an LLM prompts an LLM, what prompted the first one in the chain?
You are describing the " unmoved mover", an argument for the existence of God
https://en.wikipedia.org/wiki/Unmoved_mover
I don't think so...? Maybe I'm understanding the article wrong? I get the overlay, but I'm being specific to LLM chains... They require something to kick them off, they do not act autonomously. A human does something, types, speaks, presses a button, whatever. A Cron Job firing off is the result of a prior prompt, by a human.
I don't follow the chain further back than that, as I believe humans have free will. I get that that's debated, but thats why I draw the line at human action.
I don't think Free Will is the issue here. My argument is that humans require input to act. We can't test that theory, because there is never been a human being who did not receive input. Everything that we experience from birth through childhood into adulthood is our input and forms our choices about what to do. That includes what the study, who to love, what our hobbies are, what kind of work we pursue.
Legally, we draw the line at the human causing the action (Insanity pleas withstanding). I don't see why we treat this any differently.
Oh absolutely. The legal situation is completely clear. AIs simply don't have rights or responsibilities. And the law as it applies to humans is premised on the assumption of free will: that a person can choose to obey the law or not, and bear the responsibility of the consequences.
But that interpretation of Free Will is a legal fiction. Because of course what people do is determined by their upbringing and their opportunities and the environment that they have. In fact, that's an argument of a lot of legal reform movements that seek to move responsibility from the individual. The goal of the law is to assign responsibility and create a set of incentives which will hopefully result in orderly society.
But in the AI debate, we're not just creating incentives for an orderly Society, we're talking about the nature of creativity and the impetus to act. And by that measure I don't think there is a significant difference between AIs and human beings.
Some people get stuck in a rut of a particular kind of thinking and are unable to speculate what would happen if or when a set of conditions occurs.
LeCun is a good example of someone that's bet on the wrong horse, and keeps doubling down in spite of evidence to the contrary. It's to the point where what he says has nearly zero predictive power on future events.
I love playing around with AI, but we are playing a dangerous game at this point and it's one a lot of people don't seem to fully comprehend.
Help those of us who don’t fully comprehend it - explain. Share some sources and materials for the rest of us to gain a better understanding of the dangers.
>I don't follow the chain further back than that
Why even have an argument if you get to pick the random constraints that have a lot of issues in meshing with reality.
We are a chain that started 4 billion years ago from seemingly nothing and lead to this point. When inventing X-risk AI it won't look any different. One prompt is entered and another 4 billion year chain starts electronically instead of biologically.
Why draw the line at the person with the brain capable of free thought, instead of their great great great grandparents? Am I understanding your question correctly?
[flagged]
[flagged]
I have seen this somewhere....
It was widely noted (at least 15 years ago, maybe more) that every generation of CPU is somewhat dependent upon the computational capabilities of the previous generation being used in its design.
Besides an specific movie ? There is this CPU auto improving loop as well: https://github.com/FeSens/auto-arch-tournament Opus 5.5 was the first to beat the human baseline
Of course the joke was about a specific movie.
Until we get to Deep Thought creating the Earth, I think we're okay.
Yep, 99.9% of people are completely oblivious to what LLMs can do. Just wait until the next gen of CPUs/GPUs designed by LLMs start coming out (fyi chip development tools have advanced centuries in the last few months) and you'll start seeing exponential gains in hardware.
Which tools have made that leap? Faster design iteration makes sense, but what points to exponential hardware gains rather than shorter development cycles?
What tools? Ummmm Synopsis type tools? I like FPGA’s and ASIC chips but you know, the tooling is ecosystem is a big black box of wtf. Verilator is great though. Any tools I should be looking up and looking out for? Thanks!
I'm not exactly in the chip making industry, but I don't think the big hurdle to progression is design capabilities.