"If AI inference remains as desirable as Kimball expects, the evolution is likely to follow the same trajectory as the CPU. The CPU didn’t improve along a single axis but instead across simultaneously. Once transistor scaling slowed, chip and system architecture innovations of all kinds proliferated. The list of individual innovations that led to today’s ubiquitous, powerful personal compute could fill dozens of books. A few decades from now, the history of AI inference innovation will show similar depth."
Of the areas mentioned in the article, which are the most likely to have the most prominent innovative impact, and what will they entail?
A large fraction of the innovation in CPUs is driven by working around the memory wall. I anticipate AI inference will follow the same trend, and innovations that work around the autoregressive nature will be enormously impactful.
Speculative decoding is an example. An accurate draft model can reduce the number of times you stream through memory by a factor of 4x.
How do you work around the memory wall when you're going to have to stream all weights, no matter what? Latency-hiding tricks don't matter when you're bandwidth constrained.
One of the biggest limitations right now is memory capacity (storing large models/contexts in memory) and bandwidth (transferring the relevant data/weights to the silicon that is performing the operations on that data). This would cover things like:
1. having more memory on the card/chip and/or faster access to that memory;
2. integrated memory and compute units optimized for matrix and vector multiply add operations;
3. optimized load circuitry to e.g. read memory in the stride and span (next row, next column) access patterns common to matrices or ensure that no/few parts of the chip are stalled waiting on data or operations to complete.
Another aspect is quantizations. These are similar to SIMD vector operations in that you are performing an operation on a block of n-bit data values at the same time, so can have optimized circuitry.
For 2 or 3 valued quantizations you can reduce various addition and multiplication operations to logic operations, avoiding circuitry for things like the half-adder, full-adder, and carry-lookahead.
Then there's adding specific circuitry for common operations such as ReLU like is done in hardware acceleration of image, video, etc. processing. There's a trade off here as optimized hardware would perform better at the specific operations but if those are too specific then they can't be used by different/newer model architectures. (Though it does make sense to try and optimize common operations/logic where possible.)
It would be interesting to see if these designs can/will benefit training as well, as that would bring down the time/cost/energy of training large models as well as making it easier for local fine-tuning.
Some kind of compute-in-memory architecture is a good candidate, I think. There are many alternatives here, researched for many years prior to the LLM craze. However economies of scale dominate in chip industries, and this tends to favor more conventional or incremental approaches (to piggyback on existing scale). Alternatively someone needs to have a way of bootstrapping the insane scales needed to be competitive with a better-but-different approach.
So it could be that boring and straightforward stuff like two-chip prefill+decode takes most.
I am missing a mention of ROM in the article. Keeping read-only weights in RAM is rather wasteful, as ROM can be implemented more cheaply. Approaches like High Bandwidth Flash (HBF) are relevant here, and should come to market in a few years.
Further optimization may be possible by tailoring for sequential access, since inference of a particular model is very predictable.
Exactly. If we ever get models that stabilize in terms of capabilities (and arent leapfrogging each other month after month) you have a "cartridge" like thing that just plugs in with the latest model in a read only ROM thats super fast.
As far as the interconnect to the GPU/CPU - thats a different story. But with Nvidia acquiring Mellanox and Nvlink getting faster and faster I assume well get there
Modern ROM is not locked in stone. For example, Flash memory is considered a modern, advanced type of Read-Only Memory (ROM), specifically a specialized form of Electrically Erasable Programmable Read-Only Memory (EEPROM), and can be re-written.
Isnt that slower though? Maybe Im not thinking of ROM specifically (the architecture/chip) but a different form of read only cache that can be located close to the compute and is super fast
I like how the author uses the analogy of scrabble word creation to describe LLM training but unfortunately the analogy didn't continue to inference and I got lost trying to keep up.
Excellent article. I believe the majority of benchmark performance gains moving forward will come from this side of the stack enabling faster iteration/recursion.
> Tensordyne is expected to accelerate AI inference with a logarithmic number system that leans on a property of logarithms: The log of A times B equals the log of A plus the log of B. So, storing numbers as their exponents lets the chip add where it would otherwise multiply. That matters in silicon because multiplier circuits draw more power and use more die area than adders do. Tensordyne says its rack-scale hardware, called Napier, can produce up to 1,300 tokens per second per user, and can do so while using less than a tenth as much power as comparable Nvidia hardware.
Did not know about this cool trick about storing numbers as exponents! Is there a name for this technique? Wouldn’t there be overhead in converting back and forth between the exponent and the number?
no. IEEE splits the power of two exponent from the base 2 mantissa. Yes, the exponents are added during a multiply, but the mantissas do an ordinary multiply.
The idea is rather than storing a number x as (exponent, mantissa), just store (log x) as a fixed precision number. Multiplying two such numbers is just addition, dividing is just subtraction. TBH I didn't read the article, but my reaction is that yes, that works, but one must sum all those products, and now summing becomes an expensive operation. Maybe the total cost saves area and power, but it beggars belief that it is 10x more efficient. They must be doing PR math: our low precision log scheme is 10x more efficient than a higher precision traditional approach.
Another thing to keep in mind is a lot of inference is done using very low precision math and so the cost of doing multiplies isn't that bad. Yes, it is still (n bits) squared, but as n gets small, n^2 still isn't too bad.
This is essentially the way many 8 bit games did 3d rendering ( for example the world famous Elite )... you just need two tables, one linear2log and another log2linear, with careful measurement of the ranges and number of elements needed in you application ( which is easy for inference ).
"If AI inference remains as desirable as Kimball expects, the evolution is likely to follow the same trajectory as the CPU. The CPU didn’t improve along a single axis but instead across simultaneously. Once transistor scaling slowed, chip and system architecture innovations of all kinds proliferated. The list of individual innovations that led to today’s ubiquitous, powerful personal compute could fill dozens of books. A few decades from now, the history of AI inference innovation will show similar depth."
Of the areas mentioned in the article, which are the most likely to have the most prominent innovative impact, and what will they entail?
A large fraction of the innovation in CPUs is driven by working around the memory wall. I anticipate AI inference will follow the same trend, and innovations that work around the autoregressive nature will be enormously impactful.
Speculative decoding is an example. An accurate draft model can reduce the number of times you stream through memory by a factor of 4x.
Speculative decoding is software, though.
How do you work around the memory wall when you're going to have to stream all weights, no matter what? Latency-hiding tricks don't matter when you're bandwidth constrained.
2x is more typical, and only for dense models, not the sparse models all frontier labs use.
One of the biggest limitations right now is memory capacity (storing large models/contexts in memory) and bandwidth (transferring the relevant data/weights to the silicon that is performing the operations on that data). This would cover things like:
1. having more memory on the card/chip and/or faster access to that memory;
2. integrated memory and compute units optimized for matrix and vector multiply add operations;
3. optimized load circuitry to e.g. read memory in the stride and span (next row, next column) access patterns common to matrices or ensure that no/few parts of the chip are stalled waiting on data or operations to complete.
Another aspect is quantizations. These are similar to SIMD vector operations in that you are performing an operation on a block of n-bit data values at the same time, so can have optimized circuitry.
For 2 or 3 valued quantizations you can reduce various addition and multiplication operations to logic operations, avoiding circuitry for things like the half-adder, full-adder, and carry-lookahead.
Then there's adding specific circuitry for common operations such as ReLU like is done in hardware acceleration of image, video, etc. processing. There's a trade off here as optimized hardware would perform better at the specific operations but if those are too specific then they can't be used by different/newer model architectures. (Though it does make sense to try and optimize common operations/logic where possible.)
It would be interesting to see if these designs can/will benefit training as well, as that would bring down the time/cost/energy of training large models as well as making it easier for local fine-tuning.
Some kind of compute-in-memory architecture is a good candidate, I think. There are many alternatives here, researched for many years prior to the LLM craze. However economies of scale dominate in chip industries, and this tends to favor more conventional or incremental approaches (to piggyback on existing scale). Alternatively someone needs to have a way of bootstrapping the insane scales needed to be competitive with a better-but-different approach. So it could be that boring and straightforward stuff like two-chip prefill+decode takes most.
> The list of individual innovations that led to today’s ubiquitous, powerful personal compute could fill dozens of books.
https://a.co/d/0dMk8urP
which is the Hennesey and Patterson computer architecture book would serve the role of the "dozens of books" hyperbole rather well.
I am missing a mention of ROM in the article. Keeping read-only weights in RAM is rather wasteful, as ROM can be implemented more cheaply. Approaches like High Bandwidth Flash (HBF) are relevant here, and should come to market in a few years. Further optimization may be possible by tailoring for sequential access, since inference of a particular model is very predictable.
Exactly. If we ever get models that stabilize in terms of capabilities (and arent leapfrogging each other month after month) you have a "cartridge" like thing that just plugs in with the latest model in a read only ROM thats super fast.
As far as the interconnect to the GPU/CPU - thats a different story. But with Nvidia acquiring Mellanox and Nvlink getting faster and faster I assume well get there
Modern ROM is not locked in stone. For example, Flash memory is considered a modern, advanced type of Read-Only Memory (ROM), specifically a specialized form of Electrically Erasable Programmable Read-Only Memory (EEPROM), and can be re-written.
Isnt that slower though? Maybe Im not thinking of ROM specifically (the architecture/chip) but a different form of read only cache that can be located close to the compute and is super fast
Great read.
I like how the author uses the analogy of scrabble word creation to describe LLM training but unfortunately the analogy didn't continue to inference and I got lost trying to keep up.
The analogy is flawed - in Scrabble you get rewarded for unlikely word combos. Also causal attention attends to prior tokens (tiles already placed).
Excellent article. I believe the majority of benchmark performance gains moving forward will come from this side of the stack enabling faster iteration/recursion.
> Tensordyne is expected to accelerate AI inference with a logarithmic number system that leans on a property of logarithms: The log of A times B equals the log of A plus the log of B. So, storing numbers as their exponents lets the chip add where it would otherwise multiply. That matters in silicon because multiplier circuits draw more power and use more die area than adders do. Tensordyne says its rack-scale hardware, called Napier, can produce up to 1,300 tokens per second per user, and can do so while using less than a tenth as much power as comparable Nvidia hardware.
Did not know about this cool trick about storing numbers as exponents! Is there a name for this technique? Wouldn’t there be overhead in converting back and forth between the exponent and the number?
hang on, isn't this the standard way to implement IEEE 754 floating-point multiply since forever?
normalize the two numbers A and B to have the same exponent, add the mantissa, then convert back to IEEE 754?
no. IEEE splits the power of two exponent from the base 2 mantissa. Yes, the exponents are added during a multiply, but the mantissas do an ordinary multiply.
The idea is rather than storing a number x as (exponent, mantissa), just store (log x) as a fixed precision number. Multiplying two such numbers is just addition, dividing is just subtraction. TBH I didn't read the article, but my reaction is that yes, that works, but one must sum all those products, and now summing becomes an expensive operation. Maybe the total cost saves area and power, but it beggars belief that it is 10x more efficient. They must be doing PR math: our low precision log scheme is 10x more efficient than a higher precision traditional approach.
Another thing to keep in mind is a lot of inference is done using very low precision math and so the cost of doing multiplies isn't that bad. Yes, it is still (n bits) squared, but as n gets small, n^2 still isn't too bad.
This is essentially the way many 8 bit games did 3d rendering ( for example the world famous Elite )... you just need two tables, one linear2log and another log2linear, with careful measurement of the ranges and number of elements needed in you application ( which is easy for inference ).
ps. Also used in the original circuits of the Yamaha DX7 synthetiser ( https://www.righto.com/2021/11/reverse-engineering-yamaha-dx... ).
Also I wonder how this affects the distribution and precision needed for storing the log weights compared to regular ones.
but aren't additions in exponents expensive? as in each operation is an fma
> And Anthropic is paying LLM competitor SpaceXAI over a billion dollars per month to lease spare compute
I knew of this but not the $ amount. Wow
Big money from taking Tesla's place in line for an Nvidia shipment during a huge shortage without compensation.