>"Features of matrix multipliers differ across vendors and architectures of the same vendor [...] As a result, reproducibility of small matrix multiplier results [differences] across devices is not possible and cannot be achieved by software control. Implementation details of matrix multipliers are not documented, making it difficult to interpret discrepancies in the computed results."
I'm guessing (but not knowing) that small subtle differences in matrix multiply across different vendor's architectures (and product generations of an individual vendor's architecture) is responsible for a good portion of software crashes when trying to run a local LLM on a different architecture or with a different stack (ROCm vs. CUDA, for example) than the ones it has been explicitly tested on.
As such, this marks a rather significant problem for the future, which can basically be stated as:
There needs to be a standard matrix multiply specification (much like IEEE-754 is/was for floating point operations) that all future vendors of AI accelerators (any GPU, CPU, NPU or IC manufacturer whose circuits implement matmul) adhere to, such that the matmul of one vendor is exactly and precisely compatible with the matmul of another.
Hardware vendors of course, are free to compete in terms of speed, power efficiency, number of matmul engines on a given piece of silicon, parallelization optimizations, etc., but the basic matmul operation should be exactly and precisely compatible across vendors and across future product versions.
Step 1: We need a spec for this... (Maybe IEEE is already working on one? If so, that's a good step forward!)
Step 2: Hardware vendors need to implement it, to be universally compatible in all of their IC's that use matmul, in the future...
To be clear, basic IEEE-754 scalar operations are already not compatible across different CPU hardware vendors. At best, we can say that if you run the same code generated by the same compiler targeting the same hardware, you'll get identical results. But nothing else is guaranteed.
Just off the top of my head:
1. The chip that gave us the IEEE-754 spec, the Intel 8087, internally worked with 80-bit extended precision floats that nothing else supports. Things would get truncated to standards-compliant 64- or 32-bit when spilled to memory, but you'd have to actually do this between every operation to get standard[1] rounding behavior. At least until the 80387 which let you set the internal precision.
2. Many, many RISC chips implemented a fused multiply-add (FMA) operation that could multiply and add faster than issuing separate instructions. Naturally the FMA unit would also result in fewer roundings. x86 did not have a widely supported FMA instruction until 2014[0].
In general, every floating point operation is going to have different rounding characteristics and that is the source of all floating-point hardware variance. Every time the number or order of operations changes, the output changes. In order to specify an "exactly and precisely compatible" calculation mode you have to freeze in place those operations, forever, across both silicon and compilers. This goes against the basic idea of a matmul accelerator: every time someone finds a way to multiply two matrices faster through parallelization, different tiling, or a different sequence of operations, that changes the rounding and numerical stability characteristics of the matmul, and now we need a new spec.
[0] To make matters worse, AMD shipped an incompatible FMA extension that was later removed in Zen 2!
[1] This is actually fairly tame in terms of "hardware vendors deviating from IEEE spec" - the Sony PS2 shipped with completely out-of-spec garbage floating point hardware that infamous tainted ports of games to other competing consoles.
Pretty sure the internal precision setting works the same way on the 8087, it's documented in the original 8087 datasheet. Windows sets it to 53-bit so there are no excess precision surprises with doubles, and you could run Windows 95 with an 80386 + 80287.
For instance, when I created script to train my model it worked fine on RTX 5080, but when I rented H100 to hopefully wait less for completion of training, the training would collapse just in a few epochs, suggesting they compute things differently (RTX 5080 would run thousands of epochs without collapsing. The same script and the same data).
Yep, 100%. In fact writing anything GPU co-processor related is nothing like porting CPU code. Sometimes the entire kernel needs to be rewritten outright when switching between GPU gens (of the same vendor). Things like precision et al also vary massively.
Usually these architectural differences are handled by the intermediate platform layer, which often doesn't apply if you're writing low-level kernels. Although this
> I'm guessing (but not knowing) that small subtle differences in matrix multiply across different vendor's architectures (and product generations of an individual vendor's architecture) is responsible for a good portion of software crashes when trying to run a local LLM on a different architecture or with a different stack (ROCm vs. CUDA, for example) than the ones it has been explicitly tested on.
Has to do with everything else but matrix mult precision. Feature support/drivers/alignment generally causes crashes, and again, the GPU field situation is so precarious that it's 10x worse than the AVX512 segmentation in the CPU world. You basically need to microverify whether the target GPU supports a given instruction/feature.
>"Features of matrix multipliers differ across vendors and architectures of the same vendor [...] As a result, reproducibility of small matrix multiplier results [differences] across devices is not possible and cannot be achieved by software control. Implementation details of matrix multipliers are not documented, making it difficult to interpret discrepancies in the computed results."
I'm guessing (but not knowing) that small subtle differences in matrix multiply across different vendor's architectures (and product generations of an individual vendor's architecture) is responsible for a good portion of software crashes when trying to run a local LLM on a different architecture or with a different stack (ROCm vs. CUDA, for example) than the ones it has been explicitly tested on.
As such, this marks a rather significant problem for the future, which can basically be stated as:
There needs to be a standard matrix multiply specification (much like IEEE-754 is/was for floating point operations) that all future vendors of AI accelerators (any GPU, CPU, NPU or IC manufacturer whose circuits implement matmul) adhere to, such that the matmul of one vendor is exactly and precisely compatible with the matmul of another.
Hardware vendors of course, are free to compete in terms of speed, power efficiency, number of matmul engines on a given piece of silicon, parallelization optimizations, etc., but the basic matmul operation should be exactly and precisely compatible across vendors and across future product versions.
Step 1: We need a spec for this... (Maybe IEEE is already working on one? If so, that's a good step forward!)
Step 2: Hardware vendors need to implement it, to be universally compatible in all of their IC's that use matmul, in the future...
> is responsible for a good portion of software crashes
I doubt it's the case, it should result in slightly different logits and generated tokens, but it shouldn't lead to crashes.
I suspect what you're facing are simply driver bugs…
It's memory/tiling alignment issues + wrong instruction/feaatures.
To be clear, basic IEEE-754 scalar operations are already not compatible across different CPU hardware vendors. At best, we can say that if you run the same code generated by the same compiler targeting the same hardware, you'll get identical results. But nothing else is guaranteed.
Just off the top of my head:
1. The chip that gave us the IEEE-754 spec, the Intel 8087, internally worked with 80-bit extended precision floats that nothing else supports. Things would get truncated to standards-compliant 64- or 32-bit when spilled to memory, but you'd have to actually do this between every operation to get standard[1] rounding behavior. At least until the 80387 which let you set the internal precision.
2. Many, many RISC chips implemented a fused multiply-add (FMA) operation that could multiply and add faster than issuing separate instructions. Naturally the FMA unit would also result in fewer roundings. x86 did not have a widely supported FMA instruction until 2014[0].
In general, every floating point operation is going to have different rounding characteristics and that is the source of all floating-point hardware variance. Every time the number or order of operations changes, the output changes. In order to specify an "exactly and precisely compatible" calculation mode you have to freeze in place those operations, forever, across both silicon and compilers. This goes against the basic idea of a matmul accelerator: every time someone finds a way to multiply two matrices faster through parallelization, different tiling, or a different sequence of operations, that changes the rounding and numerical stability characteristics of the matmul, and now we need a new spec.
[0] To make matters worse, AMD shipped an incompatible FMA extension that was later removed in Zen 2!
[1] This is actually fairly tame in terms of "hardware vendors deviating from IEEE spec" - the Sony PS2 shipped with completely out-of-spec garbage floating point hardware that infamous tainted ports of games to other competing consoles.
Pretty sure the internal precision setting works the same way on the 8087, it's documented in the original 8087 datasheet. Windows sets it to 53-bit so there are no excess precision surprises with doubles, and you could run Windows 95 with an 80386 + 80287.
Isn't the same amongst same architectures?
For instance, when I created script to train my model it worked fine on RTX 5080, but when I rented H100 to hopefully wait less for completion of training, the training would collapse just in a few epochs, suggesting they compute things differently (RTX 5080 would run thousands of epochs without collapsing. The same script and the same data).
Yep, 100%. In fact writing anything GPU co-processor related is nothing like porting CPU code. Sometimes the entire kernel needs to be rewritten outright when switching between GPU gens (of the same vendor). Things like precision et al also vary massively.
Usually these architectural differences are handled by the intermediate platform layer, which often doesn't apply if you're writing low-level kernels. Although this
> I'm guessing (but not knowing) that small subtle differences in matrix multiply across different vendor's architectures (and product generations of an individual vendor's architecture) is responsible for a good portion of software crashes when trying to run a local LLM on a different architecture or with a different stack (ROCm vs. CUDA, for example) than the ones it has been explicitly tested on.
Has to do with everything else but matrix mult precision. Feature support/drivers/alignment generally causes crashes, and again, the GPU field situation is so precarious that it's 10x worse than the AVX512 segmentation in the CPU world. You basically need to microverify whether the target GPU supports a given instruction/feature.
If you're an author of this, please follow up with the SME2 cores in Apple Silicon, and the AMX cores in Intel server processors.