Matt Pharr (author of the SIMD programming language ISPC) nailed it with the insight "Auto-vectorization is not a programming model."
As Matt writes ([1]) : "
The problem with an auto-vectorizer is that as long as vectorization can fail (and it will), then if you’re a programmer who actually cares about what code the compiler generates for your program, you must come to deeply understand the auto-vectorizer. Then, when it fails to vectorize code you want to be vectorized, you can either poke it in the right ways or change your program in the right ways so that it works for you again. This is a horrible way to program; it’s all alchemy and guesswork and you need to become deeply specialized about the nuances of a single compiler’s implementation—something you wouldn’t otherwise need to care about one bit."
I’m not sure I follow their point. By going with structs of arrays you’re already getting your max SIMD bandwidth 80% there even with the most naive implementations. Learning which math operations are hardware SIMDable gets you another 10%. It’s the last 10% where you rely on math expressions (eg. trig identities), restricted pointers, and ternary “masking magic” where the author may have a point.
Vectorization doesn't imply SIMD, of course. The first vectorizing compilers were for CDC(?) systems long before SIMD. Today you have SVE in Arm, for instance, distinct from SIMD Neon.
Anyway, I'm familiar with optimizing numeric loops in C (and Fortran) rather than Rust. I've rarely seen simply using SIMD intrinsics work where GCC auto-vectorization didn't with the same semantics (like numeric equivalence in reductions). In most cases you can get away with -fassociative-math, of course, and not sacrifice peak performance, e.g. BLIS passes its extensive tests with it on, but you should check, of course. (GCC also documents the option as necessary to get Arm (Neon?) to vectorize at all.) Most of the time when people tell you how much better the Itel compiler is, it's because it incorrectly defaults to something like -funsafe-math.
Regardless, GCC (like other compilers) will tell you about vectorization with the -fopt-info- options without examining assembler, and you can have some surprises. For instance, you use unsigned in C for loop indices that you know are positive, and see failed vectorization due to "loop not affine", because of C's overflow semantics; use signed types instead.
There's another reason for using properly-optimized numerical libraries (typically BLAS), is that, at least for level three (matrix-matrix) operations. Even if you get the blocking right for the memory hierarchy, you typically won't get peak performance just with vectorization because tricky preloading is needed for the inner loops.
> Vectorization doesn't imply SIMD, of course. The first vectorizing compilers were for CDC(?) systems long before SIMD. Today you have SVE in Arm, for instance, distinct from SIMD Neon.
You're right. In rust, IMHO, the problem is mostly solved by using system C libraries rather than trying to force Rust into something that looks like the optimized C/Fortran we've had for decades. In almost all cases, it's quite possible and quite painless to do it that way. For example, by just making it a matrix operation and letting the system wrapper library call out to something blas-like.
I do this. I do this a lot. My job involves processing a significant amount of weather data and flight telemetry _very quicky_.
What I have found is that getting rust to auto-vectorize is a nightmare. The options available to me have always converged around: 1) use simd-like apis 2) frame it as matrix-vector operations.
(1) is touched on in TFA
(2) is way easier, and allows use of well-tested apis and crates, each of which (sensibly) call out to better-tested C libraries. Each of those can, should, might, or will use your CPU better than you will. If you can frame it as matrix-vector operations, you will go very fast, not least of which, by stacking the operations into a _big_ matrix/vector op, which your CPU will happily tear though.
However the article proposes a third, very cool option: use algebraic ops API! Worth a read.
Rust used to not have any reasonable way to do anything like this on stable (for my own definition of reasonable), but now it does! Rust 1.98 stabilized algebraic operators for floats. These allow you to declare per-operation that you’re okay with the compiler making optimizations that may change the result as long as the optimized code is algebraically equivalent to the original. Yes, this includes potentially reordering operations.
Great insight. My (much less sophisticated) mental model is: 1: I don't know how to get things to auto-vectorize nor evaluate if they did. [I should learn]; I assume they do not. Manual SIMD it is! [I have a lib for floats and vectors which mimics core::simd but on stable and x86 only)
I will check out your insights regarding matrix-vector ops and C libs!
It has been a while, but I was quite surprised by how good gcc/g++ was at explaining why it had failed to vectorize a certain loop. At that time clang was being positioned as the better-than-gcc at optimization and error messages and it turned out that on my code it was the other way round -- hence the surprise.
I had written a expressions template C++ helper library with sort of the same functionality as Python's itertools before I was familiar with itertools.
This was for my own consumption. I expected very little from g++ and it had me impressed. Would be around 2008 - 2010.
Actually, one caveat is that GCC's optimization info can be rather inscrutable because it's in terms of compiler internal nomenclature. That's a definite area for improvement (or compiling some sort of key to it).
Indeed, GCC optimizes well. Last time I ran a set of Fortran benchmarks, the geometric mean for them was competitive with other compilers on multiple architectures, and some of the benchmarks could have been sped up considerably with specific compiler options or by re-writing a function sacrificing numerical equivalence, which the Intel compiler seemed to do itself.
I am more of an applied mathematician and a complete ignoramus in compiler technology. So I cannot emphasize enough the surprise ... wait I can actually understand what this compiler is saying and this is not clang, this is not supposed to happen on templates heavy code.
This is by no means a humble brag. Kudos to the GCC engineers. Competition with Clang certainly helped.
These days GCC has much better error messages even for non-vector code, IMO. I feel like Clang has become more of a research platform and GCC is winning on the usability front
If you turned on specific optimization and warning flags it emitted a lot of useful information to stdout/stderr. It was quite helpful even to a compiler technology ignoramus like me.
It would identify specific loops and would provide reasons why it could not vectorize it, usually some sort of aliasing that it could not rule out. I would then rewrite the code if the rewrite was simple, to make it obvious that such aliasing wouldn't occur. If it wasn't aliasing it was some sort of a cost benefit model that my loop had not crossed.
For a couple of days I experimented with auto-vectorization in my language which uses LLVM as its backend. In my example I had a simple function multiplying elements from two arrays and writing them in the third one. And it worked as expected, I did see autovectorization in the result code. But I also noticed something strange - there were a couple of pointer difference calculations and jumps prior to the unrolled/vectorized loop. Then I realized, what this means. LLVM isn't sure that it can use autovectorization, since input and output arrays may overlap. So, it just performs a runtime overlapping check and if no overlap is detected, a vectorized version of the loop is executed, otherwise it jumps to a non-vectorized loop version.
That's why C has restrict keyword. If it's provided, LLVM is sure that no runtime overlapping check is needed and can autovecrorize, sometimes even in places where such vectorization without restrict isn't possible at all.
Matt Pharr (author of the SIMD programming language ISPC) nailed it with the insight "Auto-vectorization is not a programming model."
As Matt writes ([1]) : " The problem with an auto-vectorizer is that as long as vectorization can fail (and it will), then if you’re a programmer who actually cares about what code the compiler generates for your program, you must come to deeply understand the auto-vectorizer. Then, when it fails to vectorize code you want to be vectorized, you can either poke it in the right ways or change your program in the right ways so that it works for you again. This is a horrible way to program; it’s all alchemy and guesswork and you need to become deeply specialized about the nuances of a single compiler’s implementation—something you wouldn’t otherwise need to care about one bit."
[1] https://pharr.org/matt/blog/2018/04/18/ispc-origins
I’m not sure I follow their point. By going with structs of arrays you’re already getting your max SIMD bandwidth 80% there even with the most naive implementations. Learning which math operations are hardware SIMDable gets you another 10%. It’s the last 10% where you rely on math expressions (eg. trig identities), restricted pointers, and ternary “masking magic” where the author may have a point.
A recent example: https://glouw.com/2026/08/14/Ensim5.html
Vectorization doesn't imply SIMD, of course. The first vectorizing compilers were for CDC(?) systems long before SIMD. Today you have SVE in Arm, for instance, distinct from SIMD Neon.
Anyway, I'm familiar with optimizing numeric loops in C (and Fortran) rather than Rust. I've rarely seen simply using SIMD intrinsics work where GCC auto-vectorization didn't with the same semantics (like numeric equivalence in reductions). In most cases you can get away with -fassociative-math, of course, and not sacrifice peak performance, e.g. BLIS passes its extensive tests with it on, but you should check, of course. (GCC also documents the option as necessary to get Arm (Neon?) to vectorize at all.) Most of the time when people tell you how much better the Itel compiler is, it's because it incorrectly defaults to something like -funsafe-math.
Regardless, GCC (like other compilers) will tell you about vectorization with the -fopt-info- options without examining assembler, and you can have some surprises. For instance, you use unsigned in C for loop indices that you know are positive, and see failed vectorization due to "loop not affine", because of C's overflow semantics; use signed types instead.
There's another reason for using properly-optimized numerical libraries (typically BLAS), is that, at least for level three (matrix-matrix) operations. Even if you get the blocking right for the memory hierarchy, you typically won't get peak performance just with vectorization because tricky preloading is needed for the inner loops.
> Vectorization doesn't imply SIMD, of course. The first vectorizing compilers were for CDC(?) systems long before SIMD. Today you have SVE in Arm, for instance, distinct from SIMD Neon.
I wouldn't say "long before" since SIMD machines like ILLIAC IV came out with the IVTRAN compiler at the same time as the ASC/CDC. SVE is also considered SIMD https://support.arm.com/documentation/102340/0100/Introducin...
You're right. In rust, IMHO, the problem is mostly solved by using system C libraries rather than trying to force Rust into something that looks like the optimized C/Fortran we've had for decades. In almost all cases, it's quite possible and quite painless to do it that way. For example, by just making it a matrix operation and letting the system wrapper library call out to something blas-like.
I do this. I do this a lot. My job involves processing a significant amount of weather data and flight telemetry _very quicky_.
What I have found is that getting rust to auto-vectorize is a nightmare. The options available to me have always converged around: 1) use simd-like apis 2) frame it as matrix-vector operations.
(1) is touched on in TFA
(2) is way easier, and allows use of well-tested apis and crates, each of which (sensibly) call out to better-tested C libraries. Each of those can, should, might, or will use your CPU better than you will. If you can frame it as matrix-vector operations, you will go very fast, not least of which, by stacking the operations into a _big_ matrix/vector op, which your CPU will happily tear though.
However the article proposes a third, very cool option: use algebraic ops API! Worth a read.
> My job involves processing a significant amount of weather data and flight telemetry _very quicky_
That sounds so interesting. Would you be at liberty to share what that job is.
Oh! nevermind, got that off your HN page.
Great insight. My (much less sophisticated) mental model is: 1: I don't know how to get things to auto-vectorize nor evaluate if they did. [I should learn]; I assume they do not. Manual SIMD it is! [I have a lib for floats and vectors which mimics core::simd but on stable and x86 only)
I will check out your insights regarding matrix-vector ops and C libs!
Very generally speaking, you want to view dissasembled sections.
You can do this at the function level.
In rust, try this: https://github.com/pacak/cargo-show-asm
However, life is much easier now, because an agent knows how to steer through this with you. Have it teach you at first, then you know how to ask.
This is one of those "back in my day we had to ... " stories, btw :D
Does Rust not have SIMD intrinsics that you could use directly instead of relying on auto-vectorization?
It does, thus (1)
It has been a while, but I was quite surprised by how good gcc/g++ was at explaining why it had failed to vectorize a certain loop. At that time clang was being positioned as the better-than-gcc at optimization and error messages and it turned out that on my code it was the other way round -- hence the surprise.
I had written a expressions template C++ helper library with sort of the same functionality as Python's itertools before I was familiar with itertools.
This was for my own consumption. I expected very little from g++ and it had me impressed. Would be around 2008 - 2010.
Actually, one caveat is that GCC's optimization info can be rather inscrutable because it's in terms of compiler internal nomenclature. That's a definite area for improvement (or compiling some sort of key to it).
Indeed, GCC optimizes well. Last time I ran a set of Fortran benchmarks, the geometric mean for them was competitive with other compilers on multiple architectures, and some of the benchmarks could have been sped up considerably with specific compiler options or by re-writing a function sacrificing numerical equivalence, which the Intel compiler seemed to do itself.
I am more of an applied mathematician and a complete ignoramus in compiler technology. So I cannot emphasize enough the surprise ... wait I can actually understand what this compiler is saying and this is not clang, this is not supposed to happen on templates heavy code.
This is by no means a humble brag. Kudos to the GCC engineers. Competition with Clang certainly helped.
These days GCC has much better error messages even for non-vector code, IMO. I feel like Clang has become more of a research platform and GCC is winning on the usability front
Explaining? How do you get it to do that?
If you turned on specific optimization and warning flags it emitted a lot of useful information to stdout/stderr. It was quite helpful even to a compiler technology ignoramus like me.
It would identify specific loops and would provide reasons why it could not vectorize it, usually some sort of aliasing that it could not rule out. I would then rewrite the code if the rewrite was simple, to make it obvious that such aliasing wouldn't occur. If it wasn't aliasing it was some sort of a cost benefit model that my loop had not crossed.
For a couple of days I experimented with auto-vectorization in my language which uses LLVM as its backend. In my example I had a simple function multiplying elements from two arrays and writing them in the third one. And it worked as expected, I did see autovectorization in the result code. But I also noticed something strange - there were a couple of pointer difference calculations and jumps prior to the unrolled/vectorized loop. Then I realized, what this means. LLVM isn't sure that it can use autovectorization, since input and output arrays may overlap. So, it just performs a runtime overlapping check and if no overlap is detected, a vectorized version of the loop is executed, otherwise it jumps to a non-vectorized loop version.
That's why C has restrict keyword. If it's provided, LLVM is sure that no runtime overlapping check is needed and can autovecrorize, sometimes even in places where such vectorization without restrict isn't possible at all.
Interesting, but would make for a much stronger article with a proper discussion of performance, backed by measurements.
This would allow the reader to better weigh the balance of code damage against any gains in performance.
Unless I missed it, it just seemed to be hinted at, but not presented.
This is cool, but I mostly wanna see the numbers.
How much faster did it actually get?
If I have to make the code kinda weird just so the compiler behaves, I wanna know it was worth it lol.
maybe unrelated by NumPy and Cython have made it incredibly easy to write vectorized code.