> measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer.
This is so useful because it makes you actually trust a models output. A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses. But this index measures how often it is correct while penalizing wrong responses so that a high score means you can trust this model more and when it doesn't know it is more likely to tell you that it really doesn't know rather than making shit up.
Fable also performs a lot better than opus 5 here which correlates very strongly with perceived strength despite the models performing similarly on e.g. DeepSWE
Astra is a big jump from sol and performs the same or slightly better than fable here.
I don't agree. The main issue with their scoring/methodology is that the numbers make it seem like 5-6 models have little to no difference when in fact there is a significant difference between fable and opus and sol and astra for example. They are popular mainstream but most of their benchmarks are either not a representation of model strengths enough or they are not doing a good job of showcasing it properly. The fact that muse and 3.8 were high a day back shows they are just the modern version of lmareana for the mass audience and PR stunts.
This matches what I've seen building anything that uses an LLM for narrow structured output rather than open ended chat, things like classifying a diff into a fixed set of categories or summarizing a change. The aggregate benchmark score barely predicts how it behaves in production. What actually breaks a pipeline is a confident wrong answer on the small slice of inputs that don't fit the pattern it saw during training, not a lack of raw capability. A model that says it isn't sure on the edge cases is far more useful to me than one that scores higher on average but never admits uncertainty, because the wrong but confident output is the one that slips through review unnoticed.
This is really a great achievement: "Astra dominates the output token frontier"
Many labs used increased thinking to boost benchmark scores and performance. Most of the Chinese models were doing that for a while. Google and Anthropic as well.
Not OpenAI. 5.6 already was much more token efficient than other models and Astra beats Sol in token efficiency by a wide margin.
Edit: Just to make the point: Astra (max) has the 2nd highest score and the third lowest output tokens (among the models shown by AA).
In real life, there is always this feedback edge from the results to the methodology.
Theoretically it's unscientific to do tweaks like this but in reality this is what actual science is because you need to see the results understand the problems in your experimental designs.
Now the interesting thing is that you could repair the bias problem. The main issue is that you will do these tweaks after a bad result not after they look fine.
Maybe we could commit ahead of time that the experiment will be reanalyzed after results regardless of what they are. Instead of only when the results disprove the hypothesis.
So like artificial analysis committing to a fixed cadence of index updates instead of when the results start looking jank.
is any of this 'scientific'? does AA allow peer review of its processes? are these published in journals of at least medium impact? what are the sample sizes? how grounded is their mechanistic reasoning?
like they have words that are dressed in scientific language on their site like
"We estimate a 95% confidence interval for Artificial Analysis Intelligence Index of less than ±1% - based on experiments with >10 repeats on certain models for all evaluation datasets included in Artificial Analysis Intelligence Index v4.2."
but where's the outcome dataset justifying this? how did they get that probability? what was the specific methodology of the tests? what variables did they account for?
there's a major difference between scientific sounding and being truly empirically rigorous. the 'research' in AI intelligence feels somehow even less trustworthy than supplement-funded studies because those are at least subjected to scrutiny by peers without profit motives
This update really gives OpenAI a boost. Not saying there's anything inaccurate or untoward about that, but the timing is unfortunate. It would have looked better had it been done prior to the Fable 5.1 and GPT 6 releases. I guess AA would say that there's no perfect time to do these updates, given the rapid fire pace of releases!
Imo the omniscience index they have has the highest correlation to actual usefulness of the models.
https://artificialanalysis.ai/evaluations/omniscience
> measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer.
This is so useful because it makes you actually trust a models output. A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses. But this index measures how often it is correct while penalizing wrong responses so that a high score means you can trust this model more and when it doesn't know it is more likely to tell you that it really doesn't know rather than making shit up.
Fable also performs a lot better than opus 5 here which correlates very strongly with perceived strength despite the models performing similarly on e.g. DeepSWE
Astra is a big jump from sol and performs the same or slightly better than fable here.
I don't agree. The main issue with their scoring/methodology is that the numbers make it seem like 5-6 models have little to no difference when in fact there is a significant difference between fable and opus and sol and astra for example. They are popular mainstream but most of their benchmarks are either not a representation of model strengths enough or they are not doing a good job of showcasing it properly. The fact that muse and 3.8 were high a day back shows they are just the modern version of lmareana for the mass audience and PR stunts.
Is there something better over there you'd recommend?
This matches what I've seen building anything that uses an LLM for narrow structured output rather than open ended chat, things like classifying a diff into a fixed set of categories or summarizing a change. The aggregate benchmark score barely predicts how it behaves in production. What actually breaks a pipeline is a confident wrong answer on the small slice of inputs that don't fit the pattern it saw during training, not a lack of raw capability. A model that says it isn't sure on the edge cases is far more useful to me than one that scores higher on average but never admits uncertainty, because the wrong but confident output is the one that slips through review unnoticed.
This is really a great achievement: "Astra dominates the output token frontier"
Many labs used increased thinking to boost benchmark scores and performance. Most of the Chinese models were doing that for a while. Google and Anthropic as well.
Not OpenAI. 5.6 already was much more token efficient than other models and Astra beats Sol in token efficiency by a wide margin.
Edit: Just to make the point: Astra (max) has the 2nd highest score and the third lowest output tokens (among the models shown by AA).
GPT-6 has a looped transformer / recurrent depth architecture, so it gets some internal CoT reasoning "for free" with no output tokens.
This also makes it much harder to monitor its reasoning.
They realized Astra having the same score as Sol was silly so they rushed to update the index so it fits what people expect.
The old index was clearly bad (Astra is way better than Sol) but it's also unscientific to tweak it like this.
In real life, there is always this feedback edge from the results to the methodology.
Theoretically it's unscientific to do tweaks like this but in reality this is what actual science is because you need to see the results understand the problems in your experimental designs.
Now the interesting thing is that you could repair the bias problem. The main issue is that you will do these tweaks after a bad result not after they look fine.
Maybe we could commit ahead of time that the experiment will be reanalyzed after results regardless of what they are. Instead of only when the results disprove the hypothesis.
So like artificial analysis committing to a fixed cadence of index updates instead of when the results start looking jank.
is any of this 'scientific'? does AA allow peer review of its processes? are these published in journals of at least medium impact? what are the sample sizes? how grounded is their mechanistic reasoning?
like they have words that are dressed in scientific language on their site like
"We estimate a 95% confidence interval for Artificial Analysis Intelligence Index of less than ±1% - based on experiments with >10 repeats on certain models for all evaluation datasets included in Artificial Analysis Intelligence Index v4.2."
but where's the outcome dataset justifying this? how did they get that probability? what was the specific methodology of the tests? what variables did they account for?
there's a major difference between scientific sounding and being truly empirically rigorous. the 'research' in AI intelligence feels somehow even less trustworthy than supplement-funded studies because those are at least subjected to scrutiny by peers without profit motives
This update really gives OpenAI a boost. Not saying there's anything inaccurate or untoward about that, but the timing is unfortunate. It would have looked better had it been done prior to the Fable 5.1 and GPT 6 releases. I guess AA would say that there's no perfect time to do these updates, given the rapid fire pace of releases!
What version the intelligence vs cost graph is using? they didn't ran v4.2 to all models.
It's very different than it was earlier today, so pretty sure it's v4.2.
Why have they not included ARC-Agi-3 on their index?
Clearly that would move things around.
How to view the previous version to compare?