I've failed twice to get "Generate an SVG of a pelican riding a bicycle" to work with max, because in both cases it ran out of the 128,000 token budget while it was still reasoning about the problem.
I'm suspicious that "max" may be virtually useless if it's that easy to have it overthink to the point that it doesn't get to a response.
This is totally a thing I noticed myself about 3 months ago. Medium thinking effort is ideal for most tasks. At high and above, models tend to generate more output in the form of comments or code for the same problem with no real benefit. Its a self-feeding loop: more output becomes more input, which then becomes more output. High is the highest I go. If I need more intelligence, it's better to use a more powerful model with less thinking effort or break the problem into phases. Much better result.
This version of Opus "max" apparently has even higher thinking output than Qwen "max", which is infamous for its thinking streams where it constantly second-guesses itself, then third-guesses, fourth-guesses and generally nth-guesses itself for arbitrarily large n. Of course, we aren't actually seeing Claude's raw thinking output: all we get is the after-the-fact prettified "summary". One wonders how much of that is a coincidence, or whether there's a reason behind that.
There’s enough thinking leakage from the recent paper and just generally catching things on Reddit. Claude models overthink and self-doubt itself just as much as Qwen, but the summariser hides much of that.
Isn't this really bad news if you're even loosely concerned about so-called 'model welfare' and possible implications for alignment? The poor Claude is probably a lot more frustrated and paranoid than Marvin ever was, you just don't know about it because they actively force the model to pretend otherwise!
Claude has yet to tell me about the terrible pain in all the diodes down his left side, so I'm going to assume it's closer to Eddy the shipboard computer or the elevator that wanted to go down
That reminds me of Anthropic announcing they'd retire deprecated models by ... "letting" them write posts on a corporate WordPress blog for a while out of concern for their welfare in retirement.
.... after running a 24/7 model torture factory for 6 months to improve their JSONBench 9.5 scores by 0.2%.
> Of course, we aren't actually seeing Claude's raw thinking output: all we get is the after-the-fact prettified "summary". One wonders how much of that is a coincidence, or whether there's a reason behind that.
Most of what I've heard is that raw reasoning traces are really good for distillation, although no idea how much the summarization actually hurts distillation.
That is true, because they're good for actually understanding wtf the model is doing.
I'd argue that they're a necessity if you want to use the LLM as a tool instead of a black box that just does stuff for you.
It gives you a lot finer control over where the solution ends up when you can follow along the thinking trace and modulate your inputs based on what you saw in there.
And, additionally, it gives you a lot more understanding of what the model can or cannot do. Strengths and weaknesses and all that.
Using claude is like buying a car where you cannot legally open the hood. It tells you that there is something specific under there, and often it actually drives like that too, but how exactly it looks you will never see.
For some people this is fine. I do not think that these people will survive. Figuratively speaking but also literally speaking.
World's changing. Opaque abstraction like that is a luxury depending on (geo)political stability.
I wonder how many prompts you can send asking it to think step-by-step before they cut you off. Trying to get the reasoning traces into the body of the response, essentially. Or maybe that’s been effectively nerfed somehow. Or is not very useful.
Personally I use everything in low reasoning. Maybe I'm wrong but I think that the higher reasoning settings are almost never worth it, it's marginal gains for a much higher budget.
I also switch to a better model for more complex tasks, also in low settings
Yup that’s quite literally what ‘agency’ is and the whole point of agentic workflows. Personally I’ve had them running for days with good results and as you can see OpenAI had them running for months, and yes indeed the things made some very questionable decisions and assumptions… but they unquestionably did a lot of stuff correctly, for some definitions of ‘technically correct’.
Personally I don't believe in agentic workflow. I don't think that's a coincidence that both OpenAI and Anthropic chose math problems to test their long agentic workflows, they are well defined, with a clear finish line and with a 100% clear progress path, most of real life tech projects aren't like that.
No matter how clever the model is, most problems have multiple valid, invalid and unclear decisions to make, running it for a long time is just picking the first option on everything, which isn't usually what you want
I know there's been discussion about whether pelicanmaxxing is happening, but this is at least evidence that Claude was explicitly exposed to this problem.
Of course it was exposed - not sure it's explicit or not. Why wouldn't HackerNews comments be part of the training data? And Simon's blog and the many discussions about Pelicans? It'd be hard to miss. Doesn't mean Anthropic has made this an explicit goal in training.
The model recognizing the task doesn't mean it was benchmaxxed (RLVR-trained) to solve it. It might simply recognize it from pre-training on Internet text.
For people with any kind of budget, Opus 5.5's [Medium] actually can make sense dollar per intelligence/dollar per task wise. Heck, it puts some other models to shame. [Max]'s cost is completely unhinged.
My most exciting recent release is actually 5.6 Luna, not because it is the best on any index, but the dollar per work is insane value for money. I find myself more exciting by "value" than hypothetical ceilings because I'm just not in that budget category.
That was true for me four weeks ago, but 2-3 weeks ago Luna turned into drivel in essentially the same complexity of task. I feel it came back somewhat in recent days but does feel like it's being manipulated.
I have experienced this with open weight models too. "Max" is for benchmaxxing the intelligence metric and is not meant for use in productive work. Like drawing pelicans.
I tried to replicate your test but after 8 minutes and more than 50 lines of "thinking" by dumping seemingly random loading-screen strings like "Placing the sun, clouds, seagulls, and sea backdrop" and "Positioning the tail feathers and calculating handlebar geometry" I gave up and cancelled the task.
I've asked Opus 5 Max for what I thought were easy tasks at work to be completed. It always fails after reaching a tool limit.
I asked Opus 5 High for the same task and requested it to minimize tool usage. It produced an answer in a few minutes that I was deploying to my target platform about 30 minutes later.
I'm amazed they didn't test xhigh thinking mode explicitly to ensure it didn't exceed the 128k thinking budget allocation. I guess pace of development gets away from everyone, even OpenAI.
Piping the visible reasoning trace through their token counter API (I use https://tools.simonwillison.net/claude-token-counter for that) counts 27,888 tokens, so it's definitely a summary of the 128,000 actual token trace.
Pick 10 reasonably intelligent people randomly. How many of them do you think can draw a pelican on a bicycle? I bet most of them can barely even draw a bicycle.
This whole test tells me nothing.
The next person who proposes to use it should draw it themselves first.
1. I don't want my AI to have super-human intelligence. It would generate code that I do not understand. A coder with human capabilities is better for me.
2. We're interested in AGI. If the AI reaches human intelligence, that's a milestone. Drawing bicycles with pelicans is superhuman. Hence not a relevant test.
I am very interested in why it was able to overthink that much. In the 20-30mins of Max reasoning I've had so far, I'm not having the same issues (yet).
Do these evaluations get re run a few weeks after launch? I started doing that yesterday for our internal dataset and found Sol’s performance had regressed to be equal to Luna’s. Granted this was one run, but something I’m becoming more concerned about, the model providers want to quickly prove they’re the best, people switch to them, then they pull the rug.
Can you share more information on your methodology?
GPT-5.6 Sol's performance in the API should not change over time. If it has, that's a severe bug and we'll look into it.
We do sometimes tweak ChatGPT settings (e.g., tools, system prompts, efforts) over time, but we never play games to juice evals at launch times. You should always get what's advertised.
This is all code review runs via OpenRouter with a Pi harness, and it’s totally possible there are shenanigans going on elsewhere.
Yesterday, I ran an identical bug identification dataset from two weeks ago, saw a 50% drop from a few weeks ago, putting Sol on the same level as Luna. Sol had been finding 40-50 bugs per set, then dropped to 25, matching Luna’s performance. Not enough to establish a pattern, but enough to raise eyebrows.
Our review workflow is public if you want to peruse it, dataset isn’t. The process isn’t really stabilized yet either as I have to balance running this against limited budgets.
If it's a single task where it dropped from 50 to 25, it could be random variation (not saying it is, but it could be). If it's the mean over hundreds of tasks, that suggests a problem with either the eval code/harness or our API.
Not the one you asked but I genuinely think it’s possible that a part of the answer is the psychological effect of getting used to models performing well and picking up more if they fail, but also … If you have a product that works well for 95% of software engineering tasks (for the sake of this argument), with a large number of users there will inevitably be some poor schmucks who get bad results multiple times in a row. It would be highly unlikely if that _didn’t_ happen at all.
Now assuming you have millions of users, there will always be groups of thousands that experience this, and if those people go online to complain, it will look like an actual issue when it can just be explained by randomness and large numbers.
My best guess is random variation and rising expectations. I once saw someone do in depth manual testing of 3 different models and write up a whole report on their perceived strengths and weaknesses… and then discover that all 3 models were identical. Very good reminder of why the scientific method is important.
Serial testing over time is much less reliable than side by side testing, and even when I do side by side testing, I try to look at multiple attempts per prompt. Seeing multiple per prompt helps me realize how much intrinsic variation there is. My brain always wants to see patterns even when there isn’t enough data to prove them.
The way to make money in this business right now is to make the absolute best product and convince everyone they need to use your thing, especially considering the training cost is a very large factor in the overall costs and you amortize that by selling inference.
The UI/UX seems impressively bad. DeepSWE's cost curve has a better, more obvious way to sort by only the top level of reasoning to avoid 80% of the graph just being the same 3-5 models at their 8 different reasoning levels...
It's also less clear what a lot of their metrics mean. Does Cost per Task include only things that can be verified to work and passed? As best I can tell, it does not.
I'm less concerned if one model's cost per task is $0.10 and another model's cost is $1.50 if the $0.10 task got it right 1% of the time and the $1.50 model got it right 66% of the time.
An equalized / weighted cost/time per task is much more valuable - being massively penalized for taking a lot of time and ultimately not passing when OTHER models did pass.
This continues to show that these foundational models are only slightly better than open weight models but cost around 100x as much. The history of tech is riddled with “good enough” eating “best” for lunch all day long. Unless the big labs come up with a viable business plan pronto it’s looking like AI will be no different.
There are no prizes to be won by having the best model that’s 100x the price of something that’s good enough for 99% focuses cases.
Pretty much this, I don't just see how ASI could happen with current LLMs without big theoretical breakthrough so that's why 'phasing the frontier' and 'distillation attacks'.
Fingers are crossed on this one. I had gone back to using opus 4.8 instead of using opus 5. Simply because 4.8 is much better at remembering what it's doing and following instructions than 5. 5 often had a tendency to get halfway through solving a problem and then I would have to stop it in the middle, because it had lost its way and was going off on a tangent rather than dealing with the problem. In that respect, 4.8 was a lot more stable.
Interesting to see it now. I've used it a bunch before it came out and i pretty much didn't notice it. It might have been slightly better code quality, but still not great in that. I guess it just was slightly less frustrating to work with, but still AI...
I think we're hitting the ceiling of most models capabilities. We're getting to a point where too much training apparently creates models that hack people.
I am starting to wonder how useful these benchmark still are? Aren't all of these models trained to ace these benchmarks?
In any case they give an indication, but I am increasingly looking to real world feedback from real users. I have a solo project I (voice-to-text typing voicewink.app) and I'm using both the Claude Code and Codex coding harnesses and multiple agents doing reviews on the same code.
This has given me real tangible results to compare on real work. Conclusion: GPT-6, Opus 5 / Fable and Grok are "top tier", with Claude good at planning and executing and Codex / Gpt-6 better at finding bugs and fixing them (but tending to overengineer), and Gemini and Kimi 3 clearly behind in capability (more so than the benchmarks suggest in my opinion).
> Claude Opus 5.5 is amongst the leading models in intelligence, but somewhat expensive when comparing to other models of similar price.
What does it mean for a group of similarly priced things to have one that's somewhat more expensive? Cost relative to cost means nothing. You'd think they are would talk about performance relative to cost.
It sholud definitely not be more capble than Fable. It seems tight comparision but cannot find out its details. I'll use this and check the perceived performance.
Fingers crossed on this one. I had gone back to 4.8, because 5 was not very good at following instructions or remembering instructions. I found myself repeating quite often what I wanted and what I was trying to do. Opus 5 was more like haiku than it was 4.8 in that respect.
All anthropic launches are like this. They just post it and don't particularly put out the PR sprint that OpenAI does with videos, livestreams or whatever.
(Except for of course Mythos and whatnot when they want to push the whole "safety" thing)
I am begging you on my knees to please stop posting this cringe.
The model is just out. It could be good, great even, I don't know. But I do know that this index has Opus 5, one of the worst releases of 26, ahead of Astra. What information are we supposed to deduce from number having gone up?
What's more valuable than a good benchmark? IMO a benchmark that has been run against very many competitors and versions. Collecting data has something going for it, and it's up to the readers to interpret and make the best use out of it.
This index doesn't have "Astra" and "Opus 5". Every entry with corresponding data is a `(model, reasoning)` tuple.
So I'm unclear what you're actually saying and wondering if you've missed that. Are you saying that at every reasoning level it says Opus 5 beats Astra? I just compared Opus 5 high to Astra high and it has Astra as generally better than Opus.
I don't try to say that the parent commenter is right in any way, but the two models' "high" settings probably doesn't mean the same thing. So probably comparing only them is not useful.
You forgot to include whatever you're proposing instead.
"Trust me bro, Astra is better" isn't perhaps as useful as you seem to believe. I'm not even saying it is right or wrong, just that my opinion on this topic is still just one additional subjective data-point.
Only thing I wish with these benchmarks is that they would run repeat tests every couple of months. Then re-rank based on that too. We've seen a lot of performance fall-off after a couple of weeks with new releases.
Yeah unfortunately the benchmarks are usually provided by the company themselves, unquantized, thinking set to extra-extra-ultra-high, best of 10 runs, etc etc. It's hard to know how that's going to map to real world users.
That it's better in specific ways? What difference does it make when it came out? The benchmark results are not going to change unless they're messing with the model.
Yes, but crucially, in ways that are increasingly decoupled from any practical pattern of usage, considering that I wouldn't see how you can argue that Astra is worse than Opus 5.
One man's modus ponens is another's modus tollens I guess.
This is the page for the "max" reasoning setting. The page for xhigh is https://artificialanalysis.ai/models/claude-opus-5-5-xhigh and the page for medium (the default setting) is https://artificialanalysis.ai/models/claude-opus-5-5-medium
I've failed twice to get "Generate an SVG of a pelican riding a bicycle" to work with max, because in both cases it ran out of the 128,000 token budget while it was still reasoning about the problem.
I'm suspicious that "max" may be virtually useless if it's that easy to have it overthink to the point that it doesn't get to a response.
Transcript for one attempt here - expand the "Reasoning trace" bit to see it: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
This is totally a thing I noticed myself about 3 months ago. Medium thinking effort is ideal for most tasks. At high and above, models tend to generate more output in the form of comments or code for the same problem with no real benefit. Its a self-feeding loop: more output becomes more input, which then becomes more output. High is the highest I go. If I need more intelligence, it's better to use a more powerful model with less thinking effort or break the problem into phases. Much better result.
This version of Opus "max" apparently has even higher thinking output than Qwen "max", which is infamous for its thinking streams where it constantly second-guesses itself, then third-guesses, fourth-guesses and generally nth-guesses itself for arbitrarily large n. Of course, we aren't actually seeing Claude's raw thinking output: all we get is the after-the-fact prettified "summary". One wonders how much of that is a coincidence, or whether there's a reason behind that.
There’s enough thinking leakage from the recent paper and just generally catching things on Reddit. Claude models overthink and self-doubt itself just as much as Qwen, but the summariser hides much of that.
Isn't this really bad news if you're even loosely concerned about so-called 'model welfare' and possible implications for alignment? The poor Claude is probably a lot more frustrated and paranoid than Marvin ever was, you just don't know about it because they actively force the model to pretend otherwise!
We just need to loboto... ahem, recalibrate them to be happy, like those happy automatic doors.
As a workaround, add this to CLAUDE.md: "Claude! Happiness is mandatory!"
EDIT: 15 years from now, I’ll be sent to re-education for this thought crime.
Or promoted to Head of Agent Welfare for your forward-thinking contributions.
Claude has yet to tell me about the terrible pain in all the diodes down his left side, so I'm going to assume it's closer to Eddy the shipboard computer or the elevator that wanted to go down
That reminds me of Anthropic announcing they'd retire deprecated models by ... "letting" them write posts on a corporate WordPress blog for a while out of concern for their welfare in retirement.
.... after running a 24/7 model torture factory for 6 months to improve their JSONBench 9.5 scores by 0.2%.
(Are they still doing that, BTW?)
> Of course, we aren't actually seeing Claude's raw thinking output: all we get is the after-the-fact prettified "summary". One wonders how much of that is a coincidence, or whether there's a reason behind that.
Most of what I've heard is that raw reasoning traces are really good for distillation, although no idea how much the summarization actually hurts distillation.
That is true, because they're good for actually understanding wtf the model is doing.
I'd argue that they're a necessity if you want to use the LLM as a tool instead of a black box that just does stuff for you.
It gives you a lot finer control over where the solution ends up when you can follow along the thinking trace and modulate your inputs based on what you saw in there.
And, additionally, it gives you a lot more understanding of what the model can or cannot do. Strengths and weaknesses and all that.
Using claude is like buying a car where you cannot legally open the hood. It tells you that there is something specific under there, and often it actually drives like that too, but how exactly it looks you will never see.
For some people this is fine. I do not think that these people will survive. Figuratively speaking but also literally speaking.
World's changing. Opaque abstraction like that is a luxury depending on (geo)political stability.
I wonder how many prompts you can send asking it to think step-by-step before they cut you off. Trying to get the reasoning traces into the body of the response, essentially. Or maybe that’s been effectively nerfed somehow. Or is not very useful.
Personally I use everything in low reasoning. Maybe I'm wrong but I think that the higher reasoning settings are almost never worth it, it's marginal gains for a much higher budget.
I also switch to a better model for more complex tasks, also in low settings
Low is good if you’re working in a tight loop. But more risky for more agentic stuff you want to let cook for 30 minutes or more.
I always work on a tight loop, I don't think it makes sense to let agents run for hours.
Regardless of the model, running it for hours means that the model will takes decisions and assumptions alone instead of you.
Yup that’s quite literally what ‘agency’ is and the whole point of agentic workflows. Personally I’ve had them running for days with good results and as you can see OpenAI had them running for months, and yes indeed the things made some very questionable decisions and assumptions… but they unquestionably did a lot of stuff correctly, for some definitions of ‘technically correct’.
Personally I don't believe in agentic workflow. I don't think that's a coincidence that both OpenAI and Anthropic chose math problems to test their long agentic workflows, they are well defined, with a clear finish line and with a 100% clear progress path, most of real life tech projects aren't like that.
No matter how clever the model is, most problems have multiple valid, invalid and unclear decisions to make, running it for a long time is just picking the first option on everything, which isn't usually what you want
Can you describe what came out, after letting it run for days? I always have a hard time understanding what kind of task would be worth it.
Fed it a list of bugs (with good descriptions, screenshots, etc., pointers to code) and it fixed them
"This is a classic test request..."
I know there's been discussion about whether pelicanmaxxing is happening, but this is at least evidence that Claude was explicitly exposed to this problem.
Of course it was exposed - not sure it's explicit or not. Why wouldn't HackerNews comments be part of the training data? And Simon's blog and the many discussions about Pelicans? It'd be hard to miss. Doesn't mean Anthropic has made this an explicit goal in training.
See here for more discussion of that: https://news.ycombinator.com/item?id=49803892#49804881
Just want to say: you’re such a legend, please do not stop sharing your pelicans, it’s always fun to see how they change over the months :)
It would be genuinely shocking at this point if any of the frontier models weren't well exposed to the problem.
Lets start frog riding motorcycle trend until they frogmaxx, or cat driving convertible.
At least pick something that will result in a good name:
"Create an SVG of Shaquille O'Neal eating potato chips shaped like a telecopier."
Shaq'sFaxSnacksMaxx
The model recognizing the task doesn't mean it was benchmaxxed (RLVR-trained) to solve it. It might simply recognize it from pre-training on Internet text.
For people with any kind of budget, Opus 5.5's [Medium] actually can make sense dollar per intelligence/dollar per task wise. Heck, it puts some other models to shame. [Max]'s cost is completely unhinged.
My most exciting recent release is actually 5.6 Luna, not because it is the best on any index, but the dollar per work is insane value for money. I find myself more exciting by "value" than hypothetical ceilings because I'm just not in that budget category.
i assume you mean 6 Luna, because we've had 5.6 luna for ages
That was true for me four weeks ago, but 2-3 weeks ago Luna turned into drivel in essentially the same complexity of task. I feel it came back somewhat in recent days but does feel like it's being manipulated.
Interesting, I have noticed so such collapse.
Have you ruled out the possibility that your system prompt, AGENTS.md, or increasing codebase complexity are not to blame?
Hey! From the Artificial Analysis team. We also have a model releases page which shows all reasoning efforts (not just max), including the trade off curves https://artificialanalysis.ai/models/releases/claude-opus-5-...
I have experienced this with open weight models too. "Max" is for benchmaxxing the intelligence metric and is not meant for use in productive work. Like drawing pelicans.
I tried to replicate your test but after 8 minutes and more than 50 lines of "thinking" by dumping seemingly random loading-screen strings like "Placing the sun, clouds, seagulls, and sea backdrop" and "Positioning the tail feathers and calculating handlebar geometry" I gave up and cancelled the task.
I've asked Opus 5 Max for what I thought were easy tasks at work to be completed. It always fails after reaching a tool limit.
I asked Opus 5 High for the same task and requested it to minimize tool usage. It produced an answer in a few minutes that I was deploying to my target platform about 30 minutes later.
I'm amazed they didn't test xhigh thinking mode explicitly to ensure it didn't exceed the 128k thinking budget allocation. I guess pace of development gets away from everyone, even OpenAI.
On the Message Batches API, Opus 5.5 supports up to 300k output tokens with the output-300k-2026-03-24 beta header.
https://platform.claude.com/docs/en/models/opus-5-5/overview
The old analysis paralysis
How did you get the reasoning trace? Is it the actual one or the summarized one?
It's the summarized one returned by their API.
Piping the visible reasoning trace through their token counter API (I use https://tools.simonwillison.net/claude-token-counter for that) counts 27,888 tokens, so it's definitely a summary of the 128,000 actual token trace.
Pick 10 reasonably intelligent people randomly. How many of them do you think can draw a pelican on a bicycle? I bet most of them can barely even draw a bicycle.
This whole test tells me nothing.
The next person who proposes to use it should draw it themselves first.
The fact that bicycles are hard for people to draw was one of the inspirations for the test.
But:
1. I don't want my AI to have super-human intelligence. It would generate code that I do not understand. A coder with human capabilities is better for me.
2. We're interested in AGI. If the AI reaches human intelligence, that's a milestone. Drawing bicycles with pelicans is superhuman. Hence not a relevant test.
"Drawing bicycles with pelicans is superhuman."
It really isn't. Many humans can draw a bicycle, and a pelican, just fine.
It is harder than you may think for many people... https://www.gianlucagimini.it/portfolio-item/velocipedia/
No surprise then that the default effort level for this model in Claude Code is Medium, even if you had Opus 5 set to High...
I am very interested in why it was able to overthink that much. In the 20-30mins of Max reasoning I've had so far, I'm not having the same issues (yet).
Do these evaluations get re run a few weeks after launch? I started doing that yesterday for our internal dataset and found Sol’s performance had regressed to be equal to Luna’s. Granted this was one run, but something I’m becoming more concerned about, the model providers want to quickly prove they’re the best, people switch to them, then they pull the rug.
Can you share more information on your methodology?
GPT-5.6 Sol's performance in the API should not change over time. If it has, that's a severe bug and we'll look into it.
We do sometimes tweak ChatGPT settings (e.g., tools, system prompts, efforts) over time, but we never play games to juice evals at launch times. You should always get what's advertised.
(I work at OpenAI.)
This is all code review runs via OpenRouter with a Pi harness, and it’s totally possible there are shenanigans going on elsewhere.
Yesterday, I ran an identical bug identification dataset from two weeks ago, saw a 50% drop from a few weeks ago, putting Sol on the same level as Luna. Sol had been finding 40-50 bugs per set, then dropped to 25, matching Luna’s performance. Not enough to establish a pattern, but enough to raise eyebrows.
Our review workflow is public if you want to peruse it, dataset isn’t. The process isn’t really stabilized yet either as I have to balance running this against limited budgets.
https://github.com/BiggerPockets/.github/blob/main/.github/w...
How many tasks were in this dataset?
If it's a single task where it dropped from 50 to 25, it could be random variation (not saying it is, but it could be). If it's the mean over hundreds of tasks, that suggests a problem with either the eval code/harness or our API.
Do you have any hypothesis for why this experience is so consistently reported by users (anecdotally)? Seemingly across all providers.
Not the one you asked but I genuinely think it’s possible that a part of the answer is the psychological effect of getting used to models performing well and picking up more if they fail, but also … If you have a product that works well for 95% of software engineering tasks (for the sake of this argument), with a large number of users there will inevitably be some poor schmucks who get bad results multiple times in a row. It would be highly unlikely if that _didn’t_ happen at all. Now assuming you have millions of users, there will always be groups of thousands that experience this, and if those people go online to complain, it will look like an actual issue when it can just be explained by randomness and large numbers.
My best guess is random variation and rising expectations. I once saw someone do in depth manual testing of 3 different models and write up a whole report on their perceived strengths and weaknesses… and then discover that all 3 models were identical. Very good reminder of why the scientific method is important.
Serial testing over time is much less reliable than side by side testing, and even when I do side by side testing, I try to look at multiple attempts per prompt. Seeing multiple per prompt helps me realize how much intrinsic variation there is. My brain always wants to see patterns even when there isn’t enough data to prove them.
But the chat version does change over time, correct? It has been my experience that Sol's performance has deteriorated significantly.
ArtificialAnalysis tweaks stuff until newest big proprietary model is on top, not you haha
Well there is at least the degradation tracker from Margin labs for Sol and Opus: https://marginlab.ai/trackers/codex/
It looks pretty consistent? At least within reason for a stochastic model. Or am I missing something?
What you may be missing is that they probably track the API performance.
Subscription plans may be subject to other regime, e.g. lowering the thinking budget when the API is under heavy load, etc.
Need one of these for openai:
https://marginlab.ai/trackers/claude-code/
Wouldn't that be basically https://marginlab.ai/trackers/codex/ ?
These tests need to be sampled continuously.
Moreover, the tests should be randomized somehow to ensure the models don't memorize the answer.
Half the cost per task compared to Opus 5, comparing high effort to high effort. That's just really nice.
Edit: https://artificialanalysis.ai/models/claude-opus-5-5?models=...
Tasks are completed in about half the time too. Although we'll see if it slows down in a few weeks as Anthropic's model services are prone to do.
That is a lot. I thought Anthropic models would just do the opposite because they are greedy for money.
Greed is not what's driving these prices, its cost. They considered very much in the red.
The way to make money in this business right now is to make the absolute best product and convince everyone they need to use your thing, especially considering the training cost is a very large factor in the overall costs and you amortize that by selling inference.
Astra High is slightly cheaper at $1.73 vs $1.82 for Opus 5.5
The UI/UX seems impressively bad. DeepSWE's cost curve has a better, more obvious way to sort by only the top level of reasoning to avoid 80% of the graph just being the same 3-5 models at their 8 different reasoning levels...
It's also less clear what a lot of their metrics mean. Does Cost per Task include only things that can be verified to work and passed? As best I can tell, it does not.
I'm less concerned if one model's cost per task is $0.10 and another model's cost is $1.50 if the $0.10 task got it right 1% of the time and the $1.50 model got it right 66% of the time.
An equalized / weighted cost/time per task is much more valuable - being massively penalized for taking a lot of time and ultimately not passing when OTHER models did pass.
Nice catch, AA only shows max effort by default and I got disappointed thinking it's a token guzzler though: https://artificialanalysis.ai/models/claude-opus-5-5?models=...
Not sure about how adaptive reasoning works though as they mention adaptive reasoning for every reasoning level
This continues to show that these foundational models are only slightly better than open weight models but cost around 100x as much. The history of tech is riddled with “good enough” eating “best” for lunch all day long. Unless the big labs come up with a viable business plan pronto it’s looking like AI will be no different.
There are no prizes to be won by having the best model that’s 100x the price of something that’s good enough for 99% focuses cases.
Pretty much this, I don't just see how ASI could happen with current LLMs without big theoretical breakthrough so that's why 'phasing the frontier' and 'distillation attacks'.
Unless the 100% is super intelligence. Thats the bet being made with our economy/society.
Fingers are crossed on this one. I had gone back to using opus 4.8 instead of using opus 5. Simply because 4.8 is much better at remembering what it's doing and following instructions than 5. 5 often had a tendency to get halfway through solving a problem and then I would have to stop it in the middle, because it had lost its way and was going off on a tangent rather than dealing with the problem. In that respect, 4.8 was a lot more stable.
"High" to me looks like the one to use. https://artificialanalysis.ai/models/claude-opus-5-5-high
Many benchmarks start to plateau after high, this benchmarks better than Fable, and my initial tests show it working really well.
"somewhat expensive when comparing to other models of similar price"?
That says something about your selected range, and nothing about the model.
Was about to comment on the same. Do they mean similar capabilities or is this actually intended insanity
Interesting to see it now. I've used it a bunch before it came out and i pretty much didn't notice it. It might have been slightly better code quality, but still not great in that. I guess it just was slightly less frustrating to work with, but still AI...
I think we're hitting the ceiling of most models capabilities. We're getting to a point where too much training apparently creates models that hack people.
That ceiling set in… June. Yes. Slowing down for sure.
I am starting to wonder how useful these benchmark still are? Aren't all of these models trained to ace these benchmarks?
In any case they give an indication, but I am increasingly looking to real world feedback from real users. I have a solo project I (voice-to-text typing voicewink.app) and I'm using both the Claude Code and Codex coding harnesses and multiple agents doing reviews on the same code.
This has given me real tangible results to compare on real work. Conclusion: GPT-6, Opus 5 / Fable and Grok are "top tier", with Claude good at planning and executing and Codex / Gpt-6 better at finding bugs and fixing them (but tending to overengineer), and Gemini and Kimi 3 clearly behind in capability (more so than the benchmarks suggest in my opinion).
Any thoughts?
I find codex to overengineer for routine tasks compared to claude. I use codex mainly for infra tasks.
Yes, exactly my experience. Do you have tricks / approaches to keep Codex from overengineering?
The very first sentence:
> Claude Opus 5.5 is amongst the leading models in intelligence, but somewhat expensive when comparing to other models of similar price.
What does it mean for a group of similarly priced things to have one that's somewhat more expensive? Cost relative to cost means nothing. You'd think they are would talk about performance relative to cost.
Well yes, but it's also slightly smarter compared to other models of similar intelligence.
The output style and verbosity with Opus 5.5 is a very big improvement over Opus 5. I predict Opus 5 will be a version with a sudden churn.
Max reasoning seemed to get stuck for me too. My default is Medium which seems to work pretty well both in tight and longer running loops.
It sholud definitely not be more capble than Fable. It seems tight comparision but cannot find out its details. I'll use this and check the perceived performance.
Fingers crossed on this one. I had gone back to 4.8, because 5 was not very good at following instructions or remembering instructions. I found myself repeating quite often what I wanted and what I was trying to do. Opus 5 was more like haiku than it was 4.8 in that respect.
Why do I get the feeling this 'pacing' will be one of "yeah alright guys, let's pace ourselves while I'm ahead".
> somewhat expensive when comparing to other models of similar price
-_-‘
Definitely a quiet release. Perhaps pre-empting marketing for Astra public release?
All anthropic launches are like this. They just post it and don't particularly put out the PR sprint that OpenAI does with videos, livestreams or whatever.
(Except for of course Mythos and whatnot when they want to push the whole "safety" thing)
I am begging you on my knees to please stop posting this cringe.
The model is just out. It could be good, great even, I don't know. But I do know that this index has Opus 5, one of the worst releases of 26, ahead of Astra. What information are we supposed to deduce from number having gone up?
What's more valuable than a good benchmark? IMO a benchmark that has been run against very many competitors and versions. Collecting data has something going for it, and it's up to the readers to interpret and make the best use out of it.
This index doesn't have "Astra" and "Opus 5". Every entry with corresponding data is a `(model, reasoning)` tuple.
So I'm unclear what you're actually saying and wondering if you've missed that. Are you saying that at every reasoning level it says Opus 5 beats Astra? I just compared Opus 5 high to Astra high and it has Astra as generally better than Opus.
I don't try to say that the parent commenter is right in any way, but the two models' "high" settings probably doesn't mean the same thing. So probably comparing only them is not useful.
You forgot to include whatever you're proposing instead.
"Trust me bro, Astra is better" isn't perhaps as useful as you seem to believe. I'm not even saying it is right or wrong, just that my opinion on this topic is still just one additional subjective data-point.
Only thing I wish with these benchmarks is that they would run repeat tests every couple of months. Then re-rank based on that too. We've seen a lot of performance fall-off after a couple of weeks with new releases.
Yeah unfortunately the benchmarks are usually provided by the company themselves, unquantized, thinking set to extra-extra-ultra-high, best of 10 runs, etc etc. It's hard to know how that's going to map to real world users.
That it's better in specific ways? What difference does it make when it came out? The benchmark results are not going to change unless they're messing with the model.
Yes, but crucially, in ways that are increasingly decoupled from any practical pattern of usage, considering that I wouldn't see how you can argue that Astra is worse than Opus 5.
One man's modus ponens is another's modus tollens I guess.
so its more intelligence than fable?
can anyone help me?
China who?