On a related note: Would love to see cyclomatic complexity as a supporting metric for readability. I increasingly find AI-adjacent and AI-written code suffering from this.
On the one hand, I absolutely agree that we should try to measure and reduce verbosity that LLMs are slopping into our codebases.
On the other hand, the fact that Astra is "winning" this bench is not great signal for me, since imo Astra produces basically unreadable code. Astra plays code golf. My own revealed preference is that I'd rather have the excessive comments and tests of Claude rather than the inscrutability of Astra.
I keep going in between the two and I'm currently in an Astra season, deeply disliking the tone and modes of Claude.
I wonder if the two will diverge enough that this becomes the tab-vs-spaces/vim-vs-emacs of the LLM era and they'll manage to keep users moving over at some point.
Nobody is going to care about how much code an AI system generates in the very near future, some even now, once everyone understands the scale at which a vector space truly operates at. Complexity is not much of an issue for an AI system.
LLM systems start to Frankenstein things together at a very modest amount of code. The lack of abstractions cause huge and glaring problems for anything that needs to run predictably.
Abstraction is an issue for every AI coding agent. They are painfully useful, but need constant badgering in the right direction.
After 22,948 commits, 13,720 bugs and -2,899,391 lines eliminated, I've seen this too but a properly designed AI pipeline's ability to optimize and course correct with a little badgering supersedes any Frankenstein-like outcomes.
At this point, I really want all AI benchmarks to use the best coding/writing model that normal people can run locally.
For now that seems to be Qwen 3.8 27B that requires ~28GB of RAM (Q4) which is pretty doable for anyone with a beefy GPU or slightly-above-average mac.
I want to know how close open weights (open models) are to all these leading commercial models. Yes, yes, local models aren't as good, but that distance is closing as local models are already better than what we had just over a year ago in many cases.
Same with Text-to-Image generation (Flux 2, Z-Image-Turbo, etc..) or text-to-speech (Parakeet, Whisper, etc..).
Does anyone have ways to reduce the complexity and verbosity of code generated via agents?
Anything that can increase code quality would help.
On a related note: Would love to see cyclomatic complexity as a supporting metric for readability. I increasingly find AI-adjacent and AI-written code suffering from this.
On the one hand, I absolutely agree that we should try to measure and reduce verbosity that LLMs are slopping into our codebases.
On the other hand, the fact that Astra is "winning" this bench is not great signal for me, since imo Astra produces basically unreadable code. Astra plays code golf. My own revealed preference is that I'd rather have the excessive comments and tests of Claude rather than the inscrutability of Astra.
Yeah the issue is that while code that’s too verbose is bad, code that’s too terse is as well.
Astra codes like it has to pay by the keystroke (although in a way, it indeed does)
I keep going in between the two and I'm currently in an Astra season, deeply disliking the tone and modes of Claude.
I wonder if the two will diverge enough that this becomes the tab-vs-spaces/vim-vs-emacs of the LLM era and they'll manage to keep users moving over at some point.
Nice project! But I wonder does less coding actually means less work for whoever reviews it?
Nobody is going to care about how much code an AI system generates in the very near future, some even now, once everyone understands the scale at which a vector space truly operates at. Complexity is not much of an issue for an AI system.
LLM systems start to Frankenstein things together at a very modest amount of code. The lack of abstractions cause huge and glaring problems for anything that needs to run predictably.
Abstraction is an issue for every AI coding agent. They are painfully useful, but need constant badgering in the right direction.
After 22,948 commits, 13,720 bugs and -2,899,391 lines eliminated, I've seen this too but a properly designed AI pipeline's ability to optimize and course correct with a little badgering supersedes any Frankenstein-like outcomes.
At this point, I really want all AI benchmarks to use the best coding/writing model that normal people can run locally.
For now that seems to be Qwen 3.8 27B that requires ~28GB of RAM (Q4) which is pretty doable for anyone with a beefy GPU or slightly-above-average mac.
I want to know how close open weights (open models) are to all these leading commercial models. Yes, yes, local models aren't as good, but that distance is closing as local models are already better than what we had just over a year ago in many cases.
Same with Text-to-Image generation (Flux 2, Z-Image-Turbo, etc..) or text-to-speech (Parakeet, Whisper, etc..).
FWIW, I have mixed feelings about Artificial Analysis but they at least let you filter by size class and by proprietary versus open.
https://artificialanalysis.ai/?intelligence=agentic-index