The example of 'labs are optimizing for this' is the wrong thesis.
AI arbitrarily generating something of 'apparent sophistication' is not that hard - being able to produce it to spec that has invariable vague elements - and then being able to rationally modify it is the problem.
Look at image gen: you press the magic button and get a 'Pelican on a Bike' - but you can't just change the 'hat' of the Pelican. You have to press the magic button again, and you get a whole different Pelican on a Bike.
This is the fundamental conceit.
It's akin to the conceit that 'writing the code is the work' - but it's not - it's the research, the design, docs, integration and all that 'know-how' that has to 'sit somewhere' so the shape of the thing output can be adapted and moulded.
Arbitrary code output is not quite worthless but almost, it's the the '80% that means another 80% and then anther 80% to go'. It's like a nicer stating point.
The research capabilities of the AI, which don't make for nice demos, are arguably more powerful.
> Look at image gen: you press the magic button and get a 'Pelican on a Bike' - but you can't just change the 'hat' of the Pelican. You have to press the magic button again, and you get a whole different Pelican on a Bike.
I do understand what you mean and perhaps I am trying to treat it as a problem to be solved and challenge accepted but my first thoughts are if its an vector image like SVG (the famous simonw's pelican on a bike benchmark)
Then you could in theory have a layered approach and then just change the code of the hat so for example (IIRC) <--Hat--> Code. <-- Bicycle-->
So basically I am suggesting modularizing of concerns of areas so that you could have better autonomy over what exact thing you wish to change.
Though I imagine that you and I might be saying the same thing and you are suggesting that the crux of the argument is exactly that you need specific know how in reaching to that said modularity where you can best use AI.
When you might need the generation of the SVG as compared to image generation because you realize that your project might need the pelican changing lots of hats and image gen wouldn't be feasible but for that to actually say to AI, You might need to know in the first place that SVG or (HTML?) might be better use cases for this. Again, I must admit that I am not an expert in this so I can't absolutely comment on what the best thing for this particular use case could be and maybe that might be your point.
Have we arrived at similar conclusion or perhaps, is there more nuance to it?
On a side note: I actually once had a real use case of where I needed layered approach similar to Figma but generated through AI. preferably something which can just work through good ol chat app UI which could give an index.html or other code for this modification/layered approach.
Like recently there is https://bento.page which has been somewhat similar for this in some sense but for pdf's. So is there something different but for image-ish thing? If there is an expert lurking here, I would love to know the answer and gain some knowledge about it, thanks :-D
Fundamental conceit? My dude, uou can change the hat on the f*cking pelican. Image-to-image masks and controlnet have been a thing for literally years now.
I keep seeing Astra make beautiful 3d stuff online, yet when I feed it some old school RuneScape assets (even tried with some very detailed guidelines) and asked it to generate some new plausible assets it failed horribly.
I think there’s still something really off with current (frontier) models when it comes to creating “novel” stuff? Even 2004 style graphics..
My understanding is that the people are conflating the ability to translate and replicate as creativity/learning. Not to say that its not possible but its a long way to go for sure.
2. Feed it with reverse engineering research of such assets;
3. Ask it to use C++ and SDL2/3 to write a game engine to draw maps, generate enemies, make cities and shops. It doesn't have to be faithful to the game, just something passable is good enough;
This sounds unconvincing, because a) pelican test is subjective, there's simply nothing to leak as it has no available direct answers and maybe an extremely faint preference signal, and b) the same small models actually do perform well when you change the subject. Some models are genuinely trained to be better at some domain, such as 2D layouts or vector graphics in this case. It all depends on particular recipes and datasets. Which is the actual reason these tests are poor as vibe checks: they don't do anything to disentangle generalization, memorization, and training preference. One-shotting popular software in particular is definitely not a good test of anything as memorization is going to dominate it.
AAII is also not very useful, neither is any generic score/benchmark. If you want a weather forecast you aren't looking at the average temperature of Earth.
(actually when did the term "one-shot" get hijacked to mean something other than "one example"?..)
> Some models are genuinely trained to be better at some domain, such as 2D layouts or vector graphics in this case. It all depends on particular recipes and datasets.
I'd argue this is inherently true for every single model today, none of them have completely generalized to be able to solve any task, so whenever people come up with new evaluations and benchmarks, all the models score relatively poorly initially, until researchers start to tune the models to do well in the domains that the evaluations and benchmarks tests, and then we see strong improvements in that domain, which then tapers out to incremental improvements, and some other domain is chosen to be the new focus.
Models aren't better agents today merely by chance, but because it's explicitly part of the training data. They do well with software because we've talked so much about software on the internet until this point and that's part of the training data, but pit them against problems people don't talk so much about, and if the labs creating and training these models didn't consider those problems, then the model will pretty much suck at it.
I guess eventually they will literally cover every single task the model could ever come across, at least some variant/permutation of it, but until then every benchmark/evaluation will just uncover "did the labs consider this and who considered it most important before/during training?" basically.
> That’s the problem, these tests can’t tell you how good a model is anymore because it’s trivial for labs to optimise for exactly these tests by the next release.
But they don't prove the claim. Are the models amazing at recreating Minecraft, but the second you swap the word Minecraft out with another game or a custom game, it shits the bed?
That's not what I see. My feed is full of people using Astra to recreate all sorts of games from Diablo to some random idea they came up with, in ridiculously polished detail like animations that would have taken me weeks of iteration in gpt-5.6-sol but it was a single shot by Astra.
Can you point me to one of these? The only Astra game I tried was the incredibly janky Mario Kart clone from OpenAI where you could fly by spamming jump
I think it's a genuinely good implementation of the concept. It looks good and plays reasonably well and captures what it needs to capture from the original.
(Update: OK the sword fighting isn't good. As I got deeper into the game my positive first impressions wore off.)
I haven't tried any of these demos, but I'm not surprised they stop impressing once you go deep.
What got be mostly impressed were the demos of Astra doing computer use. At my job I do some RPA and can appreciate how challenging it can be. Yet they make it look extremely easy to operate a tool like Blender at super human speeds.
For every person using Astra to create a game from some random idea and sharing the impressive result, there are an unknown number who have tried the same thing and gave up in frustration.
"Generate a SVG of a PlayStation 4 controller as nicely done as you can"
Time taken: 11m 54s
No image was used
I was expecting 10s not 12m; if it takes 12m to do it, I would rather try to do it myself. And what it means "No image was used", wasn't it trained on bunch of images including probably console controllers images?
Build a flowsheet based steady state chemical process simulator, then use it to simulate and optimize a full scale oil refinery.
1) Building a solver engine that works at this scale is not a trivial problem, and the successful ones rely more on heuristics than some categorically different solution approach.
2) Defining the engineering equations relevant to this task is relies on understanding what level of fidelity is required to answer the questions people ask of steady state process models.
3) Knowing the thermophysical properties of chemicals and crude oils is possible from the open literature, but the information is diffuse and different correlations are applicable in different situations.
4) Creating a GUI which converts a flowsheet into matrix math is non-trivial, although a sequential modular approach is a bit easier.
5) Defining large scale models in such a way that they solve robustly is as much art as science. For example, completely closed recycle loops like refrigeration systems are a nightmare for solvers, so it is often better to define them in an open-loop way.
6) Optimization involves knowing the relevant commodity prices, but more importantly how to define the constraints on the model so it doesn't just say to produce infinite gasoline.
7) Troubleshooting the inevitable convergence failures is also as much art as science. There are a large number of diagnostic techniques, but fundamentally you need to be able to relate what is happening during the solver iterations with the intent of your model because more often than not the problem is that you've asserted something impossible, redundant, or irrelevant.
I used to think that way about SVGBench, after all labs can just train on the test set, right? It turns out the task was highly generalizable. Try designing a logo and you quickly see the gap between models visually. Even though there is still a gap between "shiny demo SVG" and actual real-world use.
Same thing happened with MineBench, basically SVGBench+3D, until that got "saturated".
Remember spinning hexagon bench[1]? Or the AI World Clocks[2]? Yeah, that used to be hard for frontier models.
Creating games is the next iteration that still has some signal left. Assets + Game logic + UI + Sound, it let's you assess a model's "taste" very quickly.
What else is left, once all these benchmarks get saturated?
> What else is left, once all these benchmarks get saturated?
I was thinking this thing quite recently and wish to write a blog post about it but IMO, I believe that the next AI benchmarks would be profitable businessmaking. There are/were already some trials done (The famous WSJ/Anthropic one[0]) and some shop iirc in SF which is doing this and recent example of trying to start a lemonade stand with AI[1]
In my opinion, it can't really get more meta than that plus on the more humour side, it could help these companies make some dollars as they desperately need it.
I always wonder if these AI models can self-autonomously generate profit, why would these AI companies try to let you generate the main profit while you just pay them one time or a very small amount in tokens. Why should these AI giants allow a person to be a middle man in the first place in some sense and instead not just run these agents autonomously.
Maybe it could be because these AI agents might not be the best in such a meta task (money making/business) but as such, it does feel to me that it might be a good benchmark in the first place.
On some occasions, as I have come to understand Hackernews, its not really about the content but sometimes just about the Title or literally just the sentiment behind it.
Maybe we are all being shown recreation of Minecraft or other video games as a benchmark and maybe there is a discussion to be had of the statement of (is it a benchmark or not). I, for one, just appreciate that we are all here who could talk about this topic that I was just thinking about and can have a good discussion about.
In some sense, its speaking some thought that everyone (many) might be thinking internally out loud.
Also on a related topic, I have a question that I wish to ask, there are many text files that I have which are me just writing extremely crude thoughts. I once tried to let AI re-write it in a blog format and I found it to somewhat express what I was saying, perhaps saving me hours.
I still don't upload those blog posts because I don't wish to write AI generated blog posts even though I might have spent more than hour writing those original thoughts. To me, its because I don't exactly know how to disclose things. Some of the original thoughts for example might be quite PII/sensitive. Should I share the exact prompts or the original thoughts and what are the correct hygienic ettiquetes towards this nuanced topic?
It would also take me more than hours to edit the posts and I might still not be confident in my ability of trying to create that blog post and might end up never finishing that public blog post in the first place due to procastination.
I also don't just want to sometimes write AI generated blog posts because its a slippery slope. Without proper disclosure, the other person might not even know if another human on another side of planet even gave a thought about it or not. For all intents they might think that nobody really wrote it but only an AI and just "write a blog post about X"
I also would dislike being the person on that end and I also dislike reading AI generated posts. It produces visceral reaction of hatred and closing the tab when I read AI generated slop.
I ironically wanted to create a blog post about this nuance but I couldn't capture it and I then had AI write that blog post but I didn't post it. I have had some very nuanced meta experiences about this exact topic to be honest and I would love to hear some thoughts & opinions about it.
I have the thoughts but because of all of this, in some sense, a lack of means to share. I would wish to hear other peoples thoughts about those thoughts that I had and if other people resonate with it or not but alas, some of them are just text files within my computer.
However, what is meaningful is whether something is able to create usefully adjacent output, like "let's make Minecraft, but with marching cubes, subdivision surfaces, and global illumination... and behaviorally accurate pandas..." (or something like that).
I have an 11 year old, and most of his game ideas are adjacent to other games he's played. He can make those now, or, at least, enough that he can see where it works and where it doesn't.
Yes, it's not a scientific benchmark but it's a good heuristic.
For a better eval, create a one-page prompt / mini spec related to whatever you're using the LLMs for, and see how well a particular one works for what's important to you.
I really like these tests for what its worth and I see them on youtube sometimes. I would like to ask a few things though
TLDR: Basically focusing on recreating pay to win (mobile or otherwise) games and recreating them non pay to win perhaps instead of focusing on recreation of minecraft for benchmarks could have a genuinely meaningful impact, and making these games portable as well could be another interesting idea. [so it can be played on any operating system/device so using web or if native then for (Android/IOS/Linux/Windows/MacOS) using game engines like (Preferably godot)/Unity/UE.]
Could the test focus more on pay to win games with unique dynamics.
For example: I literally wanted to create a clash royale recreation because clash royale is a highly pay to win game.
The game is unique enough to have memories but is pay to win enough that it ragebaits me as to what its current situation is, its so pay to win now. A recreation would have genuine effect whereas yet another minecraft clone wouldn't.
I have some fond memories of the game and my brother and I used to play it (my brother moreso than me). Also clash of clans and clash royale famously prevented windows users. So I remember downloading bluestacks to play it on laptop but it required 2GB of ram and back then we only had 1GB. (Ironic that we might come back to that time)
Another question that I have for you which I have been genuinely curious is: who is footing the bill for these benchmarks and youtube videos. What are the economics surrounding it?
I imagine the bill to run quite hot sometimes and I find running these benchmarks to be quite unaffordable personally.
I also wish to ask if you have any theories as to why not people on Youtube share their videos. I found this [0] Minecraft clone by Fable 5.1 extremely good yet they haven't shared the source. I am unsure as to what exact reasons might be behind most youtube videos on recreation with AI to not share the actual code. I don't find much rationale in not sharing AI generated code of a recreation of a game especially if one is making a video about it. So thanks for once for actually sharing the output code as well as I surprisingly found it to be a bit rare!
But they're not recreating Minecraft. They're recreating one of the most common tutorials on the internet that shows a simple voxel overworld and nothing else. That's not Minecraft.
I think you meant “pedantic distinction”. Everyone reading that comment understands that recreating Minecraft is a very common tutorial for game development, not literally one of the most common tutorials on the internet in general.
“Model X CHANGES EVERYTHING!” (surprised Oculus Rift face without the headset)
Some one-shot prompts for quick impact.
I get it with the incentive structure — honestly not trying to shit on any creators (hate the game, not the player) but seeing it every day is just getting tiring.
Though I somewhat think some benchmarks are silly like the article says. I saw someone on YouTube recently take a picture of a building across the street from them (seemed like it was in NYC), and asked GPT 6 Astra to make it in blender. It did a surprisingly good job in 30 minutes. So though these benchmarks don't seem to mean much you could always add a touch of randomness to them like the person in the YouTube video did, but the problem with that is how would you compare the benchmarks in any clear way if they aren't even consistent? Regardless it appears LLMs are getting this good at the general task and not just at the particular instances of said task.
I partially owe my career to benchmarking AI with Minecraft, so, I'm going to disagree with the author on this one. Games are a great way to test a model, it's not that deep.
Running Firefox here, no problems whatsoever even with UBO and Privacy Badger disabled (the twitter embeds get blocked at the DNS level, but I doubt those are the problem).
Mac here. There's nothing all that interesting going on with the JS on that page, so I would suspect a bug with Firefox and/or your desktop environment.
> The site’s graph incorrectly uses 32 texture bindings, exceeding its WebGPU device’s limit of 16.
> Firefox 153 mishandles that failed rendering operation and leaves the compositor waiting
So a firefox + website bug
With an updated firefox, the website works except the mindmap on the top right
> Uncaptured WebGPU error: Too many bindings of type SampledTextures in Stage ShaderStages(FRAGMENT), limit is 16, count was 32. Check the limit `max_sampled_textures_per_shader_stage` passed to `Adapter::request_device`
I’ve used Astra for the past day and a half. My layperson’s review is that it is impressive at computer use and 3D reasoning, and fails in similar ways to 5.6 Sol at similar rates when it comes to coding. I have no idea how it scored so high on SWE benchmarks because so far it has been very “mid” as the kids say.
Most of my and my peers PortCos run their own eval and benchmark sets, simply because they know what they need best.
The reality is, capabilities have largely converged across foundation models over the last 18 months, and much of the value add is coming from the harness layer itself now.
This has been the operating assumption for me and my peers, and has largely played out that way.
That said, this has always been an issue with benchmarking since the very beginning. DB Benchmarks, compute benchmarks, and others that were external facing were always inherently a content and product marketing tool. The actual internal benchmarking used to model, understand, and enhance your product was always a closely held secret.
Most of these conversations are happening, but largely in person and not on HN.
It's not "models have stagnated" but "models released at the same time are on the same level". Improvements are still real but the relative gaps between OpenAI, Anthropic, Meta, Grok, Gemini and open models are closer than ever. That doesn't mean progress is slowing down, it's just more widely distributed.
> The reality is, capabilities have largely converged across foundation models over the last 18 months, and much of the value add is coming from the harness layer itself now.
Can you please elaborate on what are your thoughts on open weights models (GLM 5.3, Kimi K3, deepseek etc.)
and if the value add is coming from the harness layer itself, then thoughts on open source harnesses (there are so many harnesses but to name a few: opencode, pi [omp as well], maki, codex is OSS as well, fx.sh) and you can always combine them with skills (Obra/superpowers, matt pocock skills plus using these skills and others to create some other custom skills tailored to your use case as well)
And what about the combination of both now with this cheap open weights models + open source harnesses and other things to compete over the closed garden ecosystems?
How does that comparison follow in reality
Could you in theory use these methods to save on the massively expensive $$$ token spending on Anthropic/OAI?
(Personal anecdote but I have GLM 5.3 + maki [sometimes omp/opencode but mostly maki] and its good enough for most use cases out there that I have and I dont know of too many use cases outside of say recreation of games for examples maybe that I would prefer complete SOTA models. I would also love to know where you believe that SOTA models absolutely do still make the difference discounting the benefits provided by the harness.)
@notch Hey Markus, can I get a refund on my alpha distro of minecraft? I think the currency is worth more than it used to be considering how many versions there are now :)
Right answer wrong thesis.
The example of 'labs are optimizing for this' is the wrong thesis.
AI arbitrarily generating something of 'apparent sophistication' is not that hard - being able to produce it to spec that has invariable vague elements - and then being able to rationally modify it is the problem.
Look at image gen: you press the magic button and get a 'Pelican on a Bike' - but you can't just change the 'hat' of the Pelican. You have to press the magic button again, and you get a whole different Pelican on a Bike.
This is the fundamental conceit.
It's akin to the conceit that 'writing the code is the work' - but it's not - it's the research, the design, docs, integration and all that 'know-how' that has to 'sit somewhere' so the shape of the thing output can be adapted and moulded.
Arbitrary code output is not quite worthless but almost, it's the the '80% that means another 80% and then anther 80% to go'. It's like a nicer stating point.
The research capabilities of the AI, which don't make for nice demos, are arguably more powerful.
> Look at image gen: you press the magic button and get a 'Pelican on a Bike' - but you can't just change the 'hat' of the Pelican. You have to press the magic button again, and you get a whole different Pelican on a Bike.
I do understand what you mean and perhaps I am trying to treat it as a problem to be solved and challenge accepted but my first thoughts are if its an vector image like SVG (the famous simonw's pelican on a bike benchmark)
Then you could in theory have a layered approach and then just change the code of the hat so for example (IIRC) <--Hat--> Code. <-- Bicycle-->
So basically I am suggesting modularizing of concerns of areas so that you could have better autonomy over what exact thing you wish to change.
Though I imagine that you and I might be saying the same thing and you are suggesting that the crux of the argument is exactly that you need specific know how in reaching to that said modularity where you can best use AI.
When you might need the generation of the SVG as compared to image generation because you realize that your project might need the pelican changing lots of hats and image gen wouldn't be feasible but for that to actually say to AI, You might need to know in the first place that SVG or (HTML?) might be better use cases for this. Again, I must admit that I am not an expert in this so I can't absolutely comment on what the best thing for this particular use case could be and maybe that might be your point.
Have we arrived at similar conclusion or perhaps, is there more nuance to it?
On a side note: I actually once had a real use case of where I needed layered approach similar to Figma but generated through AI. preferably something which can just work through good ol chat app UI which could give an index.html or other code for this modification/layered approach.
Like recently there is https://bento.page which has been somewhat similar for this in some sense but for pdf's. So is there something different but for image-ish thing? If there is an expert lurking here, I would love to know the answer and gain some knowledge about it, thanks :-D
Fundamental conceit? My dude, uou can change the hat on the f*cking pelican. Image-to-image masks and controlnet have been a thing for literally years now.
I keep seeing Astra make beautiful 3d stuff online, yet when I feed it some old school RuneScape assets (even tried with some very detailed guidelines) and asked it to generate some new plausible assets it failed horribly.
I think there’s still something really off with current (frontier) models when it comes to creating “novel” stuff? Even 2004 style graphics..
Or am promoting it wrong?
It's funny seeing this when I was going to mention runebench
https://maxbittker.github.io/runebench/
That's playing RuneScape though? Unless the connection is just that these are both RuneScape related?
My understanding is that the people are conflating the ability to translate and replicate as creativity/learning. Not to say that its not possible but its a long way to go for sure.
I'm wondering if the following works:
1. Feed it with Ultima II/III assets;
2. Feed it with reverse engineering research of such assets;
3. Ask it to use C++ and SDL2/3 to write a game engine to draw maps, generate enemies, make cities and shops. It doesn't have to be faithful to the game, just something passable is good enough;
4. See how it works.
The type of people that can’t get results are the people that don’t share the process that didn’t work for them.
It's a veeeeery specific artstyle
Did you try blender MCP and prompt for low-poly vertex colored assets?
This sounds unconvincing, because a) pelican test is subjective, there's simply nothing to leak as it has no available direct answers and maybe an extremely faint preference signal, and b) the same small models actually do perform well when you change the subject. Some models are genuinely trained to be better at some domain, such as 2D layouts or vector graphics in this case. It all depends on particular recipes and datasets. Which is the actual reason these tests are poor as vibe checks: they don't do anything to disentangle generalization, memorization, and training preference. One-shotting popular software in particular is definitely not a good test of anything as memorization is going to dominate it.
AAII is also not very useful, neither is any generic score/benchmark. If you want a weather forecast you aren't looking at the average temperature of Earth.
(actually when did the term "one-shot" get hijacked to mean something other than "one example"?..)
> Some models are genuinely trained to be better at some domain, such as 2D layouts or vector graphics in this case. It all depends on particular recipes and datasets.
I'd argue this is inherently true for every single model today, none of them have completely generalized to be able to solve any task, so whenever people come up with new evaluations and benchmarks, all the models score relatively poorly initially, until researchers start to tune the models to do well in the domains that the evaluations and benchmarks tests, and then we see strong improvements in that domain, which then tapers out to incremental improvements, and some other domain is chosen to be the new focus.
Models aren't better agents today merely by chance, but because it's explicitly part of the training data. They do well with software because we've talked so much about software on the internet until this point and that's part of the training data, but pit them against problems people don't talk so much about, and if the labs creating and training these models didn't consider those problems, then the model will pretty much suck at it.
I guess eventually they will literally cover every single task the model could ever come across, at least some variant/permutation of it, but until then every benchmark/evaluation will just uncover "did the labs consider this and who considered it most important before/during training?" basically.
> That’s the problem, these tests can’t tell you how good a model is anymore because it’s trivial for labs to optimise for exactly these tests by the next release.
But they don't prove the claim. Are the models amazing at recreating Minecraft, but the second you swap the word Minecraft out with another game or a custom game, it shits the bed?
That's not what I see. My feed is full of people using Astra to recreate all sorts of games from Diablo to some random idea they came up with, in ridiculously polished detail like animations that would have taken me weeks of iteration in gpt-5.6-sol but it was a single shot by Astra.
> ridiculously polished detail
Can you point me to one of these? The only Astra game I tried was the incredibly janky Mario Kart clone from OpenAI where you could fly by spamming jump
I was impressed by this 3D version of Zork.
Video: https://bsky.app/profile/emollick.bsky.social/post/3muqcwa4j...
Demo: https://zork-underground-empire.netlify.app
Code: https://github.com/emollick/zork-underground-empire
Prompts in a screenshot here: https://bsky.app/profile/emollick.bsky.social/post/3muqdyefc... - I extracted them as text here: https://gist.github.com/simonw/dab1843ef17edeaeffd18b0acbba2...
The scale of this is impressive. I think if you showed me a single puzzle / dungeon though it's something we've seen on par from previous models.
First impressions of the code quality also feel like it needs some serious refactoring, but then again I'm not a game dev:
https://github.com/emollick/zork-underground-empire/blob/cod...
https://github.com/emollick/zork-underground-empire/blob/cod...
Overall impression though yeah it's crazy AI can do this.
that is actually REALLY cool, wow
Impressed like that's cool ai is doing it? Or just impressed in general like its high quality?
I think it's a genuinely good implementation of the concept. It looks good and plays reasonably well and captures what it needs to capture from the original.
(Update: OK the sword fighting isn't good. As I got deeper into the game my positive first impressions wore off.)
I haven't tried any of these demos, but I'm not surprised they stop impressing once you go deep.
What got be mostly impressed were the demos of Astra doing computer use. At my job I do some RPA and can appreciate how challenging it can be. Yet they make it look extremely easy to operate a tool like Blender at super human speeds.
X.com is full of unverified posts, but here's a few:
- https://x.com/MengTo/status/2096213835460084184?s=20
- https://x.com/birdabo/status/2096101381925114327?s=20
Seems really impressive at first glance, but there's no guarantee that they are real
You are falling for selection bias.
For every person using Astra to create a game from some random idea and sharing the impressive result, there are an unknown number who have tried the same thing and gave up in frustration.
I call bullshit
>GPT-6-Astra-Max
"Generate a SVG of a PlayStation 4 controller as nicely done as you can"
Time taken: 11m 54s
No image was used
I was expecting 10s not 12m; if it takes 12m to do it, I would rather try to do it myself. And what it means "No image was used", wasn't it trained on bunch of images including probably console controllers images?
The benchmark I want to see people adopt is:
Build a flowsheet based steady state chemical process simulator, then use it to simulate and optimize a full scale oil refinery.
1) Building a solver engine that works at this scale is not a trivial problem, and the successful ones rely more on heuristics than some categorically different solution approach.
2) Defining the engineering equations relevant to this task is relies on understanding what level of fidelity is required to answer the questions people ask of steady state process models.
3) Knowing the thermophysical properties of chemicals and crude oils is possible from the open literature, but the information is diffuse and different correlations are applicable in different situations.
4) Creating a GUI which converts a flowsheet into matrix math is non-trivial, although a sequential modular approach is a bit easier.
5) Defining large scale models in such a way that they solve robustly is as much art as science. For example, completely closed recycle loops like refrigeration systems are a nightmare for solvers, so it is often better to define them in an open-loop way.
6) Optimization involves knowing the relevant commodity prices, but more importantly how to define the constraints on the model so it doesn't just say to produce infinite gasoline.
7) Troubleshooting the inevitable convergence failures is also as much art as science. There are a large number of diagnostic techniques, but fundamentally you need to be able to relate what is happening during the solver iterations with the intent of your model because more often than not the problem is that you've asserted something impossible, redundant, or irrelevant.
I used to think that way about SVGBench, after all labs can just train on the test set, right? It turns out the task was highly generalizable. Try designing a logo and you quickly see the gap between models visually. Even though there is still a gap between "shiny demo SVG" and actual real-world use.
Same thing happened with MineBench, basically SVGBench+3D, until that got "saturated".
Remember spinning hexagon bench[1]? Or the AI World Clocks[2]? Yeah, that used to be hard for frontier models.
Creating games is the next iteration that still has some signal left. Assets + Game logic + UI + Sound, it let's you assess a model's "taste" very quickly.
What else is left, once all these benchmarks get saturated?
[1] https://x.com/flavioAd/status/1885449107436679394
[2] https://news.ycombinator.com/item?id=45930151
> What else is left, once all these benchmarks get saturated?
I was thinking this thing quite recently and wish to write a blog post about it but IMO, I believe that the next AI benchmarks would be profitable businessmaking. There are/were already some trials done (The famous WSJ/Anthropic one[0]) and some shop iirc in SF which is doing this and recent example of trying to start a lemonade stand with AI[1]
In my opinion, it can't really get more meta than that plus on the more humour side, it could help these companies make some dollars as they desperately need it.
I always wonder if these AI models can self-autonomously generate profit, why would these AI companies try to let you generate the main profit while you just pay them one time or a very small amount in tokens. Why should these AI giants allow a person to be a middle man in the first place in some sense and instead not just run these agents autonomously.
Maybe it could be because these AI agents might not be the best in such a meta task (money making/business) but as such, it does feel to me that it might be a good benchmark in the first place.
[0]: We Let AI Run a Vending Machine. It Lost All the Money. | WSJ: https://www.youtube.com/watch?v=SpPhm7S9vsQ
[1]: Everything AI Does When You Ask It to Start a Lemonade Stand: https://www.youtube.com/watch?v=6Ide5pRLR8Y
I would be surprised if we're not there yet. If Jane Street can generate $500 million per MW of compute right now I'm sure it's already possible.
We might still be figuring out how to benchmark these models by the time next gen comes.
Very low quality content. Unsurprisingly pangram 100%.
I just write this way lol, hard to be "pangram 100%" when I'm getting criticized for typos in another corner
I'm not sure myself, there are a lot of comma splices and other such grammar mistakes. Maybe just edited with AI?
On some occasions, as I have come to understand Hackernews, its not really about the content but sometimes just about the Title or literally just the sentiment behind it.
Maybe we are all being shown recreation of Minecraft or other video games as a benchmark and maybe there is a discussion to be had of the statement of (is it a benchmark or not). I, for one, just appreciate that we are all here who could talk about this topic that I was just thinking about and can have a good discussion about.
In some sense, its speaking some thought that everyone (many) might be thinking internally out loud.
Also on a related topic, I have a question that I wish to ask, there are many text files that I have which are me just writing extremely crude thoughts. I once tried to let AI re-write it in a blog format and I found it to somewhat express what I was saying, perhaps saving me hours.
I still don't upload those blog posts because I don't wish to write AI generated blog posts even though I might have spent more than hour writing those original thoughts. To me, its because I don't exactly know how to disclose things. Some of the original thoughts for example might be quite PII/sensitive. Should I share the exact prompts or the original thoughts and what are the correct hygienic ettiquetes towards this nuanced topic?
It would also take me more than hours to edit the posts and I might still not be confident in my ability of trying to create that blog post and might end up never finishing that public blog post in the first place due to procastination.
I also don't just want to sometimes write AI generated blog posts because its a slippery slope. Without proper disclosure, the other person might not even know if another human on another side of planet even gave a thought about it or not. For all intents they might think that nobody really wrote it but only an AI and just "write a blog post about X"
I also would dislike being the person on that end and I also dislike reading AI generated posts. It produces visceral reaction of hatred and closing the tab when I read AI generated slop.
I ironically wanted to create a blog post about this nuance but I couldn't capture it and I then had AI write that blog post but I didn't post it. I have had some very nuanced meta experiences about this exact topic to be honest and I would love to hear some thoughts & opinions about it.
I have the thoughts but because of all of this, in some sense, a lack of means to share. I would wish to hear other peoples thoughts about those thoughts that I had and if other people resonate with it or not but alas, some of them are just text files within my computer.
"When a measure becomes a target, it ceases to be a good measure."
Goodhart's law strikes again.
https://en.wikipedia.org/wiki/Goodhart%27s_law
However, what is meaningful is whether something is able to create usefully adjacent output, like "let's make Minecraft, but with marching cubes, subdivision surfaces, and global illumination... and behaviorally accurate pandas..." (or something like that).
I have an 11 year old, and most of his game ideas are adjacent to other games he's played. He can make those now, or, at least, enough that he can see where it works and where it doesn't.
Compared to a few years ago, that's pretty cool.
>"When a measure becomes a target, it ceases to be a good measure."
Pelicans?
I heard Gemini was fast so I tested the new one, asked it to clone a popular online game. It took 4 minutes, and worked perfectly first try.
I might need to sit down.
I use Minecraft (and Warcraft and an arcade flying simulator) as a silly but directionally correct indication of the models' capabilitites.
Compare Astra[0] with GPT 5.4[1] which was OpenAI's state of the art just six months ago.
(all tests on more models with code and prompts available here: https://senko.net/vibecode-bench )
Yes, it's not a scientific benchmark but it's a good heuristic.
For a better eval, create a one-page prompt / mini spec related to whatever you're using the LLMs for, and see how well a particular one works for what's important to you.
0: https://senko.net/vibecode-bench/2026/rts-gpt-6-astra.html
1: https://senko.net/vibecode-bench/2026/rts-gpt-5.4.html
I really like these tests for what its worth and I see them on youtube sometimes. I would like to ask a few things though
TLDR: Basically focusing on recreating pay to win (mobile or otherwise) games and recreating them non pay to win perhaps instead of focusing on recreation of minecraft for benchmarks could have a genuinely meaningful impact, and making these games portable as well could be another interesting idea. [so it can be played on any operating system/device so using web or if native then for (Android/IOS/Linux/Windows/MacOS) using game engines like (Preferably godot)/Unity/UE.]
Could the test focus more on pay to win games with unique dynamics.
For example: I literally wanted to create a clash royale recreation because clash royale is a highly pay to win game.
The game is unique enough to have memories but is pay to win enough that it ragebaits me as to what its current situation is, its so pay to win now. A recreation would have genuine effect whereas yet another minecraft clone wouldn't.
I have some fond memories of the game and my brother and I used to play it (my brother moreso than me). Also clash of clans and clash royale famously prevented windows users. So I remember downloading bluestacks to play it on laptop but it required 2GB of ram and back then we only had 1GB. (Ironic that we might come back to that time)
Another question that I have for you which I have been genuinely curious is: who is footing the bill for these benchmarks and youtube videos. What are the economics surrounding it?
I imagine the bill to run quite hot sometimes and I find running these benchmarks to be quite unaffordable personally.
I also wish to ask if you have any theories as to why not people on Youtube share their videos. I found this [0] Minecraft clone by Fable 5.1 extremely good yet they haven't shared the source. I am unsure as to what exact reasons might be behind most youtube videos on recreation with AI to not share the actual code. I don't find much rationale in not sharing AI generated code of a recreation of a game especially if one is making a video about it. So thanks for once for actually sharing the output code as well as I surprisingly found it to be a bit rare!
[0]: https://youtu.be/I0do_vbnMBI?t=361
But they're not recreating Minecraft. They're recreating one of the most common tutorials on the internet that shows a simple voxel overworld and nothing else. That's not Minecraft.
Only Minecraft is Minecraft. Is that the bar for creating Minecraft?
Important distinction, but I would debate the "most common tutorials on the internet" claim.
I think you meant “pedantic distinction”. Everyone reading that comment understands that recreating Minecraft is a very common tutorial for game development, not literally one of the most common tutorials on the internet in general.
We are on HN, I thought that goes without saying.
It’s a tiring YouTube formula:
“Model X CHANGES EVERYTHING!” (surprised Oculus Rift face without the headset)
Some one-shot prompts for quick impact.
I get it with the incentive structure — honestly not trying to shit on any creators (hate the game, not the player) but seeing it every day is just getting tiring.
Though I somewhat think some benchmarks are silly like the article says. I saw someone on YouTube recently take a picture of a building across the street from them (seemed like it was in NYC), and asked GPT 6 Astra to make it in blender. It did a surprisingly good job in 30 minutes. So though these benchmarks don't seem to mean much you could always add a touch of randomness to them like the person in the YouTube video did, but the problem with that is how would you compare the benchmarks in any clear way if they aren't even consistent? Regardless it appears LLMs are getting this good at the general task and not just at the particular instances of said task.
I don't remember seeing playstation controller SVGs before, I doubt they benchmaxxed that
Definitely not.
It has to be Doom or Crysis, aren't they the ones people usually ask if it can run?
I partially owe my career to benchmarking AI with Minecraft, so, I'm going to disagree with the author on this one. Games are a great way to test a model, it's not that deep.
I cant select text nor click links on this page with firefox.
Who cares? Restart your browser maybe
Running Firefox here, no problems whatsoever even with UBO and Privacy Badger disabled (the twitter embeds get blocked at the DNS level, but I doubt those are the problem).
Works on phone but not on my linux desktop.
Mac here. There's nothing all that interesting going on with the JS on that page, so I would suspect a bug with Firefox and/or your desktop environment.
Codex found the root cause:
> The site’s graph incorrectly uses 32 texture bindings, exceeding its WebGPU device’s limit of 16. > Firefox 153 mishandles that failed rendering operation and leaves the compositor waiting
So a firefox + website bug
With an updated firefox, the website works except the mindmap on the top right
> Uncaptured WebGPU error: Too many bindings of type SampledTextures in Stage ShaderStages(FRAGMENT), limit is 16, count was 32. Check the limit `max_sampled_textures_per_shader_stage` passed to `Adapter::request_device`
I’ve used Astra for the past day and a half. My layperson’s review is that it is impressive at computer use and 3D reasoning, and fails in similar ways to 5.6 Sol at similar rates when it comes to coding. I have no idea how it scored so high on SWE benchmarks because so far it has been very “mid” as the kids say.
Most of my and my peers PortCos run their own eval and benchmark sets, simply because they know what they need best.
The reality is, capabilities have largely converged across foundation models over the last 18 months, and much of the value add is coming from the harness layer itself now.
This has been the operating assumption for me and my peers, and has largely played out that way.
That said, this has always been an issue with benchmarking since the very beginning. DB Benchmarks, compute benchmarks, and others that were external facing were always inherently a content and product marketing tool. The actual internal benchmarking used to model, understand, and enhance your product was always a closely held secret.
Most of these conversations are happening, but largely in person and not on HN.
> capabilities have largely converged across foundation models over the last 18 months
For reference, in March '25 the models du jour were Sonnet 3.7, gpt o4 and gemini 2.5 pro. GPT5 was in august '25.
It's been a while since we've heard the old "models have stagnated". Oh well.
It's not "models have stagnated" but "models released at the same time are on the same level". Improvements are still real but the relative gaps between OpenAI, Anthropic, Meta, Grok, Gemini and open models are closer than ever. That doesn't mean progress is slowing down, it's just more widely distributed.
Ah, I see. I misunderstood then. The thing about "gains come from the harness" made me think about it in that way.
This, and depending on the workflow and usecase, you don't necessarily need the latest and greatest with the right kind of harness engineering.
Like everything in engineering, it's about tradeoffs and what works best for your specific problem.
> The reality is, capabilities have largely converged across foundation models over the last 18 months, and much of the value add is coming from the harness layer itself now.
Can you please elaborate on what are your thoughts on open weights models (GLM 5.3, Kimi K3, deepseek etc.)
and if the value add is coming from the harness layer itself, then thoughts on open source harnesses (there are so many harnesses but to name a few: opencode, pi [omp as well], maki, codex is OSS as well, fx.sh) and you can always combine them with skills (Obra/superpowers, matt pocock skills plus using these skills and others to create some other custom skills tailored to your use case as well)
And what about the combination of both now with this cheap open weights models + open source harnesses and other things to compete over the closed garden ecosystems?
How does that comparison follow in reality
Could you in theory use these methods to save on the massively expensive $$$ token spending on Anthropic/OAI?
(Personal anecdote but I have GLM 5.3 + maki [sometimes omp/opencode but mostly maki] and its good enough for most use cases out there that I have and I dont know of too many use cases outside of say recreation of games for examples maybe that I would prefer complete SOTA models. I would also love to know where you believe that SOTA models absolutely do still make the difference discounting the benefits provided by the harness.)
It'd be more plausible if the current SOTA models weren't so good at generating SVG with all subjects instead of just pelican.
We're witnessing the mission getting fucking accomplished [0].
[0]: https://xkcd.com/810/
@notch Hey Markus, can I get a refund on my alpha distro of minecraft? I think the currency is worth more than it used to be considering how many versions there are now :)