Neat! Perhaps I'll finally get around to doing something with the 51515 Lego set I have laying around collecting dust.
> I did this in order to get agentic LLMs capable of designing buildable, physical things!
I've been experimenting with FreeCAD + freecad-mcp + Claude recently and have been similarly surprised how well this works. I've produced three useful parts from Claude's designs: a 3d printed case for an OLED display + some inputs + an ESP32, and a milled aluminium hotend mount adapter for a 3d printer, and a printed pen + pressure sensor mount to use the printer as a plotter. I just provided part/model numbers plus a couple of caliper measurements Claude wasn't able to find online and went through two or three cad revisions before making the parts. All three worked on the first hardware revision.
The biggest thing I've found it seems to struggle with is understanding how the parts will be assembled. The pen mount in particular was tricky to put together, having a part which needed to fit through a slightly too small space to get to its spot. Although it's hard to blame the AI for that when I didn't notice it either I suppose.
I've been experimenting with FreeCAD and freecad-mcp too, even managed for Claude to create a new LDraw benchmark for it.
However, given the amount of parts some LDraw models have, FreeCAD performance would not be enough, something to do with its solver trying to work out too many constraints at once.
The thing about how to assemble the parts, yes, that drove me crazy for quite a while.
A way I found that helps is to provide the agents information about contacts, e.g. what parts of a brick are actually connectors.
With that in place, it can create tools to automatically check if two parts are connected "the right way" ;-)
I just added another comment under my original as you were replying- I'll try not to spam you with too many questions, this is just so cool though!
Do you find the models generally manage to avoid making errors when placing parts (e.g. placing parts which intersect, don't connect properly, etc) or do they rely on validation and correction cycles?
When giving them enough examples, like e.g. houses, they tend to do well in that category, but not perfect.
My guess is that what happens is, given that most LEGO parts have a very limited, discrete set of legal positions and rotations, expressed as integer numbers, the agent finds patterns between different models, like e.g. a brick close to the center of a model usually has very similar (integer) coordinates and rotation components to another central brick from another model.
However, it still messes up very often, with things like e.g. +/- signs that flip parts due to rotations becoming -90deg when they should be +90deg; then, validation and correction cycles, running tools that verify the presence/absence of collisions, really helps in finishing the job.
On that note- have you considered adding constraints for part availability? It would be useful to be able to create designs using only parts which are on hand.
I recall seeing a project a while ago which could take a photo of a pile of lego as input and produce models that could be made with those parts. Although I think in that case the models were sorta just random arrangements.
Not the same but adjacent hobby, I’ve had good success with feeding 3mf files to Claude (usually just fable in desktop app) that I found on maker world to prompt for modifications to the model and it works damn well. Stuff like changing a functional model from press fit only to using inset m bolts and nuts. Way better than doing it by hand in a cad app
A very similar idea crossed my mind several times, but with a twist: altering LEGO CAD parts, but keeping the connectors the same.
Main idea was, if LEGO parts don't cover the needs for an agent to build a model, let it use the best it can find, and manufacture others if needed, but keeping same compatibility with the rest of the catalog.
And for that, instead of generating new parts from scratch, maybe mutating pre-existing parts into new shapes it needs, but leaving connectors as they were, or even adding new connectors but still compatible.
Nice job. I actually read a paper on Arxiv last year about this exact thing - using LLMs to drive model building through LDraw that might be worth reading [1].
They had some interesting experiments like giving the LLM a fixed set of parts as a constraint and asked it to "build" tools out of those available lego.
I think they were using Opus 4.5 so I'm sure with even more powerful models it'll be even better.
I'm taking a look at this paper right now, didn't came across it before.
I think it's really good, and also an approach I tried.
In my case, it was working pretty well in terms of geometry and correctness, but not in terms of creativity: too many constraints and an agentic LLM would end up building very similar buildings all the time. Too few ones, and it would drift into unbuildable models.
In the end, I think the current iterative process gives it a good balance: let it try things first and be creative, parts will be placed incorrectly at the beginning, but it will fix their placements, colours and so on bit by bit, until the end result is (hopefully!) correct.
And regarding Opus, according to my experience, I'd say definitely models quality is way better when produced by Opus 5 and even better by Opus 5.5
We introduce BrickGPT, the first approach for generating physically stable toy brick models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of brick designs, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that BrickGPT produces stable, diverse, and aesthetically pleasing brick designs that align closely with the input text prompts. We also develop a text-based brick texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We also release our new dataset, StableText2Brick, containing over 47,000 brick structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models.
I remember coming across this research paper, and I thought it was really impressive. I clearly remember the bricks guitar, built by robots =)
Currently, Nova doesn't handle any physical constraints regarding stability, just collision.
I think physics handling is a very desirable feature, maybe an agent using a physics engine directly or maybe via MCP that agent could get feedback about a model's stability, fix problems, back to more feedback, etc.
This would be similar to the current way finds and iteratively corrects defects by rendering images from the model, different perspectives, and directly examining them.
Amazing! I’ve got a similar project going except focusing on voxel to LEGO to ensure the pieces build correctly. Would love to collab if you’re interested! https://github.com/jjohnson5253/brickbuilderai
My personal advise.
Get your self a lot of small containers like from IKEA or just throw away take away food boxes.
Start with small batches. Do large parts first.
Sort by category, part and then at last if needed by color.
I do this regularly on my boys collection and we are able to even build 1000+ pieces sets again and again. We have like 15 Ikea samla boxes (multiple for bricks, 3 for plates, vehicle parts, large parts, transparent parts, multiple boxes to sortminifigs and equipment and so on) and for really tiny parts extra little boxes like 1x1 plates. One for white black and grey and another for colors.
I don't see an other way of stay sane even on small collections with just 10k+ pieces...
Thats pretty cool, i wonder how far it goes with mechanisms where the parts actually have to move correctly, rather than just being assembled correctly
That's still WIP, started already with Technic assemblies, not mechanisms yet, in order to first produce the required scaffolding to attach things like e.g. cogs, transmissions, etc. to them.
Getting mechanisms to work is one of my goals, though =)
Physics modeling is something Nova is currently lacking, and I think very interesting, given that agents currently don't verify physics, stability... only placements.
Well, that greatly depends on the thinking required to produce a single LDraw model, not so much regarding how much of a "thinker" a model is, but how much thinking it must do to get a model right.
Technic models, for example, are usually expensive, given that initial placements are usually further from the right position than non-Technic models, the "bricks" ones.
A (wild!) estimate: Astra, building a Technic mechanism, around 5$...
This is one of the reasons for creating this project: making a tooling that can be used by low-end agents to iteratively build, while high-end agents, like e.g. Astra or Opus, would do the planning, vision models the inspection... etc.
How well does this work with local models like https://github.com/receptron/laya (open source Jev) and vision capable models like Qwen 3.8 Flash Next, or even Qwen 3.8 27B on xhigh thinking?
Well, I'd really like to see that, robots building models generated by agents.
Something that comes to my mind very often is... adding more parts to the LDraw parts catalog, the ones used to build models, but these extra ones would be real-sized blocks, like the ones used to build actual buildings ;-)
New on HN, first time posting, and (I guess) my post on Show HN was rejected...
Anyway, I hope I'm not bothering anyone by posting this outside Show HN, or posting almost the same thing twice.
Past year, around December, I started experimenting with making ChatGPT and Claude generate source code in LDraw language.
This LDraw is literally an "assembly" language, a low-level programming language that describes how to assemble LEGO pieces together into models, one placement instruction at a time.
When executed by specific tools, like e.g. LDView, LeoCAD, Studio... these instructions become LEGO CAD models, that can be interacted with, modified, etc.
Or, in other words: one LDraw source file in .mpd or .ldr format is equivalent to one LEGO CAD model.
So, the idea I had was: if I manage for maybe ChatGPT or Claude to generate high-quality LDraw source files... then, they would actually be generating high-quality LEGO CAD models, right?
Then, after months of iterations and trying one thing after the other... it worked!!!
Long story short: using GPT-6 Astra and Opus 5.5, I've managed to create a python toolset, instructions, and docs for agents in general. Now, these can be used by them to generate LDraw models.
I've packed it all as a dockerized web app for others to try and experiment, with several providers (and agents) to choose from: OpenAI, Claude and OpenRouter.
I'd really appreciate feedback and comments, let's see where this goes =)
This project is perfect for a Show HN! I moved (most of) your text to the top and added "Show HN" to the title. I'll respond to this bit here:
> and (I guess) my post on Show HN was rejected... Anyway, I hope I'm not bothering anyone by posting this outside Show HN, or posting almost the same thing twice.
Unfortunately we had to put some barriers in place (https://news.ycombinator.com/showlim) because we got flooded, and still are, with Show HNs from new accounts which lack the spark of curiosity that is most important here. The hope is people will take the time to read the site, participate in threads, get a feel for the culture, and eventually post a Show HN that is a better fit.
None of that applies to your project, and if we had a better way to screen these things, we'd write software that let posts like yours through the first time. In any case I'm glad I randomly saw this one, and welcome to HN!
(Normally I'd email, but maybe it's fun for others to be in on the conversation.)
I think this post would be even better if there were more visual examples to look at. Would it be possible to link to more pictures, or even a demo video, showing different cool and/or weird things produced by the tool?
(In terms of video, you don't need anything fancy - what works best is raw and direct, with zero production values.)
Also, another thing that isn't clear to me is whether you've built any of these structures physically. If so, it would be particularly cool to see that in action! If not, it would be interesting to know what it would take to do so.
That's a gallery of by-products and results from different executions.
Videos, I'm afraid I don't have any, these CAD models can be better inspected by opening them on e.g. LDView, that to record LDView while playing with them.
Those come from opening ldraw-nova on VS Code, asking different coding agents (Codex, Claude Code, etc.) to read the instructions.md at the repo, and then to build something.
Also, from different attempts by different agents in order to learn "how" to do something, like e.g. place an axle correctly.
Those are just leftovers, usually images like those show embedded in web app chats on LDraw Nova.
If you are curious about the internals of an .mpd model, the "what was the agent picturing on its mind", open the .mpd file that got your attention on a text editor, and read the first line under the ones starting with "0 FILE".
And regarding building those, well... actually, I didn't even try, the amount of parts required alone for the larger models is actually in the order of the thousands, my guess is it would be difficult to get all the required parts for just one of those models.
I'm currently considering 3D-printing a small one, however, in order to see how it looks IRL.
Neat! Perhaps I'll finally get around to doing something with the 51515 Lego set I have laying around collecting dust.
> I did this in order to get agentic LLMs capable of designing buildable, physical things!
I've been experimenting with FreeCAD + freecad-mcp + Claude recently and have been similarly surprised how well this works. I've produced three useful parts from Claude's designs: a 3d printed case for an OLED display + some inputs + an ESP32, and a milled aluminium hotend mount adapter for a 3d printer, and a printed pen + pressure sensor mount to use the printer as a plotter. I just provided part/model numbers plus a couple of caliper measurements Claude wasn't able to find online and went through two or three cad revisions before making the parts. All three worked on the first hardware revision.
The biggest thing I've found it seems to struggle with is understanding how the parts will be assembled. The pen mount in particular was tricky to put together, having a part which needed to fit through a slightly too small space to get to its spot. Although it's hard to blame the AI for that when I didn't notice it either I suppose.
Thanks!
I've been experimenting with FreeCAD and freecad-mcp too, even managed for Claude to create a new LDraw benchmark for it.
However, given the amount of parts some LDraw models have, FreeCAD performance would not be enough, something to do with its solver trying to work out too many constraints at once.
The thing about how to assemble the parts, yes, that drove me crazy for quite a while.
A way I found that helps is to provide the agents information about contacts, e.g. what parts of a brick are actually connectors.
With that in place, it can create tools to automatically check if two parts are connected "the right way" ;-)
I got LEGO connectors info from here: https://www.melkert.net/LDCad/tech/shadowLib
I just added another comment under my original as you were replying- I'll try not to spam you with too many questions, this is just so cool though!
Do you find the models generally manage to avoid making errors when placing parts (e.g. placing parts which intersect, don't connect properly, etc) or do they rely on validation and correction cycles?
A mix of both, I think.
When giving them enough examples, like e.g. houses, they tend to do well in that category, but not perfect.
My guess is that what happens is, given that most LEGO parts have a very limited, discrete set of legal positions and rotations, expressed as integer numbers, the agent finds patterns between different models, like e.g. a brick close to the center of a model usually has very similar (integer) coordinates and rotation components to another central brick from another model.
However, it still messes up very often, with things like e.g. +/- signs that flip parts due to rotations becoming -90deg when they should be +90deg; then, validation and correction cycles, running tools that verify the presence/absence of collisions, really helps in finishing the job.
> I've been experimenting with FreeCAD + freecad-mcp + Claude recently and have been similarly surprised how well this works
Yup. I’ve been getting some pretty decent 3d models directly on stl format + the slicer file with all the right configs for my printer
It’s cool to create a simulation, run some visualizations on web directly from Claude desktop, and then choosing a model to print and build
I also added some genetic algorithms to make it fun iterating on different shapes/mechanisms
I really enjoy how much creativity these tools are unlocking by enabling bridging so many elements together
> the 51515 Lego set
On that note- have you considered adding constraints for part availability? It would be useful to be able to create designs using only parts which are on hand.
I recall seeing a project a while ago which could take a photo of a pile of lego as input and produce models that could be made with those parts. Although I think in that case the models were sorta just random arrangements.
Not the same but adjacent hobby, I’ve had good success with feeding 3mf files to Claude (usually just fable in desktop app) that I found on maker world to prompt for modifications to the model and it works damn well. Stuff like changing a functional model from press fit only to using inset m bolts and nuts. Way better than doing it by hand in a cad app
A very similar idea crossed my mind several times, but with a twist: altering LEGO CAD parts, but keeping the connectors the same.
Main idea was, if LEGO parts don't cover the needs for an agent to build a model, let it use the best it can find, and manufacture others if needed, but keeping same compatibility with the rest of the catalog.
And for that, instead of generating new parts from scratch, maybe mutating pre-existing parts into new shapes it needs, but leaving connectors as they were, or even adding new connectors but still compatible.
Nice job. I actually read a paper on Arxiv last year about this exact thing - using LLMs to drive model building through LDraw that might be worth reading [1].
They had some interesting experiments like giving the LLM a fixed set of parts as a constraint and asked it to "build" tools out of those available lego.
I think they were using Opus 4.5 so I'm sure with even more powerful models it'll be even better.
[1] - https://arxiv.org/pdf/2512.15743
Hi there, thanks!
I'm taking a look at this paper right now, didn't came across it before.
I think it's really good, and also an approach I tried.
In my case, it was working pretty well in terms of geometry and correctness, but not in terms of creativity: too many constraints and an agentic LLM would end up building very similar buildings all the time. Too few ones, and it would drift into unbuildable models.
In the end, I think the current iterative process gives it a good balance: let it try things first and be creative, parts will be placed incorrectly at the beginning, but it will fix their placements, colours and so on bit by bit, until the end result is (hopefully!) correct.
And regarding Opus, according to my experience, I'd say definitely models quality is way better when produced by Opus 5 and even better by Opus 5.5
You might also be interested in this research at CMU:
https://avalovelace1.github.io/BrickGPT/
We introduce BrickGPT, the first approach for generating physically stable toy brick models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of brick designs, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that BrickGPT produces stable, diverse, and aesthetically pleasing brick designs that align closely with the input text prompts. We also develop a text-based brick texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We also release our new dataset, StableText2Brick, containing over 47,000 brick structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models.
I remember coming across this research paper, and I thought it was really impressive. I clearly remember the bricks guitar, built by robots =)
Currently, Nova doesn't handle any physical constraints regarding stability, just collision.
I think physics handling is a very desirable feature, maybe an agent using a physics engine directly or maybe via MCP that agent could get feedback about a model's stability, fix problems, back to more feedback, etc.
This would be similar to the current way finds and iteratively corrects defects by rendering images from the model, different perspectives, and directly examining them.
Amazing! I’ve got a similar project going except focusing on voxel to LEGO to ensure the pieces build correctly. Would love to collab if you’re interested! https://github.com/jjohnson5253/brickbuilderai
This is dope, the voxel approach makes so much sense
Up next: a tool that converts a photograph of your dumped bin of old Lego into a catalogue of all the Lego in your collection..
https://brickit.app/ :)
Just wow!
ugh, want this so bad.
I bought so many sets, and I want to rebuild them, but they're a whole big mess. I want to sort them out.
Makes finding pieces for new builds much easier too
My personal advise. Get your self a lot of small containers like from IKEA or just throw away take away food boxes. Start with small batches. Do large parts first. Sort by category, part and then at last if needed by color. I do this regularly on my boys collection and we are able to even build 1000+ pieces sets again and again. We have like 15 Ikea samla boxes (multiple for bricks, 3 for plates, vehicle parts, large parts, transparent parts, multiple boxes to sortminifigs and equipment and so on) and for really tiny parts extra little boxes like 1x1 plates. One for white black and grey and another for colors. I don't see an other way of stay sane even on small collections with just 10k+ pieces...
That's really fun ! thanks for sharing !
Can it create functional mechanisms using LEGO Technic parts?
It seems to be able to! https://github.com/anteloc/ldraw-nova/blob/master/examples/m...
Thats pretty cool, i wonder how far it goes with mechanisms where the parts actually have to move correctly, rather than just being assembled correctly
That's unfinished work, however... didn't get good results with Technic models yet, except for very specific cases, like e.g. the Atlas Crane.
That's still WIP, started already with Technic assemblies, not mechanisms yet, in order to first produce the required scaffolding to attach things like e.g. cogs, transmissions, etc. to them.
Getting mechanisms to work is one of my goals, though =)
Thanks for sharing - I'm actually working with my kid on something similar but with physics modeling for First Lego League. Cool to see your approach!
Physics modeling is something Nova is currently lacking, and I think very interesting, given that agents currently don't verify physics, stability... only placements.
Any estimates for token costs to generate these models? Curious about costs for Astra, Opus and Jev for some of the examples you have in the video.
Well, that greatly depends on the thinking required to produce a single LDraw model, not so much regarding how much of a "thinker" a model is, but how much thinking it must do to get a model right.
Technic models, for example, are usually expensive, given that initial placements are usually further from the right position than non-Technic models, the "bricks" ones.
A (wild!) estimate: Astra, building a Technic mechanism, around 5$...
This is one of the reasons for creating this project: making a tooling that can be used by low-end agents to iteratively build, while high-end agents, like e.g. Astra or Opus, would do the planning, vision models the inspection... etc.
Well, who knows :)
How well does this work with local models like https://github.com/receptron/laya (open source Jev) and vision capable models like Qwen 3.8 Flash Next, or even Qwen 3.8 27B on xhigh thinking?
Didn't try this with laya, but just tested it with Qwen 3.8 27B via OpenRouter.
Tried twice, one starting directly in Agent mode, and another one starting first in Plan mode and then Agent.
Conclusions: very slow in general, starting in Plan mode gives way better and faster results, but still with defects.
See: - Notes: https://github.com/anteloc/ldraw-nova-docker/blob/experiment... - Result: https://github.com/anteloc/ldraw-nova-docker/blob/experiment...
Oh this cool, thank you, I'll have a look at this in a few weeks (I've a backlog of cool things I want to look at :D )
This is a really interesting idea. I've never seen something unique like this in a long time, thank you for sharing!
You're welcome ;-)
Fun idea. Congrats on building and releasing!
That said, LLMs encroaching into the pure-imagination world of Lego gives me a bit of the icks...
I'm interested in this for robotics. Probably need physics to do that well.
The next step after this is to have AI and bots build a real house. Great work!
Well, I'd really like to see that, robots building models generated by agents.
Something that comes to my mind very often is... adding more parts to the LDraw parts catalog, the ones used to build models, but these extra ones would be real-sized blocks, like the ones used to build actual buildings ;-)
Actually very cool and interesting stuff!
That's so cool!
Really interesting
This is great
Yeah, take the fun out of everything.
Wtf is wrong with you all people.
People find different things 'fun'. Presumably the author really enjoyed making this (and look at all the positive comments).
Who are you to decide what other people enjoy?
Hi there :-)
New on HN, first time posting, and (I guess) my post on Show HN was rejected...
Anyway, I hope I'm not bothering anyone by posting this outside Show HN, or posting almost the same thing twice.
Past year, around December, I started experimenting with making ChatGPT and Claude generate source code in LDraw language.
This LDraw is literally an "assembly" language, a low-level programming language that describes how to assemble LEGO pieces together into models, one placement instruction at a time.
When executed by specific tools, like e.g. LDView, LeoCAD, Studio... these instructions become LEGO CAD models, that can be interacted with, modified, etc.
Or, in other words: one LDraw source file in .mpd or .ldr format is equivalent to one LEGO CAD model.
So, the idea I had was: if I manage for maybe ChatGPT or Claude to generate high-quality LDraw source files... then, they would actually be generating high-quality LEGO CAD models, right?
Then, after months of iterations and trying one thing after the other... it worked!!!
Long story short: using GPT-6 Astra and Opus 5.5, I've managed to create a python toolset, instructions, and docs for agents in general. Now, these can be used by them to generate LDraw models.
I've packed it all as a dockerized web app for others to try and experiment, with several providers (and agents) to choose from: OpenAI, Claude and OpenRouter.
I'd really appreciate feedback and comments, let's see where this goes =)
This project is perfect for a Show HN! I moved (most of) your text to the top and added "Show HN" to the title. I'll respond to this bit here:
> and (I guess) my post on Show HN was rejected... Anyway, I hope I'm not bothering anyone by posting this outside Show HN, or posting almost the same thing twice.
Unfortunately we had to put some barriers in place (https://news.ycombinator.com/showlim) because we got flooded, and still are, with Show HNs from new accounts which lack the spark of curiosity that is most important here. The hope is people will take the time to read the site, participate in threads, get a feel for the culture, and eventually post a Show HN that is a better fit.
None of that applies to your project, and if we had a better way to screen these things, we'd write software that let posts like yours through the first time. In any case I'm glad I randomly saw this one, and welcome to HN!
Thanks dang!
I really appreciate you helping me with this, and totally get the bit about getting flooded.
Anyway, I hope HN community will like this, and maybe contribute, have some crazy ideas about how and where to move forward =)
(Normally I'd email, but maybe it's fun for others to be in on the conversation.)
I think this post would be even better if there were more visual examples to look at. Would it be possible to link to more pictures, or even a demo video, showing different cool and/or weird things produced by the tool?
(In terms of video, you don't need anything fancy - what works best is raw and direct, with zero production values.)
Also, another thing that isn't clear to me is whether you've built any of these structures physically. If so, it would be particularly cool to see that in action! If not, it would be interesting to know what it would take to do so.
Well, here it is:
https://anteloc.github.io/index-samples.html
That's a gallery of by-products and results from different executions.
Videos, I'm afraid I don't have any, these CAD models can be better inspected by opening them on e.g. LDView, that to record LDView while playing with them.
Those come from opening ldraw-nova on VS Code, asking different coding agents (Codex, Claude Code, etc.) to read the instructions.md at the repo, and then to build something.
Also, from different attempts by different agents in order to learn "how" to do something, like e.g. place an axle correctly.
Those are just leftovers, usually images like those show embedded in web app chats on LDraw Nova.
If you are curious about the internals of an .mpd model, the "what was the agent picturing on its mind", open the .mpd file that got your attention on a text editor, and read the first line under the ones starting with "0 FILE".
Great - I've added https://anteloc.github.io/index-samples.html to the toptext, along with the last sentence of your comment here. I hope that's ok!
Sure! Thanks dang ;)
And regarding building those, well... actually, I didn't even try, the amount of parts required alone for the larger models is actually in the order of the thousands, my guess is it would be difficult to get all the required parts for just one of those models.
I'm currently considering 3D-printing a small one, however, in order to see how it looks IRL.
[flagged]
[dead]
[dead]