Very very cool! Impressed that it can be done with only fine tuning
On a very similar theme, I recently built a “lossy git”, which stores your code by training a neural network, then it generates the code back to you on checkout
The web demo runs entirely on the browser, both the version control and the neural network training/inference. You can commit and checkout both images or code - you can also paste/“upload” your own images, no server requests (everything clears from the page on reload)
Just to point out your site text appears too small on mobile (no mobile view) and it doesn't have a direct link to the repo. If either is by choice then that's fine of course.
That's very cool project.
I read this blog post a few days ago about one way to look at memory encoding in models https://alexwang.ai/posts/associative-memory/ that you might find relevant
> Commit code, it trains. Checkout, it reconstructs from memory.
I see a "Commit" button, but I don't see any checkout, I guess that is what happens automatically? Although that panel says "Reconstruction", not "Checkout"... You might want to do a pass after the AI created the initial version, and make it "humanly consistent" or similar, as right now it seems a bit incoherent.
> Run the JS code before and after a commit
How am I suppose to "run the JS code", manually writing it in my local JS console? Or is this just "Click the green button in the UI" but expressed in the way only a LLM would write it?
To share a joke. Whereas by your logic then George Carlin was not a comedian, but a policy wonk who just happened to use the stage and mic to solicit feedback.
Actually, to be fairer to GP - their _git with vibed storage_ project is not a joke, it’s *art*. And the world needs more art.
Right, a joke that is better because it's inconsistent and contains instructions to do stuff the UI doesn't facilitate? I guess I might be overly strict, but I like my tech jokes to make sense, at least internally consistent, and at least seemingly made by humans.
> And the world needs more art.
The world needs more human art, not just "any art created by any manner", that's just slop.
>I used Qwen2.5-Coder-0.5B as the starting point. I needed a small model to get close to realtime performance. Being a coding model, it had also already seen plenty of CLI usage.
Interesting. I wanted to ask, how that model compares with the fine-tuned version, but I just saw the author provided benchmarks. Interesting that fine-tuning improved the memory so significantly, wouldn't have expected that.
Though, on second thought, I had an experience where I was able to, as a human, fine-tune my memory beyond what I previously thought was possible!
Reminds me of one of my favorite jokes.
Excuse me, can you tell me how to get to Carnegie Hall?
One of the first very cool demos I've seen of ChatGPT was in a post where the author prompted AI "pretend to be a terminal". Then went on to execute commands like `cd .fbi`, `ls files`, `cat files/confidential.txt`. Very fun demo. Looking forward to exploring similar idea with this terminal
I think this was right at the beginning, in the GPT-3.5 era still.
I always wanted to put it behind an open SSH server as a "fake" honeypot of sorts, see what attackers and script kiddies do with it. But I guess by now someone's tried that already..
i agree absolutely with you. and this kind of reinforce my idea that, like this predicted terminal could not be considered a real terminal, predicted intelligence could not be considered real intelligence
Thats an very interesting way to think about this, never thought to invert it that way.
I allways found it interesting to look at this issue in the way the imitation game was modeled. For me I concluded, that as long as we can replicate the intution we associate with intelligence, there is not much room left to debate about "real intelligence".
The chinese room experiment I find in that sense a valuable extension, cuz the turing model more or less is a very conversational "voided of action" kind of test. The room model adds the act of translating without having the knowledge of translation.
But your take invalidates my stances here clearly. There is a possible scenario, in which we can replicate the intuition for operating a real terminal but still keeping the act of the terminal itself a "voided" sort of action by itself, cuz well, its not a real terminal, reading real files and so on...
Thank you for this, I will have to chew on that :)
Like with many things, it depends entirely on how you define "real terminal"; that is, what a "real terminal" must do before it satisfies your definition. If it's "takes input and deterministically executes its instructions, producing output along with side effects," a well-trained model with argmax sampling could well meet the bar. if given the appropriate external interfaces.
I remember a couple of these from a few years ago. Long gone now of course but a fun demo of AI hallucinating the internet endlessly, generated pages from URLs that had more URLs in them to navigate to.
I remember doing something like this in the ancient era of GPT 3 or 4. I just told it that it was a terminal responding to commands and it was surprisingly consistent.
I wonder how much a small model could do if it had a tool calling API for a filesystem, an embedded busybox, and a single output token for 'emit what busybox would do here'. Then presumably the model parameters could learn everything that diverges from what busybox can handle.
That seems as likely as anything, but that just moves the problem to counting, doesn't it? The model gets the duration, but I was under the impression LLMs struggled with the arithmetic that would make that useful.
Idk about Qwen 2.5 Coder that this is based on, but modern models don't struggle with arithmetic. I see Qwen 3.8 do long-hand math in thinking traces (ie. it outputs 17+35+12=64 in the thinking trace) so I would imagine this is something they train into the model specifically. Sometimes it creates and runs a simple python script to check its work.
It's still strange to me that an ability to perform arithmetic (and not just memorize a giant table of results) could emerge from training on language.
In principle a large enough neural network can provably approximate any continuous function. Since arithmetic can be represented with a concrete grammar, it's not surprising to me that an LLM could contain an abstract representation if trained correctly.
I regretfully suspect this actually is the future of UI. You just need smart local models running fast enough to keep the framerate high, but that’s inevitable in the long term, isn’t it?
We had decades to work on making software more performant, but instead most users' space-age 2nm chips spend most of their time running Javascript because it was faster to make software that way.
I think it's hard to understand how much better your experience could be if all your software was 10x faster and using 0.1x the resources, since you can't test both experiences side by side
It's like riding a bike with a tiny bit of resistance in the bearings. You will get used to it, and will probably feel like your bike is fine. But if you replace the hub and remove that friction, you will wonder how you ever tolerated riding a bike that way.
Even assuming that the hardware and models catch up, wouldn't there be an issue with the model not being completely deterministic? You don't really want any sort of randomness in your terminal commands.
Why would you do that instead of having the model do the task you're using the UI for? It could generate UI widgets on-the-fly as needed to take richer inputs.
It might be fine for one-off applications, but it's incredibly inefficient for something that has to be done over and over.
Why would you use a GPU and gigabytes of VRAM running a non-deterministic process for something a program can do on the CPU in milliseconds with kilobytes of memory?
The logic here is a bit different. Consumer software is slow because it's the user paying for the compute to run those bloated Electron apps. Unless we're all running local inference in the future, it would be the vendor paying for the compute to run the AI-based UI.
Of course they would pass that along to the consumer, but nobody is going to pay token-based pricing for something that used to be free and ad-supported, or a cheap monthly subscription.
There's a good reason why the UI for Excel, Photoshop and Word has hardly changed and remains consistent for years and that has worked for billions of people.
Many HN readers have forgotten the Windows 8 UI disaster already.
Thank you for linking that. I distinctly remember this moment as an important point in time, played around with emulating a terminal for hours, couldn’t believe how realistic it was, it even remembered when I changed a file earlier etc. Another point was when people on twitter with early access were like “oh my god it can write code too! I asked it to design a website and it just works!”, programming wasn’t even what LLMs were about, and now it’s one of the main use cases. Another revelation was when it could kinda play chess. A single model that can one shot a huge array of different problems, still mind blown how well all of this works to this day.
I tried it and is not working remotely as close as the demo gif. It keeps telling me that "test1" folder that I want to create already exists (not true), then letting me do "mkdier test2" (sic) but then the folder is not there when ls.
I mean all its expected, this has no state, follow no rules and is making up things from the current context, but is not as "magical" as I thought when watching the demo.
The base model already have a good grasp of language, so that man pages look like coherent sentences. It also knows python and similar language well, which means that things like 1+5 or print(["hi" for _ in range(5)]) work.
I did try to train from scratch, but it was much weaker (also added a lot of aux losses).
Terminal world models have many applications, but empirical observation of successful pretraining or fine-tuning can be significant for encapsulating from rather expensive Bash traces, such as ones obtained from systems with scarce accelerators, etc.
No, the model is simulating the entire TUI, not just terminal commands. It also actually is that slow (because it's generating a diff for the entire screen each time you make a keypress).
Very very cool! Impressed that it can be done with only fine tuning
On a very similar theme, I recently built a “lossy git”, which stores your code by training a neural network, then it generates the code back to you on checkout
* web demo: https://nicobrenner.github.io/lossy-git/demo.html
* cli version: https://nicobrenner.github.io/lossy-git/
The web demo runs entirely on the browser, both the version control and the neural network training/inference. You can commit and checkout both images or code - you can also paste/“upload” your own images, no server requests (everything clears from the page on reload)
That is so silly, I love it.
Just to point out your site text appears too small on mobile (no mobile view) and it doesn't have a direct link to the repo. If either is by choice then that's fine of course.
That's hilarious. I bet if I committed my code into that repo, it would check out with fewer bugs.
That's very cool project. I read this blog post a few days ago about one way to look at memory encoding in models https://alexwang.ai/posts/associative-memory/ that you might find relevant
Something like this might work well for managing long context for LLMs!
> Commit code, it trains. Checkout, it reconstructs from memory.
I see a "Commit" button, but I don't see any checkout, I guess that is what happens automatically? Although that panel says "Reconstruction", not "Checkout"... You might want to do a pass after the AI created the initial version, and make it "humanly consistent" or similar, as right now it seems a bit incoherent.
> Run the JS code before and after a commit
How am I suppose to "run the JS code", manually writing it in my local JS console? Or is this just "Click the green button in the UI" but expressed in the way only a LLM would write it?
Relax
Sorry for giving feedback to someone who clearly is looking for feedback, why post it otherwise?
To share a joke. Whereas by your logic then George Carlin was not a comedian, but a policy wonk who just happened to use the stage and mic to solicit feedback.
Actually, to be fairer to GP - their _git with vibed storage_ project is not a joke, it’s *art*. And the world needs more art.
Right, a joke that is better because it's inconsistent and contains instructions to do stuff the UI doesn't facilitate? I guess I might be overly strict, but I like my tech jokes to make sense, at least internally consistent, and at least seemingly made by humans.
> And the world needs more art.
The world needs more human art, not just "any art created by any manner", that's just slop.
> Is it useful? Not really.
So interestingly enough Anthropic has been using models of this kind for their alignment evaluations: https://alignment.anthropic.com/2026/reward-seeker/#:~:text=...
>I used Qwen2.5-Coder-0.5B as the starting point. I needed a small model to get close to realtime performance. Being a coding model, it had also already seen plenty of CLI usage.
Interesting. I wanted to ask, how that model compares with the fine-tuned version, but I just saw the author provided benchmarks. Interesting that fine-tuning improved the memory so significantly, wouldn't have expected that.
Though, on second thought, I had an experience where I was able to, as a human, fine-tune my memory beyond what I previously thought was possible!
Reminds me of one of my favorite jokes.
Excuse me, can you tell me how to get to Carnegie Hall?
—Practice, man... Practice...
> … I had an experience where I was able to, as a human, fine-tune my memory beyond what I previously thought was possible!
Please elaborate.
Yes please. I'm curious
I haven’t run extensive tests yet, but larger models also get better at memory.
One of the first very cool demos I've seen of ChatGPT was in a post where the author prompted AI "pretend to be a terminal". Then went on to execute commands like `cd .fbi`, `ls files`, `cat files/confidential.txt`. Very fun demo. Looking forward to exploring similar idea with this terminal
I think this was right at the beginning, in the GPT-3.5 era still.
I always wanted to put it behind an open SSH server as a "fake" honeypot of sorts, see what attackers and script kiddies do with it. But I guess by now someone's tried that already..
This is super cool! If you publish the weights I would love to bake it into Rune to run a screensaver hallucinating a terminal.
Holy cow that’s amazing. I’d bet future apps will be built this way.
It's very tricky to make it hallucinate fake commands. I've been trying `dogsay` (instead of `cowsay`) but no luck. Any tips?
> I had a blast building this. Is it useful? Not really.
Maybe instead of simulating a terminal you could use it to predict what you'd type next?
So simulate _you_, not the terminal.
I think this project is a very good way to explore the question:
>What's the difference between a model that predicts intelligent output, and intelligence itself?
It doesn't directly attack the question, but gives a neatly constrained thought experiment which you can see running.
Isn't that just the turing test? I suppose you have you answer now that models can easily pass it.
i agree absolutely with you. and this kind of reinforce my idea that, like this predicted terminal could not be considered a real terminal, predicted intelligence could not be considered real intelligence
Thats an very interesting way to think about this, never thought to invert it that way.
I allways found it interesting to look at this issue in the way the imitation game was modeled. For me I concluded, that as long as we can replicate the intution we associate with intelligence, there is not much room left to debate about "real intelligence".
The chinese room experiment I find in that sense a valuable extension, cuz the turing model more or less is a very conversational "voided of action" kind of test. The room model adds the act of translating without having the knowledge of translation.
But your take invalidates my stances here clearly. There is a possible scenario, in which we can replicate the intuition for operating a real terminal but still keeping the act of the terminal itself a "voided" sort of action by itself, cuz well, its not a real terminal, reading real files and so on...
Thank you for this, I will have to chew on that :)
Like with many things, it depends entirely on how you define "real terminal"; that is, what a "real terminal" must do before it satisfies your definition. If it's "takes input and deterministically executes its instructions, producing output along with side effects," a well-trained model with argmax sampling could well meet the bar. if given the appropriate external interfaces.
This is just "because this predicted terminal is a bad terminal, all predicted terminals must necessarily be bad terminals".
Very cool. Fun project. I feel like there’s a wide field to be explored once you understand fine tuning good existing models
I was searching the blog but couldn't find details on the fine-tuning process. What was the hardware and software used? Any learnings there?
The next step is training a model to pretend to be the internet.
(You could run lynx in that terminal to train/test it without images)
I remember a couple of these from a few years ago. Long gone now of course but a fun demo of AI hallucinating the internet endlessly, generated pages from URLs that had more URLs in them to navigate to.
I guess, with enough latency and/or memory, everything can be a filesystem
It would be interesting to expose this on the internet. I wonder what the scrapers etc… would do with it.
this is quite similar to amazon-science Cyber-Zero ?
https://github.com/amazon-science/Cyber-Zero
the paper is quite interesting - https://arxiv.org/html/2508.00910v2
There was a whole fake gui OS with similar concept right
VibeOS - Fully Hallucinated Operating System
https://www.youtube.com/watch?v=z3pV6FHvcgM
Was the video also hallucinated? Doesn't seem to exist.
Edit: Maybe this was it—https://youtube.com/watch?v=7NfyZhV1dKM
> This video isn't available anymore
athropic had an experiment that did exactly this. we need more experimentation in this space.
Questions for folks familiar with fine-tuning: what are the best resources to learn fine tuning?
I remember doing something like this in the ancient era of GPT 3 or 4. I just told it that it was a terminal responding to commands and it was surprisingly consistent.
Very cool to do this as a Terminal!
I did it for web apps last year, basically a web server that feeds all requests into an LLM.
https://www.ohad.com/blog/voidware/
Why did it return -1 to 10 - 4
I wonder how much a small model could do if it had a tool calling API for a filesystem, an embedded busybox, and a single output token for 'emit what busybox would do here'. Then presumably the model parameters could learn everything that diverges from what busybox can handle.
I think the toolcalling should be system prompts then the agent is the userspace
That's a whole other kettle of trout. Maybe still worth a go mind.
On thin that would be useful is an entirely new permission system for files to allow for users and agents acting for users.
Reminds me of this project from Microsoft. A Quake II “simulation” model where the whole application was just a model.
https://www.microsoft.com/en-us/research/articles/whamm-real...
These are called world models, which is why the project is called WorldTUI. For another example, take a look at Google's recent Genie models. https://deepmind.google/blog/genie-2-a-large-scale-foundatio...
Wait, how did it `sleep 3`? I thought LLMs didn't have a concept of time
It’s still rendering frames constantly, no? So it just figures the next X frames are all doing nothing?
Normal only keypresses invoke the next ”frame”. There is a sleep token that the runner could use to indicate that time is moving.
from the post:
Time is an input too. If nothing is pressed, the model gets a WAIT token with a duration instead of a key, which is how snake keeps moving on its own.
That seems as likely as anything, but that just moves the problem to counting, doesn't it? The model gets the duration, but I was under the impression LLMs struggled with the arithmetic that would make that useful.
Idk about Qwen 2.5 Coder that this is based on, but modern models don't struggle with arithmetic. I see Qwen 3.8 do long-hand math in thinking traces (ie. it outputs 17+35+12=64 in the thinking trace) so I would imagine this is something they train into the model specifically. Sometimes it creates and runs a simple python script to check its work.
It's still strange to me that an ability to perform arithmetic (and not just memorize a giant table of results) could emerge from training on language.
In principle a large enough neural network can provably approximate any continuous function. Since arithmetic can be represented with a concrete grammar, it's not surprising to me that an LLM could contain an abstract representation if trained correctly.
wonder if the PHP on this hallucinated terminal is patched already.
Perhaps it has a hallucinated PHP that never had those bugs.
or it's kind of a schroedinger's PHP. as long as you don't debug it, it simultaneously includes every security hole ever, and none of them.
I regretfully suspect this actually is the future of UI. You just need smart local models running fast enough to keep the framerate high, but that’s inevitable in the long term, isn’t it?
We had decades to work on making software more performant, but instead most users' space-age 2nm chips spend most of their time running Javascript because it was faster to make software that way.
even my ten year old laptop browses the web fine ^_^
If I had to complain about any site/app it'd be datadog... each tab uses like 1gb in firefox.
I think it's hard to understand how much better your experience could be if all your software was 10x faster and using 0.1x the resources, since you can't test both experiences side by side
It's like riding a bike with a tiny bit of resistance in the bearings. You will get used to it, and will probably feel like your bike is fine. But if you replace the hub and remove that friction, you will wonder how you ever tolerated riding a bike that way.
Even assuming that the hardware and models catch up, wouldn't there be an issue with the model not being completely deterministic? You don't really want any sort of randomness in your terminal commands.
> You don't really want any sort of randomness in your terminal commands.
Maybe I do, if by "randomness" you also mean the ability to handle unexpected usecases. Imagine running something like:
$ ps "which processes are slowing my PC down??"
Why would you do that instead of having the model do the task you're using the UI for? It could generate UI widgets on-the-fly as needed to take richer inputs.
It might be fine for one-off applications, but it's incredibly inefficient for something that has to be done over and over.
Why would you use a GPU and gigabytes of VRAM running a non-deterministic process for something a program can do on the CPU in milliseconds with kilobytes of memory?
Probably for the same reasons most of our desktop applications have become slow Chromium instances consuming multiple GBs of RAM, each:
It’s cheaper to hire developers, and quicker to ship software. The user experience be damned.
The logic here is a bit different. Consumer software is slow because it's the user paying for the compute to run those bloated Electron apps. Unless we're all running local inference in the future, it would be the vendor paying for the compute to run the AI-based UI.
Of course they would pass that along to the consumer, but nobody is going to pay token-based pricing for something that used to be free and ad-supported, or a cheap monthly subscription.
This demo is already using a local model. Which is why I made the Electron comparison.
I don't really disagree, but... why do we use Javascript for so many UIs? Badly designed web pages with ads can easily use gigabytes of RAM.
Edit: oh, I saw you wrote a related comment after I posted this. So you already know that direct LLMs for UIs are all but inevitable...
What if we're really just living in a world model? How big of a GPU is required to run the universe I wonder?
This makes absolutely no sense what so ever.
There's a good reason why the UI for Excel, Photoshop and Word has hardly changed and remains consistent for years and that has worked for billions of people.
Many HN readers have forgotten the Windows 8 UI disaster already.
Wow! This this reminds me of when we tried doing this with gpt 3 and.. it worked!
The tmux thing had me confused for a moment until i realized it was emulating tmux. Super fun. Is the model window width aware?
The first part of the context is the window width:
``` BOS SCREEN_START 32 96 ```
I trained it on three different resolutions, but it should kind of work with in between ones too.
good old days https://news.ycombinator.com/item?id=33847479
Thank you for linking that. I distinctly remember this moment as an important point in time, played around with emulating a terminal for hours, couldn’t believe how realistic it was, it even remembered when I changed a file earlier etc. Another point was when people on twitter with early access were like “oh my god it can write code too! I asked it to design a website and it just works!”, programming wasn’t even what LLMs were about, and now it’s one of the main use cases. Another revelation was when it could kinda play chess. A single model that can one shot a huge array of different problems, still mind blown how well all of this works to this day.
I tried it and is not working remotely as close as the demo gif. It keeps telling me that "test1" folder that I want to create already exists (not true), then letting me do "mkdier test2" (sic) but then the folder is not there when ls. I mean all its expected, this has no state, follow no rules and is making up things from the current context, but is not as "magical" as I thought when watching the demo.
Verry Verrry Cool and neat!
Haha nice!
I'm a noob in this, could you enlighten me: what role does the base model play which you started with?
Thank you.
The base model already have a good grasp of language, so that man pages look like coherent sentences. It also knows python and similar language well, which means that things like 1+5 or print(["hi" for _ in range(5)]) work.
I did try to train from scratch, but it was much weaker (also added a lot of aux losses).
Just gonna leave this here: https://en.wikipedia.org/wiki/Aggressive_mimicry
Where is the point of this project if i have terminal ?
Sometimes it's just fun to do something absurd because you can. This is one of the reasons software is exciting.
Terminal world models have many applications, but empirical observation of successful pretraining or fine-tuning can be significant for encapsulating from rather expensive Bash traces, such as ones obtained from systems with scarce accelerators, etc.
because curiosity is the most fun of virtues!
Was thinking the same thing. I just run my usual terminal and get to work.
Because it was there
This will allow you to delete your terminal app and, eventually, all other binaries in your system!
BUG: Running in Firefox makes every keypress a slow motion show. It decodes every keypress instead of sending a command with Enter key
No, the model is simulating the entire TUI, not just terminal commands. It also actually is that slow (because it's generating a diff for the entire screen each time you make a keypress).
Exactly, I wonder if I'm not hitting the right WebGPU path for Firefox?