Not making any claims about a specific model's capabilities, but all these "can interpret model weights" posts brings to mind the line in Shakespeare's Henry IV
GLENDOWE: I can call spirits from the vasty deep.
HOTSPUR: Why, so can I, or so can any man, but will they come when you do call for them?
I can interpret weights too, but will that interpretation be correct?
These are famous games tested here though, and I tried to feed one problem to DeepSeek 4.1 Flash 1017 and it comes back at 66% confidence it is a Pac-Man (incorrect). That smells to me the problem setup heavily hint towards these 2D games. Might be resolution related to be honest.
From what I saw in the paper, no they did not. They tested that the specific change introduced increased performance on a single task (the one mentioned in the prompt). They did not study the impact of that change on overall model performance across a variety of benchmarks. But maybe I missed something.
> I can interpret weights too, but will that interpretation be correct?
Chances are much higher than a model can figure out what the patterns are, decipher them, and retain that knowledge in a meaningful way it can reason about. Me and you might take years or months to do all of that on our own, with a "map" of the weight we might be able to discuss it, but not as effortlessly as an LLM.
Honestly, I had told a few different colleagues that reading a model is probably not that exclusive to frontier models. It's probably yet another reason OpenAI / Anthropic don't release any big open weight models, but thankfully Google and Microsoft release reasonably sized models. Imagine a new GPT open weight release, it would get dissected drastically.
Thinking about it, it makes a lot of sense that LLMs could easily be trained to interpret weights. The attention mechanism of an LLM is very well suited to analyze a set of weights (in text form), since the attention to each weight in the text via key/query can sort of emulate the connection structure of the network that the weights represent.
Further than that, aren't "weights" just another sequence of tokens? We see them as vectors, pass to the API or as strings into a chatbox interface. Is it so surprising that the high-dimensionality linear algebra machine crunches them and gives us back useful vectors that can be mapped to text?
Isn't that what this whole technology paradigm is about? How is it surprising?
Granted, it is cool. But I see so much of "look at this cool thing I did with LLMs" and honestly... we need more structured text, more explicit documentation, more ergonomic and intuitive CRUD applications.
The amount of "look at cool things" is growing way faster than before and they are all sort of low effort now because the "agentic" LLM-VM looping apparatus can do a lot of heavy lifting and run with our prompts.
And I mean, if one has jargon, knowledge, and intention, they can also leverage that into great structured text, ergonomic and intuitive CRUD apps etc, all of which add great value to our mundane pursuits of applied research, software products and services and so on.
Honestly, I begin to think the problem is people dazzling themselves instead of doing the housekeeping that is the bulk of life. Technology choice is entirely secondary.
> Further than that, aren't "weights" just another sequence of tokens?
No, not at all. I recommend you learn some of the basic linear algebra about how these things work before engaging in wild and completely nonsensical speculation.
I understand linear algebra. It seems like he probably does, too.
Are you sure you have read and understood his point?
It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory, and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.
I think that's the point the person is making. Do you disagree? If so, why?
Weights are not sequences of tokens in any sense. This is pure nonsense. We don't "see them" [presumably, the weights] as vectors (they are matrices or tensors).
There is nothing obvious or expected about weights being easily explainable or being enough to explain, because the weights are static: you need to look at the inputs, the activations, and the weights all together, you can never understand a model with the weights alone, unless it is a trivial model. The person isn't making any coherent points at all.
> It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory,
Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here.
> and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.
> Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here.
You seem to really like to tell people that they don't know what they're talking about more than you seem to be parsing what they're actually saying.
I remember when I used to look for reasons to pounce on people for not knowing what they were talking about based on a superficial thing or two rather than try to understand what they were saying. I mostly got past it as a teenager, though.
Except you clearly don't understand anything you are talking about, and neither does GP. People like you have seriously degraded the value of HN because you pontificate ignorantly and incoherently about topics. That needs to be called out.
Biblically speaking, spirits from the vasty deep must be demons since they are from below and angels, God and Jesus are from above. So in that sense it’s not very impressive. Demons are something a sane person would rather avoid.
I mean, if you want to inject religion into this, that interpretation that it "must be demons" isn't even supported by the source text.
In 1 Samuel, King Saul visits a medium in Endor in order to speak to the (dead) prophet Samuel. Human dead go to Shoel (in familiar terms, it is similar to the Greek underworld) in the Old Testament. Later on in the New Testament, the idea of a really really bad place is introduced in the form of Tartarus, which is specifically where the rebelling angels were sent. But that is not the same place human dead go.
The idea that everything "not Heaven" must be Hell is an invention, caused mostly by dogma that rose from bad translations of the Greek and Hebrew in the middle ages (for example, King James - itself mostly correcting minor errors of the older Bishop's Bible - refers to all of it as Hell even though the original text most inarguably refers to different places different beings go).
>I can interpret weights too, but will that interpretation be correct?
In TFA case, given that they were able to guess what the weights did (which the researchers knew in advance and could cross-check) and to even tweak and improve them, the answer is yes.
Qwen-3.8-27B - the best thing that happened in AI so far.
This thing can accomplish the same results as the frontier models do, if you are willing to play a role of an higher level architect or decision maker. Guide it a bit here and there like a tech lead and you get everything done.
The money you pay for frontier models is exactly for outsourcing that high level direction work to agents.
> Qwen-3.8-27B - the best thing that happened in AI so far.
Qwen3.8-Flash_Next should be in the running for that title as well. It's astonishingly good. I use both of them. If I need more memory on the machine for other things I run 27B, If I can get by with much less RAM I run QFN (takes about 102GB on my Strix Halo box with 262,144 context). QFN runs quite a bit faster on my hardware - about 50t/s. 27B with MTP starts out around 22t/s but degrades over time as the context fills up. Both are great models, but I'd give QFN a bit of an edge (at least for the coding projects I've worked on). But both feel on par with Claude Opus 4.6 - 4.8.
Apparently Qwen 4 might land this week, and it uses the same architecture as 3.8-Flash-Next (so various LLM serving tools should be able to handle it already). I'm hoping Qwen 4 includes some smaller models - the last 4B and 9B from Qwen were for 3.5.
Agree, I run QFN on my AI Pro R9700 (32GB) using https://github.com/Niko1221/Strata It's even faster than on a Strix Halo. Typically I get around 90 tokens/sec with a small context, down to 70 tokens/sec when it's near the maximum context usage.
Can you tell a little more about your software setup? Been playing with local AI models but struggle to get anything as easy as ChatGPT or Claude. I have a folder with skills, and I can switch between ChatGPT and Claude, and either will do the same work. I’m struggling to find something that can do that locally
One RTX4090 with 24G gives me about 200k KV cache with Q4_K_M quant for typica coding tasks, via opencode. I use Q6_K with 60k cache if the work doesn't require scanning too many files. Q6_K is really good.
Local models requires investment of about $2500 into hardware, so that you can run models at decent tok/sec. You then need to basically write custom harnesses. All of my stuff runs on a llama.cpp backend on a dedicated inference machine, and a client runs custom code built with OpenAPI library to make inference calls to accomplish things.
The latent space of models, which basically steers the token generation into a particular direction is much coarser with the local models compared to big models. Small models known fundamentals well, but do not have the breadth to realize the particularities of combination of fundamental task given certain criteria.
As such, if you want to replicate big model behaviour, you need to basically do context augumentation. How you do that is up to you for your own needs.
But my question remains: isn't something like pi.dev good enough for this? Or are you using custom harness to describe the scripts/skills/etc that you create yourself?
I find myself having to write a lot more skills for Qwen3.8-27b to effectively use tools it hasn’t seen before, versus virtually no skill-writing needed for Claude anymore. It seems to understand novel tools and when to use them pretty intuitively.
For basic tasks, yea. For more advanced tasks, where the agents running on big llms will do more actions autonomously, the smaller models will sometimes stop execution or go off in wrong directions.
I feel like Alibaba is stealing from the frontier models, and I’m a bit queasy taking advantage of that. If they aren’t distilling and it’s actually true they have superior efficiency as they claim, then why aren’t they way ahead?
I feel like frontier models are stealing the entire internet and the entire whatever any human wrote in the latest centuries, including everything _I_, being an human on the internet, wrote. So I have a hard time being morally concerned about them being distilled.
Not like anthropic or open ai, they sure downloaded torrents of everything to train, crawl the web non stop, eat open source developers work, git repos etc, not pay ANY but they are not distilling other models at least....
You check its work and if it seems to do its job well enough most of the time, you just start to trust it.
Personally Qwen3.8-27b has not proven itself to me for the projects I’ve tried to use it for. It doesn’t seem to have as much encoded knowledge about the native Mac development stack.
This seems to be the inherent trade-off with smaller models. They can be just as "bright"
as their larger contemporaries but they necessarily can't encode as much knowledge.
I think over time we'll realize this is the way forward, you want the smartest model with the least amount of embedded knowledge and you provide the knowledge externally or merge it into the weights for the domain you're interested in. Then you don't have to train a ginormous model every 6 months to pull in new information.
Gemma4:31b is better. Qwen is really good at speed (especially with megakernels), but if its going in a wrong direction, it will go in the wrong direction hard. Gemma can self correct with its thinking.
The only reason Qwen is more popular is because it can run decently fast on MLX, while Gemma needs a discrete GPU with 24gb of ram to run at usable speeds.
Also, theoretically speaking, any small model can be as good as a big model if all they have to do is summarize text. Back before agents were a thing, we had prototype coders running of local inference models from Mistral for ai dev. The workflow was to generate llm summaries of the codebase, and a summary of changes to be made. Then we would grab search results from the web, feed it to an llm to summarize, and then finally combine all of that context into a final inference run which would generate code, then the codebase would be run automatically, any errors would be llm analyzed, and an llm would generate the next change to be made in the above cycle.
Nowdays, we can do that with pi coding agent and Gemma4:31b. But we still have plenty of custom harnesses for things.
The context window left on my 24GB card with Gemma is to low to do any major task though. Maybe small diffs or a single direct task. But Qwen has enough context for agentic tasks.
I don't understand why megakernels haven't become the default choice. The speed advantage is crazy. I made a few changes to the openjet megakernel to run in a dual gpu (older gen, not blackwell) box and doubled my token generation speed on qwen3.8-27B (4-bit though).
I keep debating adding a third matching gpu to see if I can push the speed a bit more and move back to the 8-bit quant, but I'm stuck on the idea of just building a Blackwell gpu system despite the cost factor. Plus a third gpu is going to going to be weird for the tensor split and P2P data syncs and my workstation is old with 3rd gen PCIe etc etc.
Architectures (like Gemma4) have things like sliding window attention that make it pretty impractical to do on a GPU, you would have to do additional things to make it work, and the talent to do that, even with LLM assistance is very small.
> This thing can accomplish the same results as the frontier models do
What frontier models are you using? I'm literally sitting with Qwen 3.8 27B locally and trying my damnest to get it to work at least 80% of the quality Astra et al have, and can't get close to it. It keeps being so verbose, regardless of inference parameters, system prompts, reasoning effort, and keeps going on tangents. I'm running the BF16 weights via vLLM, can't imagine how much worse the experience must be with the quantized weights people run with the average consumer GPU.
Even with tight guiding, as I guess you could say I do more "AI pair-programming" than vibecoding, it just loses the plot to often and waste time, enough that it's clearly not an alternative to today's frontier models.
Unless you are trying to prove some hard math, or writing extremely complex code, for general coding tasks (which represent 90% of code out there) 27B-Q6_K is more than enough. Q4 does it well too, but might need a few more iterations.
For comparison, I also used GLM-5.3, Muse-Spark etc, with no significant improvement for ordinary/BAU coding tasks.
Perhaps I’m missing this, but can Qwen 3.8 alter Qwen 3.8 weights?
There are a lot of known neurons in the older Qwen models that are long known to induce behaviour changes (see evil Qwen by changing one weight).
My question boils down to how much is trained vs being able to actually understand the weights. Presumably a same gen model isn’t trained on any sources dissecting that generation. And how much do longer known conceits like steering vectors, J spaces, LoRAs play into this?
This doesn’t explain the game training cases, which are very impressive to me.
Ok now the idea that models can alter weights is out there, does that mean there will be models in the future that are heavily trained to modify model weights?
A Qwen3.8-27B model can probably do something with a Qwen3.5-0.8B model. I don't think this is so much the version (Qwen 3.8 vs Qwen 3.5) but the model size (27B vs 0.8B). But ya, having research on Qwen 3.5 definitely helps.
I was thinking along these lines a couple of weeks ago[0] - specifically whether we could treat weight manipulation as a sort of programming/compilation task. I expected it to require a lot more training with synthetic datasets. It's surprising that existing models are already somewhat decent at it.
> I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
That sounds a bit like biology; maybe AI will become like that.
Words like "dominance" and "superior" and such in this context are more interesting in how they're employed than in any inherent phenomenon they are intended to describe.
Some poor-ass professor -- fighting tooth and nail for tenure, or at least some form of stability from all-powerful admin staff -- can't keep themselves from trying to figure out who's "Dominant" or "Alpha" looking at a bunch of apes or bugs. When in reality the network isn't directed, but a mesh of arbitrary weights, vectors, and dimensionality.
Perhaps if ancient China had beat us to Enlightenment and Industrial Revolution we would be seeing scientists talking about how respectful mammals are of their ancestors.
Simon, could you try reproducing Experiment 5 from the blog post, if you have a chance? Apparently this person tried [1] with API models and only the latest models can do the impressive 10-20x vs. PPO baseline.
Only asking because I remember injecting bit errors in my local LLM back in 2024 with mixed and sometimes weird results so I definitely knew it was possible; finding out you can do this in a controlled way is frankly mind-blowing.
Why would you want to change weights directly versus running through a new training cycle? For example, I have collected a bunch of data I would like to train weights that I use local inference for. I figured a LoRa would be the way to go, perhaps be more cost effective and generalized too.
Why would you do a whole new expensive, time-consuming training run if you can just tweak some numbers to get the result you want?
I think your use case of incorporating novel data probably calls for training, but I'm sure there are use cases where you can tweak the weights to get what you want (say I wanted to tweak the model to be less sycophantic, or found a way to shrink it by removing weights associated with a language that I don't ever use)
Because text-to-weights is a learnable function. Everything else seems to be. So, why not? Why train weights if you could just prompt for them. If we managed to train image edit models, then why can't we train model-edit models?
If you could figure out exactly what certain weights do it would be way quicker to directly edit them, whereas training is non-deterministic and takes ages.
And the modifications of those weights could be tracked in source control and overlayed on top of the base model? I see, there is a lot of potential in that.
Speed, instant finetunes? Highly optimized iteration cycles? The applications are the same for any reason why you might want local llms. This is crazy exciting.
I wouldn't be surprised at all if 3.8-27B could tweak 3.5-0.8B the same way Astra did, but it wouldn't be a big deal either because that part of the paper has nothing to do with directly tweaking the weights like the first part of the paper, but is instead a fairly straightforward abliteration method on a model that is known for a specific pathological behavior (ending up stuck in a thinking loop).
isn't consciousness just that, a thinking loop with interruptions for sleep until we break beyond repair? and, just maybe, if we managed to figure out the meaning of life, our thinking would satisfyingly stop?
What I want to see someone try and succeed at would be to get the really highly quantized models to loop less. Many people with only 16GB VRAM are running the IQ3_XXS or IQ3_S quants of Qwen3.8-27B. If it could figure out how to improve those quants, that'd actually be pretty substantial. That's all they did for those improved math results is removed the looping. Why not? Please someone. Please.
It's fairly straightforward to write a function that detects looping output -- I wrote one for promptfoo in [1].
For training you would need to hook it up to a LoRA training harness, identify the looping weight direction, and adjust the weights accordingly. I've not (yet) looked into how to do this.
i feel like this could turn out to be a over unity demo where they say the model can improve a model better than it's own. (more energy output than input)
Isn't the network too shallow to be challenging to decode? One can easily produce few layer network to solve specific goal with construction techniques, so not a surprising result in my view.
Semi-related question I was wondering about recently - how is it that Alibaba has a better consumer-friendly open model (for coding anyway) than research-leading companies like Google?
Well Google could open source Gemini 4 Argon but they choose to release the lower tier models open source instead... Its an active choice they all make.
I did exactly the same thing with Qwen 3.8 27B but i also asked it something else: a couple years ago i wrote a simple upscaler[0] when i was learning about neural networks. The network is dead simple: it takes a 25 inputs for a 5x5 grid around a pixel and outputs the 2x2 pixel grid that the input pixel should be replaced for a (width2)x(height2) upscale. I trained it on a bunch of random images (in hindsight i should have trained it on game screenshots but w/e) and it looked quite nice.
So i asked Qwen 3.8 27B two things:
1. What it is this network all about? I only gave it the ann.c code that implemented a simple artificial neural network library in C and the setup code that specified the network's shape but not its purpose.
Qwen 3.8 27B tried a bunch of stuff, taking the layers one by one, proding numbers through it, etc and finally figured out it is some sort of 5x5 to 2x2 filter, most likely working with pixels (i never mentioned pixels or graphics), though it wasn't certain about the exact purpose. It did rule out a bunch of uses like being a downscaler or classifier, from its behavior.
2. After i revealed what it was, i asked it if it can improve it but keep it 100% compatible with the existing program that used the weights file. And sure enough, it did improve it. I gave it the original training C code and dataset. It made a few experiments (and wrote its own training C code because it found mine too slow :-P) and ended up improving the weights. The difference[1] is small, but it basically eliminated some small noise that was always there and made some transitions less "halo-y". Of course for the purpose i wanted it (use it to upscale games in realtime) it isn't that great (too heavy) but it was learning material anyway. Though one bonus thing i noticed is that it upscales my random doodles much better than any Krita filter[2] :-P.
The above isn't Qwen 3.8 27B observing the weights directly and making sense out of them (ok actually it did seem to make some sense of the weights in the "#1" but i didn't dump the entire network to it - though thinking about it, it is only around 16384 floats so it might work). It just used a bunch of scripts it wrote in Python (and later C) but i still found it interesting to watch it go at it.
I hacked together a 4x (via applying it twice) scale once when i was modifying Gamescope to see how it'd look but it was both very slow and i didn't like the result so i only kept the 2x one[0].
Out of curiosity i did apply it twice to the drawing i posted in my previous comment with a 4x bilinear upscale made in Krita[1]. TBH i think a dedicated 4x network would do a better job though even then, there is only so much a network can hallucinate to fill in details :-P.
> It’s a cool paper, but some of the wider commentary I’ve seen around it does not feel justified. I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
The age of source code is definitely not going to end anytime soon if ever. But the future will involve incomprehensible machines editing other models in a "fairly certain" way, but we are going to need a lot of tooling to keep up with the process and not just take the model updates as "trust me bro".
It's a blog post which received feedback from collaborators.
It hasn't gone through the academic paper lifecycle ... the authors themselves refer to it as a post and as preliminary.
I feel like I'm taking crazy pills, but can we be precise when talking about work of this form? It started with referring to arXiv pre-prints as papers, and now we're at blog posts with an academic template as papers.
A paper is a fucking paper: something which has been vetted by the scientific community, with at least some form (often, a few rounds) of anonymous review from people outside your circle of collaborators
This is not there yet, and clearly so due to the various feedback from people (including the two pushback posts on HN) critiquing the lack of discussion about the existence of these techniques in mech interp, etc?
Like you said, in the AI space a pre-print on arXiv is massively more likely than anything that's been through peer-review.
I called it a paper here because it's by people with an academic affiliation and is of comparable quality (if not better than) many of the other "papers" we see.
I also like it when a paper isn't a PDF. PDFs are a bad way of communicating information.
(I will note that the authors themselves didn't call it a paper, their own BibTeX reference calls it a "blog post":)
@misc{takehi2026weightspace,
title = {{Sparks of Weight-Space Programming: LLMs can natively read \& write neural network weights}},
author = {Takehi, Rikiya and Teoh, Jayden and Zhang, Alex and Khattab, Omar},
year = {2026},
month = nov,
howpublished = {\url{https://rikiyat.github.io/blog/weight-space-programming/}},
note = {Blog post}
}
> PDFs are a bad way of communicating information.
They have the strength of one PDF being frozen & (mostly) treated online as immutable - compared to a webpage which might change, or a blog post which might be edited. If a PDF needs to be replaced there'll be a notice.
And the single-file nature of PDFs makes them more likely to sit in an archive not managed by the author (vs a blog) so they tend to live as long as the archive rather than the author's blog.
For something that needs to Be referenced by others those are useful properties - I'd argue they outweigh how good PDF is at communicating information.
But for (1, 2), just to point out, neither of your criteria would pass muster in the scientific communities who are ultimately responsible for vetting this work, right?
The ideal is: we don't accept papers at NeurIPS (or wherever) because of affiliation, and the badge of "quality" is bestowed by peer review, from other scientists.
> is of comparable quality (if not better than)
But the posts that you refer to in your own post point out several ways in which the quality is not good ...
But, in any case, I find it a bizarre choice to call it a paper when the authors themselves do not refer to it as such. We can agree to disagree, but just offering feedback.
Of course, I agree with you in many respects - but I’ve found the academic paper lifecycle to be tremendously helpful in terms of polishing my contributions (admittedly, not in ML - but in PL / systems) and avoiding content which could be construed as “misleading”
You, being of sound mind, tried the technique and drew reasonable conclusions … but just look at the original post for this blog post: there are a large number of responses by people who clearly don’t understand the technique, missed the experimental weaknesses, couldn’t place the post into the lineage of mech interp results that preceded it … and will leave the discussion with the wrong impression.
In your mind, whose fault is that? Is that that the unavoidable cost of a field, moving too rapidly to engage in peer review?
In its ideal form, the paper lifecycle is not a gate, but a sharpening stone.
I think we have a messy version of peer review playing out right now.
I'm not convinced the alternative - delaying publication by 6 months, keeping the new information available only to a much smaller group until then - would have been better.
All of that said, I did just update my own post to include "The paper (OK, the authors classify is as a “blog post”, but it’s paper-shaped) found that frontier models..." inspired by this conversation.
Because nobody actually reviews papers anyway so it is irrelevant. That might have happened when you only had 12 papers a year and they were things like “gravity exists”, but when you get a million papers a day on topics like “the ethnocentric illusion of west-canadian yarn spinning”, nobody has time for that.
Yeah, that's not gonna happen in ML, where a positive result (even in blog form) can be (hopefully) trivially tested and confirmed.
From memory, yolo and roberta and ppo were initially rejected. After a quick google search, pagerank was initially rejected, word2vec got one weak and one strong reject in reviews, LSTM got rejected (90s but still), dropout, knowledge distillation, and many more such cases. All of these are extremely important, foundational and field defining papers that ended up on arxiv or other preprint places.
If that happened to those papers, there's 0 chance today that it'd be a good idea. The field is simply moving too fast, there's too much being published, and the "top venues" are simply swamped. Blog / preprint / carrier pigeon is ok atm, as long as someone can look at it, try it, and hopefully reproduce it.
It literally just did math on some numbers based on a set of rules provided in the prompt in order to sketch a visual of a game. It didn't "interpret model weights" any more than an MP3 player "interprets physics of electromagnetic waves". The LLM built its own decoder on the fly for the games... which is what it's doing when you ask it to write code, calculate math, estimate the structural load bearing of a beam, etc.
Not making any claims about a specific model's capabilities, but all these "can interpret model weights" posts brings to mind the line in Shakespeare's Henry IV
GLENDOWE: I can call spirits from the vasty deep.
HOTSPUR: Why, so can I, or so can any man, but will they come when you do call for them?
I can interpret weights too, but will that interpretation be correct?
They did test for correct interpretation, so what point are you making?
His point is that he has read Shakespeare but not the linked article.
These are famous games tested here though, and I tried to feed one problem to DeepSeek 4.1 Flash 1017 and it comes back at 66% confidence it is a Pac-Man (incorrect). That smells to me the problem setup heavily hint towards these 2D games. Might be resolution related to be honest.
From what I saw in the paper, no they did not. They tested that the specific change introduced increased performance on a single task (the one mentioned in the prompt). They did not study the impact of that change on overall model performance across a variety of benchmarks. But maybe I missed something.
But increasing performance on a single task was the point of the change?
> I can interpret weights too, but will that interpretation be correct?
Chances are much higher than a model can figure out what the patterns are, decipher them, and retain that knowledge in a meaningful way it can reason about. Me and you might take years or months to do all of that on our own, with a "map" of the weight we might be able to discuss it, but not as effortlessly as an LLM.
Honestly, I had told a few different colleagues that reading a model is probably not that exclusive to frontier models. It's probably yet another reason OpenAI / Anthropic don't release any big open weight models, but thankfully Google and Microsoft release reasonably sized models. Imagine a new GPT open weight release, it would get dissected drastically.
Thinking about it, it makes a lot of sense that LLMs could easily be trained to interpret weights. The attention mechanism of an LLM is very well suited to analyze a set of weights (in text form), since the attention to each weight in the text via key/query can sort of emulate the connection structure of the network that the weights represent.
Further than that, aren't "weights" just another sequence of tokens? We see them as vectors, pass to the API or as strings into a chatbox interface. Is it so surprising that the high-dimensionality linear algebra machine crunches them and gives us back useful vectors that can be mapped to text?
Isn't that what this whole technology paradigm is about? How is it surprising?
Granted, it is cool. But I see so much of "look at this cool thing I did with LLMs" and honestly... we need more structured text, more explicit documentation, more ergonomic and intuitive CRUD applications.
The amount of "look at cool things" is growing way faster than before and they are all sort of low effort now because the "agentic" LLM-VM looping apparatus can do a lot of heavy lifting and run with our prompts.
And I mean, if one has jargon, knowledge, and intention, they can also leverage that into great structured text, ergonomic and intuitive CRUD apps etc, all of which add great value to our mundane pursuits of applied research, software products and services and so on.
Honestly, I begin to think the problem is people dazzling themselves instead of doing the housekeeping that is the bulk of life. Technology choice is entirely secondary.
> Further than that, aren't "weights" just another sequence of tokens?
No, not at all. I recommend you learn some of the basic linear algebra about how these things work before engaging in wild and completely nonsensical speculation.
I understand linear algebra. It seems like he probably does, too.
Are you sure you have read and understood his point?
It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory, and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.
I think that's the point the person is making. Do you disagree? If so, why?
Weights are not sequences of tokens in any sense. This is pure nonsense. We don't "see them" [presumably, the weights] as vectors (they are matrices or tensors).
There is nothing obvious or expected about weights being easily explainable or being enough to explain, because the weights are static: you need to look at the inputs, the activations, and the weights all together, you can never understand a model with the weights alone, unless it is a trivial model. The person isn't making any coherent points at all.
There are also deep reasons to expect hard limits in intrepretability in general: https://www.seangoedecke.com/alignment-is-impossible/#why-me....
EDIT:
> It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory,
Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here.
> and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.
This is how it does... all tasks.
A vector is a type of a matrix is a type of a tensor.
> Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here.
You seem to really like to tell people that they don't know what they're talking about more than you seem to be parsing what they're actually saying.
I remember when I used to look for reasons to pounce on people for not knowing what they were talking about based on a superficial thing or two rather than try to understand what they were saying. I mostly got past it as a teenager, though.
Except you clearly don't understand anything you are talking about, and neither does GP. People like you have seriously degraded the value of HN because you pontificate ignorantly and incoherently about topics. That needs to be called out.
If you care about the quality of HN: produce the high quality content you want; don't demean others.
There's not really a point in talking back to abuse. Think what you will.
I will point out that your comment was the first reply ever to someone completely new to HN, though.
I really am not surprised someone is such an arse around this place. Being totally factually right is the most important thing in life always, right?!
And this reminds me of the king in the little prince, who had the power to order the sun to rise or go down. But only once a day at a certain time.
Biblically speaking, spirits from the vasty deep must be demons since they are from below and angels, God and Jesus are from above. So in that sense it’s not very impressive. Demons are something a sane person would rather avoid.
I mean, if you want to inject religion into this, that interpretation that it "must be demons" isn't even supported by the source text.
In 1 Samuel, King Saul visits a medium in Endor in order to speak to the (dead) prophet Samuel. Human dead go to Shoel (in familiar terms, it is similar to the Greek underworld) in the Old Testament. Later on in the New Testament, the idea of a really really bad place is introduced in the form of Tartarus, which is specifically where the rebelling angels were sent. But that is not the same place human dead go.
The idea that everything "not Heaven" must be Hell is an invention, caused mostly by dogma that rose from bad translations of the Greek and Hebrew in the middle ages (for example, King James - itself mostly correcting minor errors of the older Bishop's Bible - refers to all of it as Hell even though the original text most inarguably refers to different places different beings go).
Crazy how much the quote went over your head. Literature written for peasants 400 years ago and you can't grasp the point
Love this reference
>I can interpret weights too, but will that interpretation be correct?
In TFA case, given that they were able to guess what the weights did (which the researchers knew in advance and could cross-check) and to even tweak and improve them, the answer is yes.
interpretation is, as they say, in the eye of the beholder…
But the whole point of the article was that it was correct.
Why do this without testing? Why would you assume that?
Qwen-3.8-27B - the best thing that happened in AI so far.
This thing can accomplish the same results as the frontier models do, if you are willing to play a role of an higher level architect or decision maker. Guide it a bit here and there like a tech lead and you get everything done.
The money you pay for frontier models is exactly for outsourcing that high level direction work to agents.
> Qwen-3.8-27B - the best thing that happened in AI so far.
Qwen3.8-Flash_Next should be in the running for that title as well. It's astonishingly good. I use both of them. If I need more memory on the machine for other things I run 27B, If I can get by with much less RAM I run QFN (takes about 102GB on my Strix Halo box with 262,144 context). QFN runs quite a bit faster on my hardware - about 50t/s. 27B with MTP starts out around 22t/s but degrades over time as the context fills up. Both are great models, but I'd give QFN a bit of an edge (at least for the coding projects I've worked on). But both feel on par with Claude Opus 4.6 - 4.8.
I switched back to glm-5.3-flash from qwen3.8-flash-next.
qwen was faster in t/s, but feels like glm spends less tokens thinking and seems better at finding issues, to the point where it ends up being faster.
Apparently Qwen 4 might land this week, and it uses the same architecture as 3.8-Flash-Next (so various LLM serving tools should be able to handle it already). I'm hoping Qwen 4 includes some smaller models - the last 4B and 9B from Qwen were for 3.5.
Agree, I run QFN on my AI Pro R9700 (32GB) using https://github.com/Niko1221/Strata It's even faster than on a Strix Halo. Typically I get around 90 tokens/sec with a small context, down to 70 tokens/sec when it's near the maximum context usage.
Which quant of QFN are you running on your Strix Halo?
HF is telling me that IQ4-XS is the largest one likely to fit without much contortion.
> Qwen3.8-Flash_Next should be in the running for that title as well.
Unfortunately it still hallucinates, which limits it's usefulness for real world tasks. :(
Every model hallucinates, if not given the right context and tools.
Can you tell a little more about your software setup? Been playing with local AI models but struggle to get anything as easy as ChatGPT or Claude. I have a folder with skills, and I can switch between ChatGPT and Claude, and either will do the same work. I’m struggling to find something that can do that locally
One RTX4090 with 24G gives me about 200k KV cache with Q4_K_M quant for typica coding tasks, via opencode. I use Q6_K with 60k cache if the work doesn't require scanning too many files. Q6_K is really good.
Local models requires investment of about $2500 into hardware, so that you can run models at decent tok/sec. You then need to basically write custom harnesses. All of my stuff runs on a llama.cpp backend on a dedicated inference machine, and a client runs custom code built with OpenAPI library to make inference calls to accomplish things.
> need to basically write custom harnesses
Curious, why/how does that help? Or do you have some niche needs?
The latent space of models, which basically steers the token generation into a particular direction is much coarser with the local models compared to big models. Small models known fundamentals well, but do not have the breadth to realize the particularities of combination of fundamental task given certain criteria.
As such, if you want to replicate big model behaviour, you need to basically do context augumentation. How you do that is up to you for your own needs.
But my question remains: isn't something like pi.dev good enough for this? Or are you using custom harness to describe the scripts/skills/etc that you create yourself?
> You then need to basically write custom harnesses.
Why? Most existing harnesses let you point them at whatever API you want. I'm currently have a great time with Juggler.
I find myself having to write a lot more skills for Qwen3.8-27b to effectively use tools it hasn’t seen before, versus virtually no skill-writing needed for Claude anymore. It seems to understand novel tools and when to use them pretty intuitively.
For basic tasks, yea. For more advanced tasks, where the agents running on big llms will do more actions autonomously, the smaller models will sometimes stop execution or go off in wrong directions.
I feel like Alibaba is stealing from the frontier models, and I’m a bit queasy taking advantage of that. If they aren’t distilling and it’s actually true they have superior efficiency as they claim, then why aren’t they way ahead?
I feel like frontier models are stealing the entire internet and the entire whatever any human wrote in the latest centuries, including everything _I_, being an human on the internet, wrote. So I have a hard time being morally concerned about them being distilled.
That happened to individuals, so it is fine. This crime is happening against a company, that is wrong and bad.
Not like anthropic or open ai, they sure downloaded torrents of everything to train, crawl the web non stop, eat open source developers work, git repos etc, not pay ANY but they are not distilling other models at least....
Don't forget destructively scanning books too.
> Guide it a bit here and there like a tech lead and you get everything done
How do you validate and verify that outcome?
Or does Qwen-3.8-27B do the validation and verification for you as well? If so, how do you ensure that's correct?
You check its work and if it seems to do its job well enough most of the time, you just start to trust it.
Personally Qwen3.8-27b has not proven itself to me for the projects I’ve tried to use it for. It doesn’t seem to have as much encoded knowledge about the native Mac development stack.
This seems to be the inherent trade-off with smaller models. They can be just as "bright" as their larger contemporaries but they necessarily can't encode as much knowledge.
I think over time we'll realize this is the way forward, you want the smartest model with the least amount of embedded knowledge and you provide the knowledge externally or merge it into the weights for the domain you're interested in. Then you don't have to train a ginormous model every 6 months to pull in new information.
I think this is what the n-gram table in Qwen 4 (and 3.8-Flash-Next) is for. We’ll see if it comes down to the 27b variant.
A RAG solution would make time to first token awful.
Gemma4:31b is better. Qwen is really good at speed (especially with megakernels), but if its going in a wrong direction, it will go in the wrong direction hard. Gemma can self correct with its thinking.
The only reason Qwen is more popular is because it can run decently fast on MLX, while Gemma needs a discrete GPU with 24gb of ram to run at usable speeds.
Also, theoretically speaking, any small model can be as good as a big model if all they have to do is summarize text. Back before agents were a thing, we had prototype coders running of local inference models from Mistral for ai dev. The workflow was to generate llm summaries of the codebase, and a summary of changes to be made. Then we would grab search results from the web, feed it to an llm to summarize, and then finally combine all of that context into a final inference run which would generate code, then the codebase would be run automatically, any errors would be llm analyzed, and an llm would generate the next change to be made in the above cycle.
Nowdays, we can do that with pi coding agent and Gemma4:31b. But we still have plenty of custom harnesses for things.
Realistically a 32GB card is what you need to run Gemma4:31b well. Doing it in 24GB leaves almost no room for context / KV cache.
The context window left on my 24GB card with Gemma is to low to do any major task though. Maybe small diffs or a single direct task. But Qwen has enough context for agentic tasks.
...especially with megakernels...
I don't understand why megakernels haven't become the default choice. The speed advantage is crazy. I made a few changes to the openjet megakernel to run in a dual gpu (older gen, not blackwell) box and doubled my token generation speed on qwen3.8-27B (4-bit though).
I keep debating adding a third matching gpu to see if I can push the speed a bit more and move back to the 8-bit quant, but I'm stuck on the idea of just building a Blackwell gpu system despite the cost factor. Plus a third gpu is going to going to be weird for the tensor split and P2P data syncs and my workstation is old with 3rd gen PCIe etc etc.
Architectures (like Gemma4) have things like sliding window attention that make it pretty impractical to do on a GPU, you would have to do additional things to make it work, and the talent to do that, even with LLM assistance is very small.
> This thing can accomplish the same results as the frontier models do
What frontier models are you using? I'm literally sitting with Qwen 3.8 27B locally and trying my damnest to get it to work at least 80% of the quality Astra et al have, and can't get close to it. It keeps being so verbose, regardless of inference parameters, system prompts, reasoning effort, and keeps going on tangents. I'm running the BF16 weights via vLLM, can't imagine how much worse the experience must be with the quantized weights people run with the average consumer GPU.
Even with tight guiding, as I guess you could say I do more "AI pair-programming" than vibecoding, it just loses the plot to often and waste time, enough that it's clearly not an alternative to today's frontier models.
Unless you are trying to prove some hard math, or writing extremely complex code, for general coding tasks (which represent 90% of code out there) 27B-Q6_K is more than enough. Q4 does it well too, but might need a few more iterations.
For comparison, I also used GLM-5.3, Muse-Spark etc, with no significant improvement for ordinary/BAU coding tasks.
Perhaps I’m missing this, but can Qwen 3.8 alter Qwen 3.8 weights?
There are a lot of known neurons in the older Qwen models that are long known to induce behaviour changes (see evil Qwen by changing one weight).
My question boils down to how much is trained vs being able to actually understand the weights. Presumably a same gen model isn’t trained on any sources dissecting that generation. And how much do longer known conceits like steering vectors, J spaces, LoRAs play into this?
This doesn’t explain the game training cases, which are very impressive to me.
Ok now the idea that models can alter weights is out there, does that mean there will be models in the future that are heavily trained to modify model weights?
The future is going to be wild.
I figured when Anthropic published their J space paper they use that to fine tune, this is next level though
A Qwen3.8-27B model can probably do something with a Qwen3.5-0.8B model. I don't think this is so much the version (Qwen 3.8 vs Qwen 3.5) but the model size (27B vs 0.8B). But ya, having research on Qwen 3.5 definitely helps.
I was thinking along these lines a couple of weeks ago[0] - specifically whether we could treat weight manipulation as a sort of programming/compilation task. I expected it to require a lot more training with synthetic datasets. It's surprising that existing models are already somewhat decent at it.
[0] https://x.com/o_lalonde/status/2100389409564643518
Hopefully we get a new benchmark: How well can a model tweak a set of weights to produce pelican on a bike?
I actually build something to help with the whole pelicans on bike problem! https://modelsalt.com
Nice!
> I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
That sounds a bit like biology; maybe AI will become like that.
Superior as mammals are to plants, so may AI become to us mammals.
Words like "dominance" and "superior" and such in this context are more interesting in how they're employed than in any inherent phenomenon they are intended to describe.
Some poor-ass professor -- fighting tooth and nail for tenure, or at least some form of stability from all-powerful admin staff -- can't keep themselves from trying to figure out who's "Dominant" or "Alpha" looking at a bunch of apes or bugs. When in reality the network isn't directed, but a mesh of arbitrary weights, vectors, and dimensionality.
Perhaps if ancient China had beat us to Enlightenment and Industrial Revolution we would be seeing scientists talking about how respectful mammals are of their ancestors.
That's paraphrased from Erewhon, which was written over 150 years ago and was the inspiration for Dune.
Ah, it was in the spirit of quotation. I missed that. My apologies.
Samuel Butler was a character.
Not so sure mammals are "superior" to plants, nor fungi.
Nope squishy is where it's at.
Simon, could you try reproducing Experiment 5 from the blog post, if you have a chance? Apparently this person tried [1] with API models and only the latest models can do the impressive 10-20x vs. PPO baseline.
[1] - https://nitter.netbub.com/AlexKrentsel/status/21173766837246...
All the talk is of the improvement in a capability, but has there been any study on What are these changes are breaking ?
Only asking because I remember injecting bit errors in my local LLM back in 2024 with mixed and sometimes weird results so I definitely knew it was possible; finding out you can do this in a controlled way is frankly mind-blowing.
Why would it not be able to interpret model weights in such a constructed prompt with tons of information and leading questions?
I hope you at least switched off all network access, in case the model cheats and reads HN from yesterday.
Why would you want to change weights directly versus running through a new training cycle? For example, I have collected a bunch of data I would like to train weights that I use local inference for. I figured a LoRa would be the way to go, perhaps be more cost effective and generalized too.
Why would you do a whole new expensive, time-consuming training run if you can just tweak some numbers to get the result you want?
I think your use case of incorporating novel data probably calls for training, but I'm sure there are use cases where you can tweak the weights to get what you want (say I wanted to tweak the model to be less sycophantic, or found a way to shrink it by removing weights associated with a language that I don't ever use)
Because text-to-weights is a learnable function. Everything else seems to be. So, why not? Why train weights if you could just prompt for them. If we managed to train image edit models, then why can't we train model-edit models?
If you could figure out exactly what certain weights do it would be way quicker to directly edit them, whereas training is non-deterministic and takes ages.
And the modifications of those weights could be tracked in source control and overlayed on top of the base model? I see, there is a lot of potential in that.
Because training is more expensive I guess
Something someone else brought up in one of the previous threads was to identify the refusal routes and remove them.
It would take a lot less FLOPS presumably
Speed, instant finetunes? Highly optimized iteration cycles? The applications are the same for any reason why you might want local llms. This is crazy exciting.
I wonder how far this can be taken. Could Qwen 3.8 can interpret and make improvements to Qwen 3.5 0.8B weights, or even its own weights?
I wouldn't be surprised at all if 3.8-27B could tweak 3.5-0.8B the same way Astra did, but it wouldn't be a big deal either because that part of the paper has nothing to do with directly tweaking the weights like the first part of the paper, but is instead a fairly straightforward abliteration method on a model that is known for a specific pathological behavior (ending up stuck in a thinking loop).
isn't consciousness just that, a thinking loop with interruptions for sleep until we break beyond repair? and, just maybe, if we managed to figure out the meaning of life, our thinking would satisfyingly stop?
It can, as long as you can give it benchmarks. What it can't do is optimize general intelligence without any external validation.
You can optimize a model by randomly changing its weights and validating results on your task. A monkey can type a book given enough attempts.
What I want to see someone try and succeed at would be to get the really highly quantized models to loop less. Many people with only 16GB VRAM are running the IQ3_XXS or IQ3_S quants of Qwen3.8-27B. If it could figure out how to improve those quants, that'd actually be pretty substantial. That's all they did for those improved math results is removed the looping. Why not? Please someone. Please.
It's fairly straightforward to write a function that detects looping output -- I wrote one for promptfoo in [1].
For training you would need to hook it up to a LoRA training harness, identify the looping weight direction, and adjust the weights accordingly. I've not (yet) looked into how to do this.
[1] https://news.ycombinator.com/item?id=48555634
Is RSI here? No.
https://upload.wikimedia.org/wikipedia/en/b/ba/DrawingHands....
i feel like this could turn out to be a over unity demo where they say the model can improve a model better than it's own. (more energy output than input)
Isn't the network too shallow to be challenging to decode? One can easily produce few layer network to solve specific goal with construction techniques, so not a surprising result in my view.
What do you mean? Is there prior art? Would you call this a LORA?
My first reaction to your question (rhetorical?) was uncharitable... so, I am curious what you meant that it was "easy."
Semi-related question I was wondering about recently - how is it that Alibaba has a better consumer-friendly open model (for coding anyway) than research-leading companies like Google?
Well Google could open source Gemini 4 Argon but they choose to release the lower tier models open source instead... Its an active choice they all make.
So LLMs can mind meld... Spock is watching us from the other side very disappointed.
By the way, I'm surprised on how good Qwen3.8-27b is, my custom harness is doing fantastic on it. The only think I wish is having it like x10 faster.
But that's obvious... Astra is Qwen, if Astra can is because Qwen can....
Hmm. Possibilities for RSI? Big MOE does brain surgery on and invents new expert modules in its catalog?
Still amazing to me that a model that good, Qwen3.8:27b, runs on a laptop.
In half a year we are at the frontier...
I am surprised nobody caught on "paper shaped" for blog post: I chuckled!
That's an edit, I only added that about 15 minutes ago.
What's funny about it?
The question is can a 4b or smaller improve a 0.5b llm, or would it just get caught in a loop
Off topic - anyone had the occasion to work with IBM Watson?
Watson as I understand is a collection of things. What were you I tested in specifically?
Been too busy counting paperclips.
I did exactly the same thing with Qwen 3.8 27B but i also asked it something else: a couple years ago i wrote a simple upscaler[0] when i was learning about neural networks. The network is dead simple: it takes a 25 inputs for a 5x5 grid around a pixel and outputs the 2x2 pixel grid that the input pixel should be replaced for a (width2)x(height2) upscale. I trained it on a bunch of random images (in hindsight i should have trained it on game screenshots but w/e) and it looked quite nice.
So i asked Qwen 3.8 27B two things:
1. What it is this network all about? I only gave it the ann.c code that implemented a simple artificial neural network library in C and the setup code that specified the network's shape but not its purpose.
Qwen 3.8 27B tried a bunch of stuff, taking the layers one by one, proding numbers through it, etc and finally figured out it is some sort of 5x5 to 2x2 filter, most likely working with pixels (i never mentioned pixels or graphics), though it wasn't certain about the exact purpose. It did rule out a bunch of uses like being a downscaler or classifier, from its behavior.
2. After i revealed what it was, i asked it if it can improve it but keep it 100% compatible with the existing program that used the weights file. And sure enough, it did improve it. I gave it the original training C code and dataset. It made a few experiments (and wrote its own training C code because it found mine too slow :-P) and ended up improving the weights. The difference[1] is small, but it basically eliminated some small noise that was always there and made some transitions less "halo-y". Of course for the purpose i wanted it (use it to upscale games in realtime) it isn't that great (too heavy) but it was learning material anyway. Though one bonus thing i noticed is that it upscales my random doodles much better than any Krita filter[2] :-P.
The above isn't Qwen 3.8 27B observing the weights directly and making sense out of them (ok actually it did seem to make some sense of the weights in the "#1" but i didn't dump the entire network to it - though thinking about it, it is only around 16384 floats so it might work). It just used a bunch of scripts it wrote in Python (and later C) but i still found it interesting to watch it go at it.
[0] https://mastodon.gamedev.place/@badsector/113765818432603610
[1] http://runtimeterror.com/pages/iv/images/a3d94ea369ff95ecf05...
[2] http://runtimeterror.com/pages/iv/images/4e29d027a8aa6e89db5...
I hacked together a 4x (via applying it twice) scale once when i was modifying Gamescope to see how it'd look but it was both very slow and i didn't like the result so i only kept the 2x one[0].
Out of curiosity i did apply it twice to the drawing i posted in my previous comment with a 4x bilinear upscale made in Krita[1]. TBH i think a dedicated 4x network would do a better job though even then, there is only so much a network can hallucinate to fill in details :-P.
[0] https://github.com/badsectoracula/gamescope/tree/neural-netw...
[1] http://runtimeterror.com/pages/iv/images/20b9558400bcf26511d...
Which coding agent harness did you use with Qwen for that?
I've been using Pi though i'm currently doing the NIH thing and making a custom one in C++ with a web-based frontend :-P.
Wasn’t this one of the main marketing points for the Inkling model as well?
> It’s a cool paper, but some of the wider commentary I’ve seen around it does not feel justified. I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
The age of source code is definitely not going to end anytime soon if ever. But the future will involve incomprehensible machines editing other models in a "fairly certain" way, but we are going to need a lot of tooling to keep up with the process and not just take the model updates as "trust me bro".
I wonder if Qwen releases are deliberately nerfed for some reason. It's not the first time an easy reasoning performance win has been found for one.
https://arxiv.org/abs/2506.10947
have you tried it on wspbench?
Why do people keep calling this a paper?
It's a blog post which received feedback from collaborators.
It hasn't gone through the academic paper lifecycle ... the authors themselves refer to it as a post and as preliminary.
I feel like I'm taking crazy pills, but can we be precise when talking about work of this form? It started with referring to arXiv pre-prints as papers, and now we're at blog posts with an academic template as papers.
A paper is a fucking paper: something which has been vetted by the scientific community, with at least some form (often, a few rounds) of anonymous review from people outside your circle of collaborators
This is not there yet, and clearly so due to the various feedback from people (including the two pushback posts on HN) critiquing the lack of discussion about the existence of these techniques in mech interp, etc?
Like you said, in the AI space a pre-print on arXiv is massively more likely than anything that's been through peer-review.
I called it a paper here because it's by people with an academic affiliation and is of comparable quality (if not better than) many of the other "papers" we see.
I also like it when a paper isn't a PDF. PDFs are a bad way of communicating information.
(I will note that the authors themselves didn't call it a paper, their own BibTeX reference calls it a "blog post":)
> PDFs are a bad way of communicating information.
They have the strength of one PDF being frozen & (mostly) treated online as immutable - compared to a webpage which might change, or a blog post which might be edited. If a PDF needs to be replaced there'll be a notice.
And the single-file nature of PDFs makes them more likely to sit in an archive not managed by the author (vs a blog) so they tend to live as long as the archive rather than the author's blog.
For something that needs to Be referenced by others those are useful properties - I'd argue they outweigh how good PDF is at communicating information.
Sure, I agree with point (3).
But for (1, 2), just to point out, neither of your criteria would pass muster in the scientific communities who are ultimately responsible for vetting this work, right?
The ideal is: we don't accept papers at NeurIPS (or wherever) because of affiliation, and the badge of "quality" is bestowed by peer review, from other scientists.
> is of comparable quality (if not better than)
But the posts that you refer to in your own post point out several ways in which the quality is not good ...
But, in any case, I find it a bizarre choice to call it a paper when the authors themselves do not refer to it as such. We can agree to disagree, but just offering feedback.
You're right about this, but I don't regret my choice of words.
I admit that I don't think formal "papers" deserve their elevated status in today's information economy.
I think they add a whole lot of ceremony that usually results in a format that's harder to understand.
So me calling this a "paper" was a little bit of a dig!
Of course, I agree with you in many respects - but I’ve found the academic paper lifecycle to be tremendously helpful in terms of polishing my contributions (admittedly, not in ML - but in PL / systems) and avoiding content which could be construed as “misleading”
You, being of sound mind, tried the technique and drew reasonable conclusions … but just look at the original post for this blog post: there are a large number of responses by people who clearly don’t understand the technique, missed the experimental weaknesses, couldn’t place the post into the lineage of mech interp results that preceded it … and will leave the discussion with the wrong impression.
In your mind, whose fault is that? Is that that the unavoidable cost of a field, moving too rapidly to engage in peer review?
In its ideal form, the paper lifecycle is not a gate, but a sharpening stone.
I think we have a messy version of peer review playing out right now.
I'm not convinced the alternative - delaying publication by 6 months, keeping the new information available only to a much smaller group until then - would have been better.
All of that said, I did just update my own post to include "The paper (OK, the authors classify is as a “blog post”, but it’s paper-shaped) found that frontier models..." inspired by this conversation.
Agreed! Now if the authors update their post in response — we’ll have a proper venue on our hands!
Because nobody actually reviews papers anyway so it is irrelevant. That might have happened when you only had 12 papers a year and they were things like “gravity exists”, but when you get a million papers a day on topics like “the ethnocentric illusion of west-canadian yarn spinning”, nobody has time for that.
It's not considered a paper until it's vetted by others?
Yes: before that stage, it is often called a "pre-print" ... if it is not at the state of a "pre-print", then "blog post" ... :)
In most fields of science, it's not a paper until it's accepted for publication in a journal. That normally includes peer review of some sort.
Yeah, that's not gonna happen in ML, where a positive result (even in blog form) can be (hopefully) trivially tested and confirmed.
From memory, yolo and roberta and ppo were initially rejected. After a quick google search, pagerank was initially rejected, word2vec got one weak and one strong reject in reviews, LSTM got rejected (90s but still), dropout, knowledge distillation, and many more such cases. All of these are extremely important, foundational and field defining papers that ended up on arxiv or other preprint places.
If that happened to those papers, there's 0 chance today that it'd be a good idea. The field is simply moving too fast, there's too much being published, and the "top venues" are simply swamped. Blog / preprint / carrier pigeon is ok atm, as long as someone can look at it, try it, and hopefully reproduce it.
Now how long until Anthropic forbids weight space editing too?
i don't get his example, he had a simple model that only plays pong and Qwen can tell by just looking at the weights?
You can see the full prompt which describes the challenge (and includes the model weights) for the Pong example here: https://github.com/RikiyaT/lm_understand-logs/blob/main/exp1...
It literally just did math on some numbers based on a set of rules provided in the prompt in order to sketch a visual of a game. It didn't "interpret model weights" any more than an MP3 player "interprets physics of electromagnetic waves". The LLM built its own decoder on the fly for the games... which is what it's doing when you ask it to write code, calculate math, estimate the structural load bearing of a beam, etc.