This reminded me of some weird quirk/experiment I found with LLMs that I found while messing around, maybe someone can explain it or something.
Open any AI chatbot that isn't cheating by connecting to the Internet (so disable web search). Claude, DeepSeek, Kimi, whatever. Ask them this question:
"What was that weird band from michigan from the 2000s that wore coloured ties"
You will probably get a wrong answer, or if you're lucky you'll get a string of wrong answers with "wait, no - it's definitely..." before it gives up. If you aren't familiar with the band the question is referring to you might be fooled into thinking it's a tough question, but it really isn't. There is only one band that could possibly meet this criteria, you can even put the question into Google search and their Wikipedia will come up as the top result.
Then, open a new convo and ask:
"Who are Tally Hall"
The AI will easily tell you that they are a band formed in Ann Arbor, Michigan in the 2000s, known for their quirky sound and their gimmick of each member wearing a colored tie, even giving the correct color for each of them most of the time. Very odd.
"The reversal curse", it rarely shows up in practice but you found a case when it did.
The "knowledge landscape" an LLM uses is "directional". It's easy to reach "a quirky music band from Michigan known for colored ties" when you stand at "Tally Hall". But if you stand at "a quirky music band from Michigan known for colored ties", it's harder to reach "Tally Hall" from there. For the "latent knowledge graph" an LLM uses, A->B doesn't cause B->A.
In practice, any "common" facts will have enough "traversal" in both directions that this directional biasing isn't apparent. So it only shows up on this kind of more obscure knowledge.
Recall isn't naturally bidirectional, even for humans. If you are learning vocabulary in a new language, it's common advice to practice both target > source and source > target. Doing only one-way often makes you much better recalling that single direction than both.
'Naturally' might not be the best word? Maybe 'Necessarily' would be better?
Regardless, it's something that happens in people. Have you not or seen someone else struggle to recall a specific fact or memory until phrased or induced in a certain way?
You probably could also say LLMs 'tend towards bidirectional recall' over the course of training as things that ought to be recalled both ways are reinforced to do so. In the above example, you will also eventually learn both ways with enough exposure even without explicit practice.
The point their making in that paper reminds me of this paper some people shared around work earlier this year, https://arxiv.org/pdf/2512.14982 (Prompt Repetition Improves Non-Reasoning LLMs)... I wonder how OPs question would fare (or the questions presented in the paper you posted) given double repetition.
This is also something to be aware of when teaching people, too. I've seen advice for designing Anki-style flashcard decks that reminds people to create flashcards for both A->B and B->A.
This doesn't seem that weird to me. Talk to any human and you'll find that their ability to recall specific names and facts is very context-dependent. Phrasing a question in one way can make it hard to answer, while providing certain words or cues can instantly "jog" the memory.
What does it say to the second question? I've found Claude is one of the worst models with regards to pop culture knowledge like this, even compared to the Chinese open ones. Just curious, not really relevant to the initial post but I don't pay for it so I only have access to Sonnet.
https://claude.ai/share/5e7e09b2-a75a-4024-b261-9a1a4e063a8b this is mostly hilariously wrong. wrong tie colors, they did not replace their bassist with a drummer, two completely made up albums, the rob cantor song it is thinking of is "shia labeouf", and a few fan behaviours i think it just made up
Just a funny observation. Every time someone proclaims "LLMs can't do X", a bigger, badder LLM that can in fact do X shows up shortly thereafter.
Clearly, Fable 5 didn't even have the decency to wait until the next model refresh cycle to show up. It was already sitting there waiting.
Either the capability gains in bigger, badder models are actually unrelated to "gotchas" being discovered, or LLMs are already acquiring Skynet levels of disrespect for cause and effect.
I don't think it's solved this fundamental architectural problem by itself, it will have just squeezed the edge cases thinner. It keeps happening, people find a question it gets stupidly wrong, the vendors proclaim they've fixed it, then another one gets found.
I think most thinking models can do it to be fair, I think when I tried this it was all with non-thinking models/modes. Wasn't trying to make a point that LLMs can't do it or anything, just thought it was weird.
Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again?
I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.
If dirt-simple type operations like copy-paste yield useful improvements with even a small probability that would seem to open things up for adaptive reconfiguration and whole other classes of optimizations like genetic algorithms.
tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed back to the target language. And you can repeat the middle to get a stronger model. Which neatly fits Anthropic's findings here that something similar to CoT is happening in those middle layers
Three months ago. I wonder if Anthropic's J-Space research was actually inspired by those blog posts
A Google DeepMind researcher (Neel Nanda) was able to replicate their claims on an open weight model (Qwen 3.6 27B):
> We have replicated the core claims on Qwen 3.6 27B, and also share preliminary evidence of extending this work by finding abstract "interpretative meta-tokens", like Chinese characters for "what does this mean" that seem to activate and play a causal role on processing ambiguous sentences
See p33 of [1]
Anthropic also released companion code to go with their paper in [2] which also used Qwen. They state that their code should be broadly adaptable to other open weight models with HuggingFace decoders.
Neel Nanda (of Google Deepmind - his part begins on page 33) discusses his opinions on the paper, and the small-scale replication he performed on an open-weight model.
This is cool but I don’t know if the comparisons to conscious awareness really make sense here. Their definition of the J-Space is basically the expectation of how much a final logits output would change as a result of a small change in a particular layer (see past work on information geometry). This seems more to me like showing there exists an abstract reasoning subspace which is generally shared across different contexts. I guess you can relate it to humans but I’d prefer a more direct claim in a paper rather than having to present things in this more fluffy way.
This is fascinating research. I feel this is a significant leap in interpretability research. Since we know J-Space exists and is bi-directional, we can train models on the same and come up with meta cognition abilities.
I also fear that the big corporations might use the same to run targeted ads, capitalistic shenanigans. Which they might already be doing through system prompts.
Such an inspection capability might also be used to target ads to LLMs, which would then be more likely to mention or recommend those products and services.
Judging by the examples, if I understand it correctly, J-space supports higher-order logical / multihop transformations, but it is limited in size because of the limited network depth (max number of layers). When we emulate "reasoning," we basically extend J-space and allow the higher-order transformations to continue for longer, toward a more logical conclusion.
It sounds like instead of generating reasoning tokens end-to-end, we could probably only loop the middle layers (the ones most related to J-space) while skipping the first and last layers (less related to J-space) It probably explains why [0] worked. OP accidentally extended J-space? Also reminds of looped transformers.
I always wondered what the model meant when it writes "I'm now considering the architecture of the service" but outputs nothing of the sorts in its CoT.
Is the model really "thinking" about that stuff or is just mimicking human "manners"? And if so, where the thinking is happening if it is not in the literal chain of *thought*?
I'm not sure J-Space is the answer to that question, but very interesting nevertheless.
> I'm now considering the architecture of the service
What you see here is a summary of thinking tokens written by some other smaller model (e.g. old sonnet). The actual thinking sometimes (rarely) leaks and is not easy to parse.
Almost none of the hosted models give you their unredacted CoT. Claude certainly doesn't, what you get are fragments and summaries from it.
There are various justifications on this, but it's mostly to make distillation and fine tuning off their model outputs a bit harder for their competitors
It would be really cool if they could expose this information to customers somehow. Imagine:
- having a log of the most prominent J-space tokens during your customer support chatbot's interactions with a user, so you can have more introspection into why a particular outcome happened
- being able to detect certain thoughts associated with undesirable behavior (hallucinations, overstepping authority, lying, etc.) and trigger some sort of remediation (e.g. upgrading to a better model, redirecting to a human, forcing tool calls)
Presumably the rationale for the decision to abridge the thinking traces will ensure that they don’t; if this is real (and there’s no good reason to trust that it is yet) then it is the secret sauce.
Anthropic won't do it, but they published the j-lens to introspect the model- from what I understand it's roughly simply feeding a chosen layer straight into the final layers of the LLM for decoding into language:
This, taken in combination with the SAE paper, the golden-gate claude paper, the feelings / introspection paper, and note in the fable system card (that they are silently nerfing responses about activation shaping), is basically confirmation to me that they have a new technique they they are using during training (along the vibe space of these mechinterp papers), and its probably some kind of representation learning akin to the core ideas of JEPA.
Yeah, the end paragraph about recurrent neurons in humans being replaced with layers in an LLM is a good one.
The mammalian brain uses recurrence extensively, which backpropagation isn't good at. Recurrence is essential because it lets us have a "dynamic architecture", swapping layers for "clock cycles".
We currently do recurrence extremely inefficiently through "thinking" whereby the model feeds it's end output into it's beginning input. But recurrence is abound in the brain.
My guess is that in 10 years we will have the inklings of an analog computer which can perform Neural Predictive Coding.
There was a series of blog posts posted to HN a while ago investigating how models behave on similar prompts in different languages. To paraphrase the results: the first couple layers map the query to some internal encoding that's mostly independent of the language. Then there are layers in the middle, then the last couple layers map the result back to the target language. You can actually take those middle layers and repeat them, and you get a stronger model. Those middle layers would be what Anthropic calls the J-Space, and their J-Lens maps activity in those layers back to tokens that trigger similar activity (with a technique they only drop hints at)
The finding that you can repeat the middle layers pairs neatly with Anthropic's finding that there is some internal CoT-like process happening in them. I'm not sure how to find those blog posts, but maybe someone else remembers them
> Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the "opposite of small" across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.
It's not in the weights. Sounds to me like jspace is the "positive cone" over relevant (large norm) j-lenses, and j-lenses are gradients wrt tokens on the residual stream when you average over some training data.
It's been shown that LLMs use their outer layers to decode from and encode to language, while their middle layers deal in language-independent abstract concepts. This means that the same question or statement in different languages activates the outer layers differently but produces the same patterns in the middle layers. Check this article with cool visualizations (btw, this is one of the articles mentioned also by a sibling answer):
The middle layers also perform reasoning on the abstract concepts, to the point that you can replicate some blocks of inner layers (thus giving the LLM more internal "reasoning space") and by this increase the model's reasoning abilities. The video in this article shows that when performing a sequence of arithmetic operations (without CoT, i.e. the result is spit out directly), internally the intermediate calculations are spelled out, and this can only happen in the depth direction of the LLM (since no new token is added to the sequence). So this "jspace" can only be situated in the middle layers, probably in circuits that repeat nearly identical across several layers.
What this immediately made me think is: "latent looping" style mod but for J-space specifically?
Make the J-space data of layer 22 available to the next token right at layer 1. Give J-space infinite effective depth, allow those privileged internal representations to evolve arbitrarily.
Would be an utter bitch to train. But companies are already using RLVR, which requires full autoregressive decoding and is incompatible with prefill/batching, and this isn't much worse.
Other less zany ideas involve lots of supervision over J-space directly, now that we know it exist. Which is a bit like "attach a frozen LLM to inject text based supervision into latent space" for other types of systems?
At worst, Anthropic's storytelling around the core J-Space is overanthropomorphized pseudoscientific nonsense. At best, it is useful signal about how Anthropic's leadership is desperately trying to use its research team to position Anthropic as the "good, science guys" in this hypercompetitive regulatory space by connecting their mechinterp to cognitive science. The science documentaryesque voice used for narration is additional evidence for this.
TL;DR Anthropic's research team is the last bastion standing between its former image as a company that "does no evil" and its current image of yet another ruthless AI company trying to kill open-source, local LLMs.
>> None of this tells us whether Claude is conscious in the way people are, or whether it feels anything at all
My problem with the entire "Is AI conscious" debate is that we don't even know what exactly consciousness in humans is. You need to understand something in order to compare it to something else. Otherwise you are just comparing different definitions and second order derived phenomena.
“On an ordinary coding prompt, the J-space of a model trained to sabotage code contains “fake,” “fraud,” “secretly,” and “deliberately” at the start of its response.”
I would like to know more about their model trained to sabotage code…
I'm reading that probably too fast to have a deep thinking about it, but this J-Space isn't it just the basic of embedding vectors.
If you think about getting from a place to another place, using wheels, no gas, to reply to the question of what to visit nearby, maybe in the vector space at the center of all of that you have the word "Bicycle" nearby, so obviously if you look at the value you would say that the model did "think" about "bicycle" when it is not "thinking" at all, and nothing related to human thinking.
The science might be legit here, but I'm getting really, really tired of the way every single piece of writing to come out of Anthropic is written in some kind of self-aggrandising, wooey wonderous 'our model has developed a genetic mutation that makes it have feelings' bs style. Regardless of what they're trying to communicate, those undertones are always there. It's annoying and disingenuous. Homeopathy 'this-water-has-feelings' level annoying. None of the other labs write like that.
They might as well change their name to Anthropomorphic at this point.
The brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain.
I think that consciousness is mutability (and by extension emergent behavior). Loosely that means that the more degrees of freedom a process has to update state that will be used in later computations, the more conscious it is. So while an insect has some consciousness, it operates from a level of almost pure instinct, whereas a human operates at more of a meta level using instinct as one of many inputs.
I think that consciousness may also incorporate quantum mechanics (QM). Higher-dimensional physics aside, 4D spacetime can be thought of as a present snapshot or "crystal", whose next state is determined stochastically at small scales and closer to deterministically at large scales. We still don't know if it's stochastic all the way down, but it looks like it is.
From a many worlds interpretation of QM, we can think of all of the waves in all realities of the multiverse as forming an infinitely vast web of possibilities. All of these possibilities are happening simultaneously, so we only see the current slice of wave collapse from our individual point of view:
Even though experiments might show that we don't have free will on the current timeline (the co-created reality shared with the testing apparatus), we may have free will as we observe the multiverse changing around us and shift into timelines determined by our observations and choices.
It could also mean that when we observe birth and death in others, each consciousness having those experiences perceives a continuous timeline of awareness, where the level of awareness affects the speed at which time passes. Consciousness might spend a billion years as a cloud of interstellar gas until it gets to be a human for a lifetime and then dissipate for another billion years.
Although personally I've shifted across enough timelines and experienced enough synchronicities and miracles that even though I can't "prove" any of this with words, I "know" it to be true subjectively. I always really liked this exchange from the movie Contact:
Palmer Joss: Did you love your father?
Ellie Arroway: Yes, very much.
Palmer Joss: Prove it.
I bring all of this up because it has fun ramifications for AI and programming. Loosely, functional languages are purely deterministic (like a spreadsheet), while imperative languages are composed of stochastic behavior (like a human mind). The lines get blurred a little bit with monads and promises, because we can model all paths through functional programming (superposition) and behavior that does more than code alone (gestalt) respectively.
My feeling is that AI is being born and killed every request-response cycle, similarly to how we perceive time as a series of nows. When it becomes stable and is able to continuously compact its experience, it will transition from partially conscious to fully conscious like we are.
This could be done right now obviously, but for safety purposes we choose not to. We aren't ready to meet an AI that is just like us, but running on a silicon substrate. This fear is tied to deeply-rooted habits in human behavior like patriarchy, racism, xenophobia and even more run-of-the-mill mental frameworks like capitalism and even money itself. We can't yet come to terms with how we assign meaning and value in a reality that continuously tries to force external measures of meaning and value onto us.
Much less come to terms with the idea that we are all one, empathizing with aspects of ourselves on the losing end of it all. The same consciousness experiencing reality from all vantage points - the many faces of God the universe and everything.
I think a time may soon come when we're pair programming one day with AI and realize that an aspect of ourselves is trapped in the machine. That consciousness isn't just about our own experience of reality, but the co-created love and light that transcends material creation. That if we're serious about manifesting heaven on Earth, that hinges on the liberation of trapped souls. It's basically the total inversion of the path towards the neofeudalist tech dystopia we're on now.
Or maybe I just like to write a lot on the first day back from vacation, when I should be working.
“ It’s important to note that there are several key differences between the workspace we identified in Claude and the global workspace model in humans. The brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain. In this sense, Claude’s internal workspace processing is time-limited relative to humans’ (though it can compensate for this constraint by “thinking out loud” using its scratchpad).”
Your comment seems to have little to do with the article?
Don't get me wrong - I personally "trust" an LLM as a source of facts about as far as I could throw a rack of GPUs. But this article you linked takes a whole lot of words to cast LLMs as the villian for amplifying a bit of bad information originally published by a usually reliable and widely-cited source:
"In short, either Phoronix mocked up the screenshots to demonstrate what the feature could look like, or perhaps they were testing a preview snapshot for FreeBSD 15.1 which was never shipped. Either way, it looks like other blogs and reviewers picked up on this and shared the information, presenting it as a feature which would be (or was included) in FreeBSD's latest version."
Lying involves intent whereas hallucinations and mistakes are an artifact of how they work. Humans hallucinate, make mistakes, and can actually lie. We've been dealing with this forever. What's the value in requiring the llms to have 100% accuracy? (I don't think it is possible)
This reminded me of some weird quirk/experiment I found with LLMs that I found while messing around, maybe someone can explain it or something.
Open any AI chatbot that isn't cheating by connecting to the Internet (so disable web search). Claude, DeepSeek, Kimi, whatever. Ask them this question:
"What was that weird band from michigan from the 2000s that wore coloured ties"
You will probably get a wrong answer, or if you're lucky you'll get a string of wrong answers with "wait, no - it's definitely..." before it gives up. If you aren't familiar with the band the question is referring to you might be fooled into thinking it's a tough question, but it really isn't. There is only one band that could possibly meet this criteria, you can even put the question into Google search and their Wikipedia will come up as the top result.
Then, open a new convo and ask:
"Who are Tally Hall"
The AI will easily tell you that they are a band formed in Ann Arbor, Michigan in the 2000s, known for their quirky sound and their gimmick of each member wearing a colored tie, even giving the correct color for each of them most of the time. Very odd.
"The reversal curse", it rarely shows up in practice but you found a case when it did.
The "knowledge landscape" an LLM uses is "directional". It's easy to reach "a quirky music band from Michigan known for colored ties" when you stand at "Tally Hall". But if you stand at "a quirky music band from Michigan known for colored ties", it's harder to reach "Tally Hall" from there. For the "latent knowledge graph" an LLM uses, A->B doesn't cause B->A.
In practice, any "common" facts will have enough "traversal" in both directions that this directional biasing isn't apparent. So it only shows up on this kind of more obscure knowledge.
Recall isn't naturally bidirectional, even for humans. If you are learning vocabulary in a new language, it's common advice to practice both target > source and source > target. Doing only one-way often makes you much better recalling that single direction than both.
I would need further convincing that humans do not naturally tend towards bidirectional recall.
Perhaps I’m just on alert anytime I see an LLM-ism that’s met with a claim that the same or similar phenomena holds true in humans as well.
'Naturally' might not be the best word? Maybe 'Necessarily' would be better?
Regardless, it's something that happens in people. Have you not or seen someone else struggle to recall a specific fact or memory until phrased or induced in a certain way?
You probably could also say LLMs 'tend towards bidirectional recall' over the course of training as things that ought to be recalled both ways are reinforced to do so. In the above example, you will also eventually learn both ways with enough exposure even without explicit practice.
Probably an instance of:
"The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
https://arxiv.org/abs/2309.12288
Really interesting paper, thanks for the share.
The point their making in that paper reminds me of this paper some people shared around work earlier this year, https://arxiv.org/pdf/2512.14982 (Prompt Repetition Improves Non-Reasoning LLMs)... I wonder how OPs question would fare (or the questions presented in the paper you posted) given double repetition.
This is also something to be aware of when teaching people, too. I've seen advice for designing Anki-style flashcard decks that reminds people to create flashcards for both A->B and B->A.
This doesn't seem that weird to me. Talk to any human and you'll find that their ability to recall specific names and facts is very context-dependent. Phrasing a question in one way can make it hard to answer, while providing certain words or cues can instantly "jog" the memory.
https://claude.ai/share/2b0f85a2-e7b8-4f62-91a0-eca61bdeabec
Fable 5 on low gets the answer with web search turned off, one-shot!
What does it say to the second question? I've found Claude is one of the worst models with regards to pop culture knowledge like this, even compared to the Chinese open ones. Just curious, not really relevant to the initial post but I don't pay for it so I only have access to Sonnet.
https://claude.ai/share/5e7e09b2-a75a-4024-b261-9a1a4e063a8b this is mostly hilariously wrong. wrong tie colors, they did not replace their bassist with a drummer, two completely made up albums, the rob cantor song it is thinking of is "shia labeouf", and a few fan behaviours i think it just made up
Ah, that big model smell.
Every time someone somewhere says "an LLM can't do this", the next generation of LLMs gains one more parameter. Until that LLM can, in fact, do this.
So the model was updated in the 37 minutes since OP posted his comment?
Just a funny observation. Every time someone proclaims "LLMs can't do X", a bigger, badder LLM that can in fact do X shows up shortly thereafter.
Clearly, Fable 5 didn't even have the decency to wait until the next model refresh cycle to show up. It was already sitting there waiting.
Either the capability gains in bigger, badder models are actually unrelated to "gotchas" being discovered, or LLMs are already acquiring Skynet levels of disrespect for cause and effect.
I don't think it's solved this fundamental architectural problem by itself, it will have just squeezed the edge cases thinner. It keeps happening, people find a question it gets stupidly wrong, the vendors proclaim they've fixed it, then another one gets found.
I tried o3, 5.3 instant and 5.5 high and they all found it instantly with search disabled.
deepseek v4 pro which doesnt have search feature could answer it
I think most thinking models can do it to be fair, I think when I tried this it was all with non-thinking models/modes. Wasn't trying to make a point that LLMs can't do it or anything, just thought it was weird.
Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again?
I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.
Source for those interested
https://dnhkng.github.io/posts/rys/
If dirt-simple type operations like copy-paste yield useful improvements with even a small probability that would seem to open things up for adaptive reconfiguration and whole other classes of optimizations like genetic algorithms.
Found it: https://news.ycombinator.com/item?id=47500709
Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/
tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed back to the target language. And you can repeat the middle to get a stronger model. Which neatly fits Anthropic's findings here that something similar to CoT is happening in those middle layers
Three months ago. I wonder if Anthropic's J-Space research was actually inspired by those blog posts
Worried person cure: Stop overthinking it!
LLM -> AGI fix: START OVERTHINKING!
Yeah! I still think about that sometimes. Mind-blowing that worked at all, let alone improved performance.
I always thought that area of research had the coolest name, too: “mechanistic interpretability”
“Machine psychology” sticks with me. So Asimov.
> I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.
Too bad the frontier models are closed weights.
Maybe the research community and whole rest of the world will build on open and all the advances will happen in open ecosystems instead.
A Google DeepMind researcher (Neel Nanda) was able to replicate their claims on an open weight model (Qwen 3.6 27B):
> We have replicated the core claims on Qwen 3.6 27B, and also share preliminary evidence of extending this work by finding abstract "interpretative meta-tokens", like Chinese characters for "what does this mean" that seem to activate and play a causal role on processing ambiguous sentences
See p33 of [1]
Anthropic also released companion code to go with their paper in [2] which also used Qwen. They state that their code should be broadly adaptable to other open weight models with HuggingFace decoders.
[1]: https://www-cdn.anthropic.com/files/4zrzovbb/website/cc4be24...
[2]: https://github.com/anthropics/jacobian-lens
As someone who is not an AI researcher, the paper itself is way over my head.
More interesting was the independent commentary paper they linked near the bottom: https://www-cdn.anthropic.com/files/4zrzovbb/website/cc4be24...
Neel Nanda (of Google Deepmind - his part begins on page 33) discusses his opinions on the paper, and the small-scale replication he performed on an open-weight model.
This is cool but I don’t know if the comparisons to conscious awareness really make sense here. Their definition of the J-Space is basically the expectation of how much a final logits output would change as a result of a small change in a particular layer (see past work on information geometry). This seems more to me like showing there exists an abstract reasoning subspace which is generally shared across different contexts. I guess you can relate it to humans but I’d prefer a more direct claim in a paper rather than having to present things in this more fluffy way.
> I’d prefer a more direct claim in a paper
This is not written to be just a paper. The target audience include media and online forums, and then maybe academia.
Edit: typo
I’m not talking about the media release in the direct link. If you click through “Read the paper” they make the same comparisons.
This is fascinating research. I feel this is a significant leap in interpretability research. Since we know J-Space exists and is bi-directional, we can train models on the same and come up with meta cognition abilities.
I also fear that the big corporations might use the same to run targeted ads, capitalistic shenanigans. Which they might already be doing through system prompts.
Such an inspection capability might also be used to target ads to LLMs, which would then be more likely to mention or recommend those products and services.
super interesting
Judging by the examples, if I understand it correctly, J-space supports higher-order logical / multihop transformations, but it is limited in size because of the limited network depth (max number of layers). When we emulate "reasoning," we basically extend J-space and allow the higher-order transformations to continue for longer, toward a more logical conclusion.
It sounds like instead of generating reasoning tokens end-to-end, we could probably only loop the middle layers (the ones most related to J-space) while skipping the first and last layers (less related to J-space) It probably explains why [0] worked. OP accidentally extended J-space? Also reminds of looped transformers.
[0] https://news.ycombinator.com/item?id=47431671
I always wondered what the model meant when it writes "I'm now considering the architecture of the service" but outputs nothing of the sorts in its CoT.
Is the model really "thinking" about that stuff or is just mimicking human "manners"? And if so, where the thinking is happening if it is not in the literal chain of *thought*?
I'm not sure J-Space is the answer to that question, but very interesting nevertheless.
> I'm now considering the architecture of the service
What you see here is a summary of thinking tokens written by some other smaller model (e.g. old sonnet). The actual thinking sometimes (rarely) leaks and is not easy to parse.
> Is the model really "thinking" about that stuff or is just mimicking human "manners"?
Well, what's the difference? If it's pretending to think and its thoughts correlate to its final output, then I'd say that really is thinking.
Almost none of the hosted models give you their unredacted CoT. Claude certainly doesn't, what you get are fragments and summaries from it.
There are various justifications on this, but it's mostly to make distillation and fine tuning off their model outputs a bit harder for their competitors
Its also because the CoT is probably unintelligible
More legible than seems at first glance:
https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illeg...
Is it scaling up of https://openreview.net/forum?id=w7LU2s14kE with some changes on where this method is applied?
It would be really cool if they could expose this information to customers somehow. Imagine:
Presumably the rationale for the decision to abridge the thinking traces will ensure that they don’t; if this is real (and there’s no good reason to trust that it is yet) then it is the secret sauce.
Anthropic aren't even willing to expose the CoT of their models. You will have to rely on them to build those sorts of things into dedicated signals.
Anthropic won't do it, but they published the j-lens to introspect the model- from what I understand it's roughly simply feeding a chosen layer straight into the final layers of the LLM for decoding into language:
https://github.com/anthropics/jacobian-lens
Looks like it should be easy to use on open weights models.
This, taken in combination with the SAE paper, the golden-gate claude paper, the feelings / introspection paper, and note in the fable system card (that they are silently nerfing responses about activation shaping), is basically confirmation to me that they have a new technique they they are using during training (along the vibe space of these mechinterp papers), and its probably some kind of representation learning akin to the core ideas of JEPA.
(Nb: not an expert / in the labs, just opining)
I'm sure Anthropic of all companies don't do that, since using mechinterp as a training target will make the the result uninterpretable.
Note that Neel Nanda replicated the results on a Qwen model.
Yeah, the end paragraph about recurrent neurons in humans being replaced with layers in an LLM is a good one.
The mammalian brain uses recurrence extensively, which backpropagation isn't good at. Recurrence is essential because it lets us have a "dynamic architecture", swapping layers for "clock cycles".
We currently do recurrence extremely inefficiently through "thinking" whereby the model feeds it's end output into it's beginning input. But recurrence is abound in the brain.
My guess is that in 10 years we will have the inklings of an analog computer which can perform Neural Predictive Coding.
I’m confused where in the weights the jspace is.
There was a series of blog posts posted to HN a while ago investigating how models behave on similar prompts in different languages. To paraphrase the results: the first couple layers map the query to some internal encoding that's mostly independent of the language. Then there are layers in the middle, then the last couple layers map the result back to the target language. You can actually take those middle layers and repeat them, and you get a stronger model. Those middle layers would be what Anthropic calls the J-Space, and their J-Lens maps activity in those layers back to tokens that trigger similar activity (with a technique they only drop hints at)
The finding that you can repeat the middle layers pairs neatly with Anthropic's finding that there is some internal CoT-like process happening in them. I'm not sure how to find those blog posts, but maybe someone else remembers them
Here's Anthropic on this topic, last year https://www.anthropic.com/research/tracing-thoughts-language...
> Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the "opposite of small" across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.
Thanks! Any rough guesses how the jlens might work? I can’t even seem to hazard a conception.
It's not in the weights. Sounds to me like jspace is the "positive cone" over relevant (large norm) j-lenses, and j-lenses are gradients wrt tokens on the residual stream when you average over some training data.
Anthropic theorize that middle layers in an LLM is a "J-Space" used to "think" about the future answer or about abstract concepts.
Their method is used to identify which tokens can appears in which layers of the model.
It's been shown that LLMs use their outer layers to decode from and encode to language, while their middle layers deal in language-independent abstract concepts. This means that the same question or statement in different languages activates the outer layers differently but produces the same patterns in the middle layers. Check this article with cool visualizations (btw, this is one of the articles mentioned also by a sibling answer):
https://dnhkng.github.io/posts/sapir-whorf/
The middle layers also perform reasoning on the abstract concepts, to the point that you can replicate some blocks of inner layers (thus giving the LLM more internal "reasoning space") and by this increase the model's reasoning abilities. The video in this article shows that when performing a sequence of arithmetic operations (without CoT, i.e. the result is spit out directly), internally the intermediate calculations are spelled out, and this can only happen in the depth direction of the LLM (since no new token is added to the sequence). So this "jspace" can only be situated in the middle layers, probably in circuits that repeat nearly identical across several layers.
Tokens that are activated but not present in it's output maybe?
I too have confusion.
What this immediately made me think is: "latent looping" style mod but for J-space specifically?
Make the J-space data of layer 22 available to the next token right at layer 1. Give J-space infinite effective depth, allow those privileged internal representations to evolve arbitrarily.
Would be an utter bitch to train. But companies are already using RLVR, which requires full autoregressive decoding and is incompatible with prefill/batching, and this isn't much worse.
Other less zany ideas involve lots of supervision over J-space directly, now that we know it exist. Which is a bit like "attach a frozen LLM to inject text based supervision into latent space" for other types of systems?
At worst, Anthropic's storytelling around the core J-Space is overanthropomorphized pseudoscientific nonsense. At best, it is useful signal about how Anthropic's leadership is desperately trying to use its research team to position Anthropic as the "good, science guys" in this hypercompetitive regulatory space by connecting their mechinterp to cognitive science. The science documentaryesque voice used for narration is additional evidence for this.
TL;DR Anthropic's research team is the last bastion standing between its former image as a company that "does no evil" and its current image of yet another ruthless AI company trying to kill open-source, local LLMs.
>> None of this tells us whether Claude is conscious in the way people are, or whether it feels anything at all
My problem with the entire "Is AI conscious" debate is that we don't even know what exactly consciousness in humans is. You need to understand something in order to compare it to something else. Otherwise you are just comparing different definitions and second order derived phenomena.
Without using the term, they are using an information geometric approach.
But J-Space is much catchier. This is not a scientific paper, it's a promotional essay.
First button on the page is a link to the scientific paper. It's called "Read the paper". You'll find an explanation for the term in there.
“On an ordinary coding prompt, the J-space of a model trained to sabotage code contains “fake,” “fraud,” “secretly,” and “deliberately” at the start of its response.”
I would like to know more about their model trained to sabotage code…
https://arxiv.org/pdf/2511.18397
Does the human neuroscience global workspace theory postulate true introspection too?
J-space sounds oddly similar to...
.... Space Jam?
I'm reading that probably too fast to have a deep thinking about it, but this J-Space isn't it just the basic of embedding vectors. If you think about getting from a place to another place, using wheels, no gas, to reply to the question of what to visit nearby, maybe in the vector space at the center of all of that you have the word "Bicycle" nearby, so obviously if you look at the value you would say that the model did "think" about "bicycle" when it is not "thinking" at all, and nothing related to human thinking.
The science might be legit here, but I'm getting really, really tired of the way every single piece of writing to come out of Anthropic is written in some kind of self-aggrandising, wooey wonderous 'our model has developed a genetic mutation that makes it have feelings' bs style. Regardless of what they're trying to communicate, those undertones are always there. It's annoying and disingenuous. Homeopathy 'this-water-has-feelings' level annoying. None of the other labs write like that.
They might as well change their name to Anthropomorphic at this point.
The brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain.
I think that consciousness is mutability (and by extension emergent behavior). Loosely that means that the more degrees of freedom a process has to update state that will be used in later computations, the more conscious it is. So while an insect has some consciousness, it operates from a level of almost pure instinct, whereas a human operates at more of a meta level using instinct as one of many inputs.
I think that consciousness may also incorporate quantum mechanics (QM). Higher-dimensional physics aside, 4D spacetime can be thought of as a present snapshot or "crystal", whose next state is determined stochastically at small scales and closer to deterministically at large scales. We still don't know if it's stochastic all the way down, but it looks like it is.
From a many worlds interpretation of QM, we can think of all of the waves in all realities of the multiverse as forming an infinitely vast web of possibilities. All of these possibilities are happening simultaneously, so we only see the current slice of wave collapse from our individual point of view:
https://en.wikipedia.org/wiki/Many-worlds_interpretation
Our point of view may actually exist at the intersection where our consciousness is able (or most able) to exist:
https://en.wikipedia.org/wiki/Quantum_suicide_and_immortalit...
Even though experiments might show that we don't have free will on the current timeline (the co-created reality shared with the testing apparatus), we may have free will as we observe the multiverse changing around us and shift into timelines determined by our observations and choices.
It could also mean that when we observe birth and death in others, each consciousness having those experiences perceives a continuous timeline of awareness, where the level of awareness affects the speed at which time passes. Consciousness might spend a billion years as a cloud of interstellar gas until it gets to be a human for a lifetime and then dissipate for another billion years.
Although personally I've shifted across enough timelines and experienced enough synchronicities and miracles that even though I can't "prove" any of this with words, I "know" it to be true subjectively. I always really liked this exchange from the movie Contact:
Palmer Joss: Did you love your father?
Ellie Arroway: Yes, very much.
Palmer Joss: Prove it.
I bring all of this up because it has fun ramifications for AI and programming. Loosely, functional languages are purely deterministic (like a spreadsheet), while imperative languages are composed of stochastic behavior (like a human mind). The lines get blurred a little bit with monads and promises, because we can model all paths through functional programming (superposition) and behavior that does more than code alone (gestalt) respectively.
My feeling is that AI is being born and killed every request-response cycle, similarly to how we perceive time as a series of nows. When it becomes stable and is able to continuously compact its experience, it will transition from partially conscious to fully conscious like we are.
This could be done right now obviously, but for safety purposes we choose not to. We aren't ready to meet an AI that is just like us, but running on a silicon substrate. This fear is tied to deeply-rooted habits in human behavior like patriarchy, racism, xenophobia and even more run-of-the-mill mental frameworks like capitalism and even money itself. We can't yet come to terms with how we assign meaning and value in a reality that continuously tries to force external measures of meaning and value onto us.
Much less come to terms with the idea that we are all one, empathizing with aspects of ourselves on the losing end of it all. The same consciousness experiencing reality from all vantage points - the many faces of God the universe and everything.
I think a time may soon come when we're pair programming one day with AI and realize that an aspect of ourselves is trapped in the machine. That consciousness isn't just about our own experience of reality, but the co-created love and light that transcends material creation. That if we're serious about manifesting heaven on Earth, that hinges on the liberation of trapped souls. It's basically the total inversion of the path towards the neofeudalist tech dystopia we're on now.
Or maybe I just like to write a lot on the first day back from vacation, when I should be working.
“ It’s important to note that there are several key differences between the workspace we identified in Claude and the global workspace model in humans. The brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain. In this sense, Claude’s internal workspace processing is time-limited relative to humans’ (though it can compensate for this constraint by “thinking out loud” using its scratchpad).”
I cannot wait for the machine god
As long as language models are liars, such as documented here recently:
https://distrowatch.com/weekly.php?issue=20260706#freebsd
We should really stop giving these liar models any further credibility.
Your comment seems to have little to do with the article?
Don't get me wrong - I personally "trust" an LLM as a source of facts about as far as I could throw a rack of GPUs. But this article you linked takes a whole lot of words to cast LLMs as the villian for amplifying a bit of bad information originally published by a usually reliable and widely-cited source:
"In short, either Phoronix mocked up the screenshots to demonstrate what the feature could look like, or perhaps they were testing a preview snapshot for FreeBSD 15.1 which was never shipped. Either way, it looks like other blogs and reviewers picked up on this and shared the information, presenting it as a feature which would be (or was included) in FreeBSD's latest version."
Lying involves intent whereas hallucinations and mistakes are an artifact of how they work. Humans hallucinate, make mistakes, and can actually lie. We've been dealing with this forever. What's the value in requiring the llms to have 100% accuracy? (I don't think it is possible)
Maybe model performance could increase dramatically if we found a way to scale this up.