Hi! Author here. Surprised to see this on HN now. Happy to answer any questions!
Some context about this:
- This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs
- Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost and w these few training samples.
- Sample Efficiency is one of the most important unsolved problems today in AI. That's what I was targetting with this work. We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible (also why OpenAI's Parameter Golf had fixed compute and why Modded NanoGPT is considered very sample efficient)
- Can the perf be improved? Yes but the competition is ongoing so can't talk about it
- Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho
- Fun: I was new to ML when I posted this first (dec '25). I basically used ARC as a way to learn ML
"- Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho"
Also, in curating the training data in a deliberate manner, with attention to detail. Most people just use existing datasets and call it a day. It's a lot of work, which is why there are gains on the table.
Thank you for this excellent post series. It reminds me a lot of the pre-LLM days, though I was mostly using LSTMs back then. When the original GPT paper came out, I thought the future would be using LLMs to generate tons of synthetic labeled data and then training specialized LSTM or transformer models per-task.
Had a couple of questions:
1) You note that ARC-AGI is a meta-learning task, have you tried any meta-learning algorithms such as MAML?
2) Do you think this approach could extend to ARC-AGI 3? Or do you think the interactive environments require a higher level of complexity than what can be achieved with a small model?
1) Unfortunately I didn't. I was v new to ML when I did this and didnt have time or skill to try many things. Will try them when I get some time!
2) Possibly, but it would require significant changes and effort. But much larger models would be required imo (must have capacity greater than the complexity of the problem)
I spent some time working with that approach of using LLMs to generate synthetic labeled data for use in training more specialized models. It mostly didn't work.
The problem was that getting the LLM to generate training data that sufficiently resembled real-world data was labor intensive and expensive. More labor intensive and expensive, it turns out, than just using real data.
What worked better was using the LLM to label the training data. But even there we had to be careful about introducing weird biases.
Pretty sure it's one of those "All squares are rectangles but not all rectangles are squares" situations.
Transformers are what really started the LLM Boom, and seem to be crucial to the technology. They also have other applications, such as what OP created
It gets extremely blurry, because people commonly refer to any model that uses a component associated with the Transformer architecture as a Transformer (i.e. using some kind of QKV-esque attention mechanism). I think it's easier to think of it like this:
A large language model is just what it says--a very large statistical model trained for language tasks. This covers the spectrum of GPT-style models, but also those hard to classify ones, like Liquid's "Liquid Foundation Models", which can get up to 24 billion parameters and use grouped query attention, but are closely related to state-space models as well: https://huggingface.co/LiquidAI/LFM2-24B-A2B
Also, as others have pointed out, a Transformer isn't inherently a language model. So really they're sort of two different axes, one classifying the model size and task, the other referring to a specific architecture.
In my unpopular opinion, it wasn't transformers or attention, but pretraining on language data that kicked off the LLM Boom. Alec radford in his little jupyter notebook trained a very small non-transformer to predict simply the next-character on Amazon reviews. He noticed emergence of a neuron which when toggled controlled the sentiment of the remaining text. This is published as the sentiment neuron paper.
Till then, transformers were being used primarily for stuff like translation and such and no one was even pretraining at scale, even tho transformers and attention existed.
Openai and google if you count T5 persisting with pretrained generative models was what led to the LLM boom. Yes they used transformers, but that's just one IMO minor aspect.
Also, not all language models are transformers. You can have language models based on diffusion models or state-space models, or any other model that can be used to model sequences (so all of them, as sequences are just functions).
Transformers are just the ones that are most common and successful today.
I'd argue that meaningful sequences of symbols constitute a language. This example doesn't use a human language but it does use a language IMO (at least AFAIU).
I don't understand what you're saying? An LLM is a transformer model trained on a large corpus of natural language, often with some post-training. An image model is a different type of transformer model. What's controversial here?
Your example is a meaningless sequence. So consider a different scenario where the sequence is meaningful but does not map to any human language. What exactly disqualifies it as a non-human language?
When you encounter a human language that you can't personally read presumably you don't proceed to claim that it doesn't constitute language on the basis of your own lack of ability.
To come at it from a slightly different angle - does compiled binary code count as a language? If not, why not? (I'll suggest that it's a language albeit not a natural one.)
Any sequence could be meaningful or meaningless depending on the grammar involved. Open a word document in photoshop and the program has no idea what it’s dealing with.
The reason the language term in LLM is meaningful is how the training, symbol mapping, etc is designed around human languages. The model doesn’t process raw text, instead there’s a critical processing step which allows the magic to happen.
To my mind the argument against this model qualifying as a language model is that while the sequence of tokens may technically qualify as a sort of language it doesn't appear to be generalized by any reasonable interpretation. Further, the model doesn't appear to be able to handle unstructured inputs and outputs in the "language" - everything seems to be highly structured.
My line of reasoning could be approximately summarized as compiled binaries constituting a "real" (though not natural) language versus a sequential listing of chess positions that represent sequential game actions only being language "shaped".
Still, it's interesting to consider that if scaled up I expect the "repeat yourself" experiment would likely apply to the internal representation of the model in the same manner.
The distinction is meaningful because the process described here isn’t bound by the same constraints, resulting in meaningful consequences.
Suppose we flipped the initial stings and fed that into the process. There would still be meaning to extract from the training set but our new Reversed English but it is not English so the preprocessing step can’t be based on that assumption.
I don't understand what point you're trying to make here. At the risk of being tangential (due to not understanding) LLMs have famously been able to accept base64 encoded natural language as input "out of the box" because (it seems) under the hood they already transparently translate all inputs into their own internal "language" system on the fly. Presumably a similar translation could be learned for input of reverse english (or rather merely adjusting attention) although at a glance it seems like output might not be feasible.
Technically correct, but not in a functionally useful way.
The “L” in LLM’s generally refers to human-language specifically. You’d expect to feed it…human text. Nitpicking that the human text also constitutes a mathematical language is like, correct, but so general as to be unhelpful.
First: this is really great technical writing, especially when you get into the rebuttals. Firm & clear without polemics -- props, and thanks for open-sourcing!
That said; I don't have the time, energy, or anywhere near the expertise to challenge you on the DL specifics, but I feel compelled to add another voice to the chorus of doubters nonetheless.
Using other ARC examples at runtime (effectively, yes?) for "transduction" may not violate what some officer behind ARC said on Twitter --and is certainly a fantastic tool for certain problem spaces-- but it just seems like a glaring and unavoidable philosophical problem in this one. My issue isn't with using the eval set per-se (though that obviously sets off well-justified alarm bells), but rather building an AGI system whose performance relies on the arbitrary size and shape of this particular dataset.
There's a lot of ways to frame this, but given the transduction citations the most appropriate is probably the AI winter's infamous 'Frame Problem':
You say upfront that this works in the first place because ARC has "very few samples... in a high dimensional space"; to me, that seems like an extremely strong indicator that the datasets are not intended to capture anywhere near the full semantic space that we would consider relevant for AGI. If true, your approach would indeed be ""cheating"" by using an arbitrary & unavoidable feature of the dataset (that they didn't have the time or money to craft 100 million high quality cases by hand instead of 1000) to solve the frame problem upfront for you. This would explain why you don't even need a full LLM here -- that's the unsolvable problem that LLMs solve for us.
That is... even if the ARC train+eval sets contain the sum of human intuition between them, superficial differences IRL would render your model unable to identify which examples are relevant to which problems, and thus unable to transductively reason.
In plainer English: surely you'd agree that your model would do worse if we swapped it out with Opus behind the scenes than the next-highest-scoring ARC model would do in the same position, yes? For coding, research, dumb questions, SVG pelicans -- the lot?
If so, that seems like hard proof that this scores high on a benchmark at the cost of the benchmark itself. Like, if this transductive approach leads to ARC1 being claimed (which I thought it was ages ago but :shrug:), they'll either have to abandon the whole benchmark or ban this approach retroactively.
If not... well, I guess I encourage you to try it! It seems like you'd need 1000 truly stellar hand-picked examples to transductively cover that whole space, for one thing.
I think this is a great question. I have some thoughts on this but no hard evidence (neither does anyone else!)
Your argument relies on the AGI system being the model arch + weights. I think that the weights are irrelevant. The training algorithm is what is AGI: You choose/find a training set that covers a task, and then some form of deep learning with a big neural net.
For LLMs (which is a weak kind of AGI), this is freezing a model after NTP pretraining + RL postraining. This allows us to do well on a wide distribution of tasks.
But if we had a scoped task, I think its possible to just do the same thing with (1) a smaller dataset that covers this task + (2) not freezing the model (ie. test time training). This is easy for ARC because the data is small and has been curated well yes, but i don't see why can't this apply to more complex/ill-defined tasks (obv lots of things unsolved to make it work today)
> even if the ARC train+eval sets contain the sum of human intuition between them, superficial differences IRL would render your model unable to identify which examples are relevant to which problems, and thus unable to transductively reason.
In this hypothetical world, the dataset becomes incredibly large, and training on it makes it close to an LLM. You can then finetune transductively and we get the same thing (other approaches already do this iirc)
--
Note: BTW I'm not claiming that this is an AGI system, the benchmark creator also was clear that ARC is not sufficient for AGI (his goal was just to point out unsolved stuff, and the stuff ARC-2 pointed out was demonstrated very clearly by large reasoning models).
In the above work, I just wanted to show that AR transformers without pretraining can perform really well on this benchmark, which was incredibly non-obvious before.
The claims in this argument are separate and I haven't proved them yet
Kind of hijacking, would you say that LLM's have solved the frame problem?
To me, the frame problem is: Can you function in an open vs closed world, and to me the answer is yes, LLM's can definitely function in an open world where the rules are fuzzy, changing, undefined, etc. At the very least, much better than all GOFAI approaches by far.
The issue is now grounding - It can "function", but what would it take to "ground" them? A personality, maybe? Actual consequences? Making them interact only with constrained tools that are formally verified?
Right now it's a combination of harness engineering, and ml philosophers arguing about compression leading to the "objectively correct intelligence", whatever that means.
I think LLM's are "A[x]I" right now in the sense of "they have the capability to integrate with everything" - but obviously you can argue how much this actually reflects "A[x]I" (if you gave someone integration with everything, is that really your success or people handing you it)? But they are still missing some oomph factors that need to be clarified IMO. Maybe it's something as "mundane" as just having actual persistent memory, or maybe it's some deep philosophical thing like qualia. Who knows.
That's true. Again, I currently view LLMs as a function of integration - what they may lack in "intrinsic smarts", whatever that means, they can tool call and we build capacities (and they build capacities!) around them and to some degree can reason and be creative.
I do think the first claim has real merit even if it's not 100% on par with humans. Second claim is just true.
I'm glad you did! Of all the procrastination techniques I have mastered, engaging smart people on HN about artificial cognition is probably one of the more useful ;)
Apologies in advance for the diatribe(s) -- I think about this stuff a lot.
...would you say that LLM's have solved the frame problem? To me, the frame problem is: Can you function in an open vs closed world, and to me the answer is yes, LLM's can definitely function in an open world where the rules are fuzzy, changing, undefined, etc.
First, a nit: I would describe your definition as a valid transformation (isomorphism?) of the original phrasings, which were about technical context and epistemological belief[1]. I mention this because A) (semi-)symbolically providing context to LLM calls is the challenge at the core of harnesses, routers, pipelines, 'orbs', and a long list of other marketing terms that must amount to an ∞-B\$/y industry by now, and B) it shows how arbitrary the phrasing was, at the end of the day. (I also prefer this to the wiki article btw, for the curious: https://plato.stanford.edu/entries/frame-problem/)
My actual answer here is a resounding "yes" and "no" at once, in the exact same way that the Turing test is both so obviously surmounted in 2023 (post-RLHF) to anyone applying 20c standards, while also somehow being so far away that we're not sure it'll ever be possible.
The key is to 'dissolve the binary' for both, if you'll excuse the phil-ism: Turing's 1950 paper Computing Machinery & Intelligence was never intended to prescribe some yes/no evaluation procedure, and the people frustrated by the Frame Problem were not worried about a single yes/no "Frame Test", either.
Instead, Turing settled on behavioral comparison on an intuitive, human level as the best shared dimension to test, but only after calling Ed Zitron "absurd" and leaving room open for ESP & ghosts to end up proving souls (one of those is literally true, the other only figuratively).
By this metric, current LLM-backed agents are clearly able to behave like a reasonable-ish human over a long-ish timeframe -- that's just objectively an incredible achievement IMO, even from 2015 standards. The promised inversion is the retort that invites, namely: the '-ish' makes all the difference! An artificial mind that behaves in completely alien ways randomly is a much less useful tool even if those events are rare; ditto for an artifical mind that loses coherence across """mere""" days.
I'm cutting this as much as possible, but hopefully it's clear why all the above applies to the Frame Problem, too -- just replace 'behavioral' with 'epistemic'.
The issue is now grounding - It can "function", but what would it take to "ground" them? A personality, maybe? Actual consequences? Making them interact only with constrained tools that are formally verified?
I think your use of 'ground' is nice and understandable, but is conflating too many things to work as a summary of the remaining work. All of the things you mentioned are absolutely being explored --both by scientists and by highschoolers collectively speedrunning 76 years of science live on HuggingFace to generate the best uncensored model for their polycule's DnD campaign-- but they hinge on distinct metrics.
For example, the last one deals with reliability, which is closely related to the "randomly alien" stuff I mentioned earlier.
"Actual consequences", OTOH, most directly relates to the camp(s) focused on "embodiment", which is basically the idea that truly human intuition is too spatial to reasonably emulate without the ability to experimentally interact with the world -- AKA the "AI needs robots" camp.
And finally, the "personality" bit... I personally think we have to rediscover the subfield of Affective Computing, but the closest lane so far is "Constitutional AI", an approach popularized by Anthropic that (wisely) just moves the whole problem over to the world of prose and optimizes it from there.
All three are important steps indeed, but I think deserve finer delineation than ~'does it connect the agent to the real world more/better/stronger/truer'.
Right now it's a combination of harness engineering, and ml philosophers arguing about compression leading to the "objectively correct intelligence", whatever that means.
Ha, totally agree on the distaste for intelligence as a single dimension. I will also say that 'harness engineering' is gonna end up being an outdated term for 'the rest of AI' over time, MMW. A more (in)famous voice beating this same drum is Gary Marcus (I know!) under the term 'neurosymbolic' (?).
they are still missing some oomph factors that need to be clarified IMO. Maybe it's something as "mundane" as just having actual persistent memory, or maybe it's some deep philosophical thing like qualia. Who knows.
Again, you have great intuitions here... I think it might help to consider how the human capacity for memory is simultaneously mundane and profound, at different levels of analysis.
I think this situation is similar: we're not gonna need to invent Memory 2.0 (and can't, probably?), but there's a long list of human-specific heuristics, control planes, and other neural machines of some vague character that must exist, only a teeny tiny portion of which have been explored by "harness" engineering as of yet (for the best, probably...)
TL;DR: We're not through the Kuhnian paradigm shift just yet -- the new episteme has far from penetrated all the subfields of cognitive science, IMHO. Predicting the landing point feels a little pointless, for that both that reason and an even bigger one: if RSI ends up being realistic (which it very likely is for our 2026 human computers, to some significant extent), this is all just the anteshock anyway. As "the singularity" implies, that kind of exponential shift could really take us anywhere (or nowhere, forever).
P.S. Never done this before, but fuck it: I'm currently seeking exciting remote work ASAP -- if you found this interesting, please consider this my cover letter. Sorry mods if against the rules, but, y'know... one-time exceptions for the singularity?
Hey thanks for sharing this. Was curious did you find the more you trained the model the more perf improved, or did it start plateauing. For example, let's say you didn't spend 67 cents, but you spent 67 dollars do you think you would get major benefits from that?
- it feels logarithmic (like most perf-compute graphs), and eventually plateaus. 44% @ 67 cents was a good stopping point for me
- more compute would require a lot of effort and dealing with new problems like training stability, cost of iterations/sweeps (didnt have the money to convincingly run larger iterations)
I've been interested in training a transformer from scratch for the same learning reasons. The GPU cost/availability seemed prohibitive to do anything useful but you seem to have flipped that on its head. I love your outside the box approach.
in the future, for general perf, I am optimistic that someone will figure out an alphazero like approach (ilya/silver/sutton/carmack seem to be working on something like this)
I think you’re asking the right questions, sample inefficiency is horrible in modern LLMs. Despite this, I saw your analysis:
> The biggest increases in scores were due to
Modern architecture (SwiGlu instead of GELU, RMSnorm not layernorm, etc.)
More data diversity, better shuffling of data
scaling up: 8 layers instead of 4
This is commonly called squeezing the lemon and is usually a bit of a last resort. You should be able to achieve near SoTa with your new method, before you squeeze any lemons. This is, because the old SoTa is typically not using new optimisers and thus your results will be distorted by a large margin.
In terms of sample efficiency I want to add two things:
Runtime per-puzzle fine tuning is a very good target that provides a LOT of information.
People have not looked at evolutionary methods to harness induction since the 90ies - if I was to work on ARC ever again I’m fairly certain this is where I’d look.
I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop?
> Training on the eval puzzles is cheating / “training on test”
> No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.
> Also, ARC is a metalearning benchmark, so you’re supposed to learn from the eval puzzles.
> Jargon: ARC has a set of train puzzles and a set of eval puzzles. Each puzzle has example pairs and test pairs. A pair consists of an input grid + output grid.
> The ARC, the label is only the test pair’s output grid in an eval puzzle.
> These labels were not trained on. They are hidden. You can delete it beforehand if you wish
I think what I gather here is that the test comes with one batch of training problems, which everyone agrees you can train on. But maybe the eval problems also come with input/output examples (to help define the problem) and training on those is controversial? I can’t see why it would be controversial but is that the criticism?
The point of ARC is essentially an "IQ Test" for AI systems. It is meant to cover abstract reasoning capabilities of generally-intelligent systems like LLMs. What the author did here was build a system that only solves ARC problems.
The other tension is the fact that this score is on the public eval set. In machine learning, you typically have 3 datasets: training, evaluation, and test. The training set is the dataset that's used to update the weights according to your loss function, you are "encoding" the patterns from the training set directly into your model. The eval set is what you use to track performance while training, it is NOT used to update model weights, but shows how well the model generalizes. The test set is a private holdout set that is only used when you're "done" developing your model. The difference between test and eval is information leakage: you can use performance against the eval set to modify your hyperparameters and model architecture to get better eval scores. So while the eval set doesn't directly update the weights, it can indirectly cause "overfitting" by tailoring your model to do well on the eval set. What you really want to see is the private test set performance, not the eval set. For all we know, this model could be ridiculously overfit on the eval set and perform poorly on the private test set.
1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, instead of being frozen after training
2) overfitting: agree that true measure is private set. It scores on private set roughly on par with TRM (a comparable model), obv with much lesser compute
--
also re LLMs: they do not follow this 3 split since (a) Incredibly hard to keep a pretrain dataset clean, (b) common in labs to benchmaxx during postraining (and known to do so on ARC)
> weight update during eval: this is a form of test time training and not really cheating.
Possibly "not really cheating", but it does make benchmark comparisons unfair - especially as the other models are unlikely to have their weights updated during the eval.
I am very new to this but applying human intuition this still feels like cheating. Knowing all the question that will be on the exam and working on understanding them even if you are never given answers will obviously give you and edge.
what you describe would be cheating. My approach is the opposite. What I did was "You are born during the exam, given access to a training set and the questions then learn from scratch during the exam"
On the private test set, the right way to evaluate this type of model, is giving i it the test question Q, which it will first train to AR predict first, and then it will inference using the just-updated weights with Q as prompt, giving you back A, and then you compare A with A_true secretly.
It seems like an interesting strategy. Based on the author’s comment, they haven’t been at it for very long. So, I guess the folks who run the private test haven’t had a chance to get to it? It’d be interesting to hear how it does.
Does ARC measure "one shot learning"? I heard that the major unsolved problem in ML was developing systems that are good at dealing with novel problems.
What you gather is correct, assuming by "the test" you mean the ARC benchmark in general. It was controversial because people are used to LLMs which are frozen at train time, where the eval problems are usually not trained on for various reasons like fragility (basically porridgeraisin's ans which is great)
Here's another explanation. Take the train dataset and test dataset of a benchmark
Train: {x_i -> f(x_i)}, Test: {x_j -> f(x_j)}
As long as f(x_j) in the test set is hidden, there is no "training on test". In a normal benchmark, each x_i is a single datapoint. But in metalearning benchmarks like ARC, x_i is the puzzle itself that has a train set and the test questions within it, hence the confusion and controversy
Basically, you have a bunch of Q,A pairs in the training dataset. Here, it was trained to next-word predict the question itself, as well as next-word predict the answer given the question as prompt. This is bog-standard, no one's complaining.
In the test dataset's Q,A pairs, it was only trained to next-word predict the question itself, and it was not given the answer at all.
It was then evaluated by seeing if it is able to output A_test given the Q_test as prompt.
What would be cheating is training it to produce A_test (given Q_test as prompt) as well, since then you can always make a model that scores 100% by just memorising Q_test, A_test pairs.
The complaints online mostly stem from not reading that properly and assuming they trained on Q_test,A_test instead of just Q_test. This is further because these days large LLMs are inadvertently trained on many benchmark solutions even unintentionally due to the massive scale of data and the infeasibility of auditing it all. But none of that is the case here.
The reason you want to train on Q_test is because in these AR transformer models, they learn useful composable encodings of Q by simply learning to next-word predict Q. So you enable the model to learn composable encodings of the test questions, so that it can hopefully "connect it" to an earlier train problem it had seen, and adapt the solution it had seen for that, much like humans do in school exams.
Without this step, you are making it difficult for the model to "connect" the test question to a train question it had seen earlier, and then it still has to adapt the solution. This way, you precompute that "this test question is like this train question" and then during the exam you only have to do the adapting the solution part after a simpler "retrieval" process.
You can just think of next-word training Q_test as a "retrieval" process.
This practice often used in continual learning or "test time training" is not yet useful in general real world ML tasks due to the differences in memory and compute requirements, and more so the general fragility of training large neural networks in a streaming realtime way (as opposed to large data, batched), versus inferencing from a static neural network. It is due to that fragility that I believe (correct me if I am wrong) this guy had to train on a batch of Q_tests. If you enforced that you will not provide Q_test_2 before they answer Q_test_1, the performance will drop.
While the increased compute and memory is difficult to solve inherently, there are various efforts being made to fix the fragility, especially in reinforcement learning where this is called "streaming RL", there is revival of interest as seen in RLC 2026.
[Note]
Arc-AGI-1 doesn't have any actual english words or such, but it's simpler to pretend it was a basic Q&A benchmark to explain the above
Further question—the model produces an answer to the question, it sends the answer, and then gets graded. Does it get to know immediately how it did, or does it get the grade back at the end after answering all the questions?
If it is the former case, it would be possible to add the generated question/answer pair into the training set as well. Would that be considered fair? (Of course this is a moot point if the answers all get graded simultaneously at the end). Then the model could explore interesting strategies around what order to answer questions in.
In my uninformed opinion, the various permutations of question ordering/answer revealing all map to different real-world scenarios… and any of them could be interesting!
It does not, if it gets the answer (or any information about them, even % of qns solved) and is able to adjust itself in response, then that is considered training on the test set and is wrong.
Even cooler is his about me mention of saving his own life https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)
Yeah but rhabdo is something literally any e.g. body builder, power lifter, etc could tell you about. Actually if somebody knows what hypertrophy is, they probably know what rhabdo is. It's a pretty normal and big concern in any sort of high intensity weight training.
I can't think of many ways that otherwise healthy and fit younger people can physically nearly kill themselves doing normal activity, so it kind of stands out - let alone it being not all that rare either. Rhabdo has even gone viral in the news like when a while back a couple of Chinese girls nearly killed themselves doing a social media 'squat challenge.' They did 1000, got rhabdo, didn't know what was happening, ended up in the ICU with kidney damage.
It also manifests in other ways too. For instance I had an elderly family member give himself rhabdo during a manic phase he was going through when he started going wild on construction and other physical tasks that were way beyond what his body was ready for.
Basically it's not some super obscure thing you'd expect only a good doctor, let alone a specialist, to know about.
there's a very large variance in doctors' abilities in India. At the very top they are close to the best in the world, esp with an insanely high workload.
but on an average, not great
Also, a lot of gymgoers and physical trainers I know hadn't heard of rhabdo either (and this is a relatively wealthy part of a tier 1 city)
I studied biochem in undergrad and my classes were full of premed students.
I loved the subject and nerded out about the course material - I spent my time designing my own experiments around gene cloning that took several semesters to run. They were sharing last year's tests with their frat buddies and laughing at us nerds.
I've never looked at doctors the same way again after college. I looked up to them as a child, yet after seeing how the sausages were made, I started to doubt everything.
I frequently ask doctors, who spend all of ten minutes with me while the nurses do all the work, about the molecular specifics of what they're talking about. They talk down to me as if they're explaining to a child, yet they're frequently quite wrong. I'm not trying to sound superior to them, but I'm shocked they seem to care so little about the subject. It doesn't give me much hope about what they know and their abilities or competency.
I suspect surgeons and specialists are a different breed and aren't like this at all.
And to be clear, this isn't everyone. But it does seem to be the majority I've interacted with throughout my life.
When they act disgruntled at patient interaction, I detest that their profession tries to cap the number of med students per year. We should be letting in as many med students as we can take. We should let doctors from overseas immigrate and easily become practicing doctors here in the US. We should provide easy paths for nurses to become doctors.
The premed students in my university were chiefly concerned about money and prestige. They drove BMWs gifted to them by their parents and laughed at what I drove and how hard I studied. I had to put up with their bullying for years. I know not everyone who studies to become a doctor is like that, but it permanently skewed my view of their profession.
> people entering the field specifically to make big money goes down...
This is a self defeating argument, because for it to work it requires that there is a larger number of people who love the field than the number of people who are in only for the $$$ (or else there will be even more scarcity of doctors)..
We have a surplus of people that want to become doctors and would work hard enough to do so. If we removed the artificial limits on residency, the supply of doctors would increase, which lowers wages somewhat, which lowers the number of people that want to become doctors. It reaches a new balance point with more doctors than before.
That itself is a good thing.
For it to also improve the motivations of doctors, all that needs to be true is that someone more motivated by money is more likely to be in the group that switches career.
It does not require the number of people that love the field to be a majority. Not that loving the field is a binary in the first place. People are a mix of motivations and cutoffs are arbitrary.
> It reaches a new balance point with more doctors than before.
But you just pull this "more doctors than before" out of your ass. There is no such assurance. If less people want to become doctors, then eventually there will be less doctors, not more.
I didn't pull it out of my ass. It's basic logic that if you remove a supply restriction then the supply goes up.
Let's attach fake numbers: Right now only 50 people can get residency each year, and 80 people starting college each year want to be doctors and would work hard enough. If you let everyone get residency, at first you'd make 80 doctors per year. This would decrease doctor pay and then only 75 people would want to be doctors, then 70. But if this ever dropped back near 50 for long, wages would spike back up, and the candidates would hit 80 again. It stabilizes in between, at 63.
With a bottleneck at X, and potential supply at Y, removing the bottleneck gets you a number between X and Y. All else equal, it can't stabilize below X. There will not 'eventually' be less doctors.
Alternatively don't even remove the bottleneck at first. Increase the residency bottleneck to 60. The number of potential doctors drops below 80, but it stays above 60, so now you have 60 doctors per year and the most money-motivated took a different major.
TL;DR: Let the number of residencies rise and the number of people that want to be doctors lower until they meet in the middle. Because residencies decide the actual number of doctors, meeting in the middle gives you more doctors.
We already know that when the doctor production is 50, the number of potential doctors settles at 80.
Supply and demand means any lower doctor production causes even higher wages and that draws in even more potential doctors.
By what mechanism would it settle at 30? In particular, if it would settle at 30 despite an even harsher doctor shortage, why is it not already at or below 30?
To the parent posters credit, he's very honest that he developed a personal complex against doctors when he was a poor student. He just sees it as a way to lash out at American doctors, the irony being that foreign medical graduates leaving their families and communities to practice in America largely do so because they are exceptionally money motivated. That doesn't make them bad doctors, but there's certainly less likelihood they're doing it purely for the love of medicine or a desire to care for their communities.
I do agree with the sentiment though that the US needs to fund more residency slots as it's an asinine professional barrier, and that we would benefit from more physicians coming from more diverse financial backgrounds.
> certainly less likelihood they're doing it purely for the love of medicine or a desire to care for their communities.
Maybe self preservation?
In east europe, public hospitals will force doctors to work 36 hours shifts (overnight ER with theoretical sleep). Doctors have a full criminal liability for mall practise.
I've been on statins for years, and I don't remember anyone talking to me about rhabdo. To be fair the education I received about my medications was a firehose of information after a heart attack and major heart surgery, so perhaps it's possible I missed a few things.
> I agree that its rare to see to face problem sets in real life where every problem is given at once. Even if it is (like an exam), humans can usually only attempt one at a time
Just one small snippet that I thought was interesting. I would always read through ~the entire exam before starting. Both so that I could find the problems most approachable to me, but also because sometimes it helps me figure out the rest of the questions :-)
> Ban offline training/pretraining. Models must train from scratch after submission
Previously this was considered impossible so rule. My model shows this is possible
Guarantees no synthetic data can be used
It makes the comparison fair across differet models. Otherwise some models like LLMs can benchmaxx ARC by using ungodly amounts of offline training. (Since the benchmark has been around a long time, many ARC-like datasets have been created)
I'm not an ML researcher, so YMMV, but... how could a model learn to answer these ARC-AGI questions without training beforehand?
Predicate logic is trivially realized by linear transformations (I.e., matrices), and these matrices are easily discovered via gradient descent with appropriate reward functions.
> I don't think a baby or even average kid could solve these
The reward functions of a typical baby or kid is not 'get a huge dopamine boost when you solve a logic puzzle' (or whatever neurotransmitter, I don't know).
> Also, I’m not sure whether “general reasoning” even exists in the first place? Maybe humans are specialised too
I have been wondering the same. We are now exposed to so many stimuli, we are tricked into thinking this is the norm - to have a reasonable understanding about everything, unless specialization is called for.
> The mentioned approach is fundamentally flawed, since the inputs are used during pretraining constituting to a leakage, a universally recognized flaw of ML training.
I saw this on the community note for the last blog you wrote - anything to do here.
>Training on the eval puzzles is cheating / “training on test”
No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.
I disagree, but i agree that training with answers is worse.
In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those questions. Sometimes it was the same exam sometimes they were 'slightly' different. In no occasions were the answers shared, and being math exams it wouldn't have made a difference, since solving the exercises is the actual training that allows replication of results and adaptation in testing.
It's a grey area for sure, but it's a quantitative matter, studying 20 different exams for 2 months is quite different than trying out 1 exam 1 week prior to an exam to verify all is well.
It's called teaching to the test, not teaching to the test and answers. In essence OP holds a naive version of what cheating is, and thus they think they are absolved, when actual cheating is much more nuanced. Many such cases.
Fwiw, the second uni I dropped out of was worse in that some students just used their phone during tests and talked with each other or googled. OP sounds like a student from Uni 1 claiming they don't do what Uni 2 students do.
And for reference, the exams I did pass I did by just reading the whole bibliography on my own, and doing exercises from the book if needed, I was passing with like 80-90%, never did I have to get a copy of a previous exam and study that, I think it's a ridiculous concept that has been normalized to meet an increasing societal pressure on everyone being an elite graduate (we can't ALL be elite), and if this repo is successful, it's because this attitude is so normalized that it's seeping into machine learning by diffusing the lines between training and testing set, and increasing the ratio between one and the other.
Hell, I'm not surprised that the software that the mass of test
-studiers develop is software that studies tests. In the same manner that the software that cheaters develop is software that cheats Guardrails and breaks ToSes
> In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those question
Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"
What you described is cheating because more time than the exam permits. ARC was designed specifically to avoid this. The exam in question (kaggle competition) is 12hrs long with 4xL4s. I trained on 1.5hrs with a single 5090 (which converted to 4xL4s is slightly longer, but still within 12hrs).
(There's also access to experts who know the answer, which kaggle bans by banning the internet)
Your arguments btw support my work over the LLMs more. LLMs today are postrained with a large amount of synthetic ARC data. (Exactly the "teach to test" criticism). Thats why they perform so well on ARC. Base models still are terrible at ARC-2
I didn't phrase that well and cant edit, so clarifying:
What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"
>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"
>What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"
This is getting into metaphorgotten territory, I'd have to do a deep dive to really understand whether it is "cheating" or, more formally, a methodology that encourages overfitting.
But for what is worth, taking the textbook to the exam (regardless of whether you were born there or not), would be cheating in a test as well. Although it is possible that in the model training context it does not lead to overfitting, it certainly doesn't preclude it.
>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"
It is comforting that someone agrees, to this day this is considered a gray-area (or not even that) by students, graduates and those that look up to the University of Buenos Aires institution.
in non metaphor terms: In many ML situations you can carry the train set with you test time. Eg: KNN, SVM, replay buffers, etc
this is one such case
--
the separate overfitting concern is fair, look at private holdout performance for that. it performs on par with TRM (a comparable model) in the private set, ofc with far less compute
I think you can do whatever you want with your train set, what would be concerning is training with the test set.
I don't think overfitting is separate, the consequence of putting the benchmarks in your training set is not that you "cheat" by breaking some moral code, it's that it breaks the purpose of the benchmark and trains your model to be good at that benchmark only, instead of being generally useful.
> Increases in LLM scores are now mainly driven by post training (evidence in next section) and are probably a function of amount of synthetic data. They are learning to solve ARC tasks, not learn general abstract reasoning
Agreed and that's for any benchmark. Private tests are better but you still have to trust the provider to not log and use them for training.
That's why I like when a new set of tests like a new ARC-AGI version is published, that's where you can see which of the models abstracted to more general capabilities instead of being focused on the previous tasks. Most models completely fail new ARC-AGI tests.
The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. You hit a ceiling very fast and investing into more compute will give you diminishing results. So yes, you can train a custom model to do somewhat decently on a specific set of tasks but then what?
> The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results.
I think thats unfair. Perf-compute is often logarithmic and will always saturate . Reaching the plateau faster is valuable as it often leads to better peaks (held true here and also look at modded nanogpt)
And more compute increases the perf (after dealing with other scaling problems)
> The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results.
Nothing about saying that it cost 67 cents implies that it will. Knowing only that it costs 67 cents you also have no reasonable basis for extrapolation. It doesn't indicate a trend whatsoever.
I don’t have any kind of ML background but I have always thought of sample efficiency as the great unsolved problem of AI. We humans have unbelievably good sample efficiency; often we can durably learn something on just a single example or two. This is the main area in which LLMs are vastly, vastly behind us.
the caveat is that we are not learning those small number of samples from scratch, since we're coming in with a large amount of training already, much of it from before we were even born
I've seen a few videos about people having their girlfriend/dad/etc play very simple video games for the very first time. It's a really great example of this. The most basic, simple, mechanics are difficult for them, and take surprising repetition to remember, because they have zero framework around any of it. It's really similar to watching a genetic algorithm learn to play Mario, but on fast forward.
No, the unsolved problem of AI is continuous learning. We never stop learning, we don't have a "training phase". You are always updating your world model even when you sleep. Also more quality training data does lead to greater learning efficiency as you have more priors to work with.
There's an updated ARC-AGI-1 chart with 5.6 Luna in each thinking level in this video from last week:
"A New Architecture [..] | MOONSHOTS " https://youtube.com/watch?v=qQfUbo7Ldc0&t=2m5s
Isn't this cheating? Or rather, are frontier agents only looking at one question at a time? If I understand correctly, you're looking at all the examples of the exam questions. If the exam was adjusted so that you can only look at one question at a time, you won't get 44% anymore.
Why would that be cheating? That's what humans do when they learn, they look for the signals and patterns that reduce the possible set of answers so they can converge on the solution and narrow the search space.
I mean, I'm interested to know if the frontier models also get to see all questions at once. Then it's more fair game than if they just see one question at a time.
Really impressive and creative research. I wonder if the leading labs do anything similar with their models? It doenst look like the open source labs do?
It means the questions including their answers are dependent. Ie, theres a data generating process for them, that the model uncovers. Like a KNN is known to have near Bayes accuracy as k/n to 0, n to infty, k to infty. The data reveals the dgp.
I suspect if you feed unrelated or even garbage questions into the eval set, it would reduce the performance.
I know that this is Transformer, and not LLM per se. But isn't this the same idea that PaulG said the other day, and many of the comments criticized him?
My above comment wasn't aimed at you. Thanks for the response though! What you did is phenomenal, and I wasn't taking a dig at you in any way.
What I intended in my comment was that, may be starting from scratch (like you did many months before, and what PaulG suggested recently) is the way to go for future job prospects and startups. Most of the comments in the thread I posted was negative for PaulG saying that.
So this achieves 44%, I'm wondering: To consider yourself a reasonably competent human, what % to achieve on ARC-AGI-1? Thanks in advance to anyone that knows, can answer!
@mods I am not sure the editorialized title is better! (It is a bit clickbaity, since this is a transformer that does not compete with LLMs at all outside the arc-agi-1 benchmark)
"I don’t understand why others didn’t figure this out"
- how about we allot the possibility that so many of presumed ML experts don't have any clue what they be doing, and are eventually API bitches, nothing more.
Is the author only running their model against one benchmark? I don't think anyone finds that difficult to achieve, the difficulty comes when you want to make the model not benchmaxxed to a specific benchmark, and generalize so it can solve problems not part of the training data, but seems this model is specifically for not this? How useful is that?
If you just wanted to pass these specific tasks in this specific benchmark, and wanted to do so cheaply, I'm sure a non-LLM-based approach would yield better results for even cheaper, since what the author's model does, seem to basically be "solve ARC puzzles", not a general LLM or "coding" LLM.
I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture. A lot of people have sort of forgot that machine learning is more than just LLMs these days.
> He is showing computers can solve these issues, without using an LLM architecture.
Isn't it a LLM he's building though? My very point is that this particular use case could be solved better without building a LLM, now you claim he is not? The description of what he's doing surely makes it sound like it's a (very small) LLM, and personally I'm still on the "if it quacks like a duck" train in life.
> A lot of people have sort of forgot that machine learning is more than just LLMs these days.
Yeah, which I guess if you make my previous comment more concise, is exactly what I state too.
> Nowhere does he say he built an llm. Hes using a transformer, not an llm.
Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results".
Obviously it's not a Large Language Model, but to me this looks more like a LLM than not, given the architecture he's chosen. But again, maybe I misunderstand?
AFAIK, the “large” qualifier came when transformers allowed to scale the size of language models compared to the recurrent models that where in fashion before. And although BERT isn't large by today's standard, it was large enough for the time.
idk the definition is fuzzy. thats why people use the "modern" qualifier to talk about decoder-only style and this is also not clean since you now have reasoning models which are separate
A LLM should at the very least be a language model, i.e. be able to take human-readable text as input or produce it as output. Transformers are used for plenty of tasks that don't involve language, for example object detection or blind source separation, where the models aren't called LMs; and on the other hand there are some LLM architectures that exclusively use linear attention variants and aren't really transformers anymore.
Transformer solves a Seq2Seq problem just like RNNs. All Seq2Seq problems need not involve a language. In this case teaching on ARC puzzles doesn't mean what he trained is now trained on a language which will be English(or any other language) in this case. So, does his training successfully models "English as a language" -> No. This implies it is not "Large" and has not modeled any "language".
Just to be clear: It was well known that you can reach such scores with small models and without an LLM if you train on the task. The author highlights those models himself - e.g. HRM/TRM.
The novelty is more that it works with such a plain transformer and low compute price.
> The whole point of his model is to optimize for a very specific benchmark.
But benchmaxxing is what we generally try to avoid for training, as there is no point really for it. We used to call it "overfitting", now you're saying this person does it intentionally? Why?
There are plenty of applications where a machine learning system needs to optimize for a very limited data set that is still intractable by linear logic systems of reasonable scale and complexity. It’s interesting, because he is using the legos of LLMs to build highly specialized machine learning systems, which is a very pragmatic approach. Obviously a lot of other ways to achieve similar goals, but it’s cool to see someone back porting the modern tools towards older style optimizations.
Also, the complexity of the task he is using occupies an interesting middle ground of ultra high dimensionality (for a “simple” problem) while being limited in width to a narrow set of solves- a space where one would be tempted to imagine you would need a much more capable system.
Overfitting, as well as the specific instances I've seen of the word benchmaxxing, involve knowing the answers and training to those answers. That did not happen here. The model is limited in scope, which means it's not being scored on generic intelligence, but neither is it defective and terrible at solving new problems inside its scope, like you get with overfitting.
Hi! Author here. Surprised to see this on HN now. Happy to answer any questions!
Some context about this:
- This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs
- Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost and w these few training samples.
- Sample Efficiency is one of the most important unsolved problems today in AI. That's what I was targetting with this work. We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible (also why OpenAI's Parameter Golf had fixed compute and why Modded NanoGPT is considered very sample efficient)
- Can the perf be improved? Yes but the competition is ongoing so can't talk about it
- Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho
- Fun: I was new to ML when I posted this first (dec '25). I basically used ARC as a way to learn ML
"- Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho"
Also, in curating the training data in a deliberate manner, with attention to detail. Most people just use existing datasets and call it a day. It's a lot of work, which is why there are gains on the table.
Thank you for this excellent post series. It reminds me a lot of the pre-LLM days, though I was mostly using LSTMs back then. When the original GPT paper came out, I thought the future would be using LLMs to generate tons of synthetic labeled data and then training specialized LSTM or transformer models per-task.
Had a couple of questions:
1) You note that ARC-AGI is a meta-learning task, have you tried any meta-learning algorithms such as MAML?
2) Do you think this approach could extend to ARC-AGI 3? Or do you think the interactive environments require a higher level of complexity than what can be achieved with a small model?
Glad to know you like it!
1) Unfortunately I didn't. I was v new to ML when I did this and didnt have time or skill to try many things. Will try them when I get some time!
2) Possibly, but it would require significant changes and effort. But much larger models would be required imo (must have capacity greater than the complexity of the problem)
I spent some time working with that approach of using LLMs to generate synthetic labeled data for use in training more specialized models. It mostly didn't work.
The problem was that getting the LLM to generate training data that sufficiently resembled real-world data was labor intensive and expensive. More labor intensive and expensive, it turns out, than just using real data.
What worked better was using the LLM to label the training data. But even there we had to be careful about introducing weird biases.
>> NOT an LLM. its a small ar transforme
Super cool project! Though, aren't most modern LLM's ar transformers internally?
Pretty sure it's one of those "All squares are rectangles but not all rectangles are squares" situations. Transformers are what really started the LLM Boom, and seem to be crucial to the technology. They also have other applications, such as what OP created
It gets extremely blurry, because people commonly refer to any model that uses a component associated with the Transformer architecture as a Transformer (i.e. using some kind of QKV-esque attention mechanism). I think it's easier to think of it like this:
A large language model is just what it says--a very large statistical model trained for language tasks. This covers the spectrum of GPT-style models, but also those hard to classify ones, like Liquid's "Liquid Foundation Models", which can get up to 24 billion parameters and use grouped query attention, but are closely related to state-space models as well: https://huggingface.co/LiquidAI/LFM2-24B-A2B
Also, as others have pointed out, a Transformer isn't inherently a language model. So really they're sort of two different axes, one classifying the model size and task, the other referring to a specific architecture.
In my unpopular opinion, it wasn't transformers or attention, but pretraining on language data that kicked off the LLM Boom. Alec radford in his little jupyter notebook trained a very small non-transformer to predict simply the next-character on Amazon reviews. He noticed emergence of a neuron which when toggled controlled the sentiment of the remaining text. This is published as the sentiment neuron paper.
Till then, transformers were being used primarily for stuff like translation and such and no one was even pretraining at scale, even tho transformers and attention existed.
Openai and google if you count T5 persisting with pretrained generative models was what led to the LLM boom. Yes they used transformers, but that's just one IMO minor aspect.
Not all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.
Also, not all language models are transformers. You can have language models based on diffusion models or state-space models, or any other model that can be used to model sequences (so all of them, as sequences are just functions). Transformers are just the ones that are most common and successful today.
In this case, what are the tokens?
9 color tokens + 4 special tokens (start, end, newline, inp_out_sep)
[flagged]
> The distinction is meaningless IMO.
Not when discussing whether it's an LLM. The second L in LLM does have a meaning.
I'd argue that meaningful sequences of symbols constitute a language. This example doesn't use a human language but it does use a language IMO (at least AFAIU).
Are number sequences a language?
Are rectangles squares? I can express a passage written in english as a sequence of numbers (ie tokens).
[flagged]
You're the first! Congratulations.
Sure, and those aren't LLMs?
[flagged]
I don't understand what you're saying? An LLM is a transformer model trained on a large corpus of natural language, often with some post-training. An image model is a different type of transformer model. What's controversial here?
Words and symbols alone don’t make a language.
Zipper and and? and coin is pimple, Ocean! run bumper.
Your example is a meaningless sequence. So consider a different scenario where the sequence is meaningful but does not map to any human language. What exactly disqualifies it as a non-human language?
When you encounter a human language that you can't personally read presumably you don't proceed to claim that it doesn't constitute language on the basis of your own lack of ability.
To come at it from a slightly different angle - does compiled binary code count as a language? If not, why not? (I'll suggest that it's a language albeit not a natural one.)
Any sequence could be meaningful or meaningless depending on the grammar involved. Open a word document in photoshop and the program has no idea what it’s dealing with.
The reason the language term in LLM is meaningful is how the training, symbol mapping, etc is designed around human languages. The model doesn’t process raw text, instead there’s a critical processing step which allows the magic to happen.
Is the same processing step not happening here?
To my mind the argument against this model qualifying as a language model is that while the sequence of tokens may technically qualify as a sort of language it doesn't appear to be generalized by any reasonable interpretation. Further, the model doesn't appear to be able to handle unstructured inputs and outputs in the "language" - everything seems to be highly structured.
My line of reasoning could be approximately summarized as compiled binaries constituting a "real" (though not natural) language versus a sequential listing of chess positions that represent sequential game actions only being language "shaped".
Still, it's interesting to consider that if scaled up I expect the "repeat yourself" experiment would likely apply to the internal representation of the model in the same manner.
The distinction is meaningful because the process described here isn’t bound by the same constraints, resulting in meaningful consequences.
Suppose we flipped the initial stings and fed that into the process. There would still be meaning to extract from the training set but our new Reversed English but it is not English so the preprocessing step can’t be based on that assumption.
I don't understand what point you're trying to make here. At the risk of being tangential (due to not understanding) LLMs have famously been able to accept base64 encoded natural language as input "out of the box" because (it seems) under the hood they already transparently translate all inputs into their own internal "language" system on the fly. Presumably a similar translation could be learned for input of reverse english (or rather merely adjusting attention) although at a glance it seems like output might not be feasible.
[flagged]
LLMs also don't use words
Yes, but they're models of _language_ - words in, words out. The fact that they're embedded to vectors does not change this.
This is incredibly pedantic if you ask me.
[dead]
It's an slm
There is no language in the training of this, so there is no l.
In computer science, that is technically a language. A formal language if you want to look it up on Wikipedia.
Technically correct, but not in a functionally useful way.
The “L” in LLM’s generally refers to human-language specifically. You’d expect to feed it…human text. Nitpicking that the human text also constitutes a mathematical language is like, correct, but so general as to be unhelpful.
First: this is really great technical writing, especially when you get into the rebuttals. Firm & clear without polemics -- props, and thanks for open-sourcing!
That said; I don't have the time, energy, or anywhere near the expertise to challenge you on the DL specifics, but I feel compelled to add another voice to the chorus of doubters nonetheless. Using other ARC examples at runtime (effectively, yes?) for "transduction" may not violate what some officer behind ARC said on Twitter --and is certainly a fantastic tool for certain problem spaces-- but it just seems like a glaring and unavoidable philosophical problem in this one. My issue isn't with using the eval set per-se (though that obviously sets off well-justified alarm bells), but rather building an AGI system whose performance relies on the arbitrary size and shape of this particular dataset.
There's a lot of ways to frame this, but given the transduction citations the most appropriate is probably the AI winter's infamous 'Frame Problem':
You say upfront that this works in the first place because ARC has "very few samples... in a high dimensional space"; to me, that seems like an extremely strong indicator that the datasets are not intended to capture anywhere near the full semantic space that we would consider relevant for AGI. If true, your approach would indeed be ""cheating"" by using an arbitrary & unavoidable feature of the dataset (that they didn't have the time or money to craft 100 million high quality cases by hand instead of 1000) to solve the frame problem upfront for you. This would explain why you don't even need a full LLM here -- that's the unsolvable problem that LLMs solve for us.
That is... even if the ARC train+eval sets contain the sum of human intuition between them, superficial differences IRL would render your model unable to identify which examples are relevant to which problems, and thus unable to transductively reason.
In plainer English: surely you'd agree that your model would do worse if we swapped it out with Opus behind the scenes than the next-highest-scoring ARC model would do in the same position, yes? For coding, research, dumb questions, SVG pelicans -- the lot?
If so, that seems like hard proof that this scores high on a benchmark at the cost of the benchmark itself. Like, if this transductive approach leads to ARC1 being claimed (which I thought it was ages ago but :shrug:), they'll either have to abandon the whole benchmark or ban this approach retroactively.
If not... well, I guess I encourage you to try it! It seems like you'd need 1000 truly stellar hand-picked examples to transductively cover that whole space, for one thing.
Thanks!
I think this is a great question. I have some thoughts on this but no hard evidence (neither does anyone else!)
Your argument relies on the AGI system being the model arch + weights. I think that the weights are irrelevant. The training algorithm is what is AGI: You choose/find a training set that covers a task, and then some form of deep learning with a big neural net.
For LLMs (which is a weak kind of AGI), this is freezing a model after NTP pretraining + RL postraining. This allows us to do well on a wide distribution of tasks.
But if we had a scoped task, I think its possible to just do the same thing with (1) a smaller dataset that covers this task + (2) not freezing the model (ie. test time training). This is easy for ARC because the data is small and has been curated well yes, but i don't see why can't this apply to more complex/ill-defined tasks (obv lots of things unsolved to make it work today)
> even if the ARC train+eval sets contain the sum of human intuition between them, superficial differences IRL would render your model unable to identify which examples are relevant to which problems, and thus unable to transductively reason.
In this hypothetical world, the dataset becomes incredibly large, and training on it makes it close to an LLM. You can then finetune transductively and we get the same thing (other approaches already do this iirc)
--
Note: BTW I'm not claiming that this is an AGI system, the benchmark creator also was clear that ARC is not sufficient for AGI (his goal was just to point out unsolved stuff, and the stuff ARC-2 pointed out was demonstrated very clearly by large reasoning models).
In the above work, I just wanted to show that AR transformers without pretraining can perform really well on this benchmark, which was incredibly non-obvious before.
The claims in this argument are separate and I haven't proved them yet
Kind of hijacking, would you say that LLM's have solved the frame problem?
To me, the frame problem is: Can you function in an open vs closed world, and to me the answer is yes, LLM's can definitely function in an open world where the rules are fuzzy, changing, undefined, etc. At the very least, much better than all GOFAI approaches by far.
The issue is now grounding - It can "function", but what would it take to "ground" them? A personality, maybe? Actual consequences? Making them interact only with constrained tools that are formally verified?
Right now it's a combination of harness engineering, and ml philosophers arguing about compression leading to the "objectively correct intelligence", whatever that means.
I think LLM's are "A[x]I" right now in the sense of "they have the capability to integrate with everything" - but obviously you can argue how much this actually reflects "A[x]I" (if you gave someone integration with everything, is that really your success or people handing you it)? But they are still missing some oomph factors that need to be clarified IMO. Maybe it's something as "mundane" as just having actual persistent memory, or maybe it's some deep philosophical thing like qualia. Who knows.
One of these is a much weaker claim than the other.
> yes, LLM's can definitely function in an open world where the rules are fuzzy, changing, undefined, etc.
> At the very least, much better than all GOFAI approaches by far
That's true. Again, I currently view LLMs as a function of integration - what they may lack in "intrinsic smarts", whatever that means, they can tool call and we build capacities (and they build capacities!) around them and to some degree can reason and be creative.
I do think the first claim has real merit even if it's not 100% on par with humans. Second claim is just true.
My actual answer here is a resounding "yes" and "no" at once, in the exact same way that the Turing test is both so obviously surmounted in 2023 (post-RLHF) to anyone applying 20c standards, while also somehow being so far away that we're not sure it'll ever be possible. The key is to 'dissolve the binary' for both, if you'll excuse the phil-ism: Turing's 1950 paper Computing Machinery & Intelligence was never intended to prescribe some yes/no evaluation procedure, and the people frustrated by the Frame Problem were not worried about a single yes/no "Frame Test", either.
Instead, Turing settled on behavioral comparison on an intuitive, human level as the best shared dimension to test, but only after calling Ed Zitron "absurd" and leaving room open for ESP & ghosts to end up proving souls (one of those is literally true, the other only figuratively).
By this metric, current LLM-backed agents are clearly able to behave like a reasonable-ish human over a long-ish timeframe -- that's just objectively an incredible achievement IMO, even from 2015 standards. The promised inversion is the retort that invites, namely: the '-ish' makes all the difference! An artificial mind that behaves in completely alien ways randomly is a much less useful tool even if those events are rare; ditto for an artifical mind that loses coherence across """mere""" days.
I'm cutting this as much as possible, but hopefully it's clear why all the above applies to the Frame Problem, too -- just replace 'behavioral' with 'epistemic'.
I think your use of 'ground' is nice and understandable, but is conflating too many things to work as a summary of the remaining work. All of the things you mentioned are absolutely being explored --both by scientists and by highschoolers collectively speedrunning 76 years of science live on HuggingFace to generate the best uncensored model for their polycule's DnD campaign-- but they hinge on distinct metrics.For example, the last one deals with reliability, which is closely related to the "randomly alien" stuff I mentioned earlier.
"Actual consequences", OTOH, most directly relates to the camp(s) focused on "embodiment", which is basically the idea that truly human intuition is too spatial to reasonably emulate without the ability to experimentally interact with the world -- AKA the "AI needs robots" camp.
And finally, the "personality" bit... I personally think we have to rediscover the subfield of Affective Computing, but the closest lane so far is "Constitutional AI", an approach popularized by Anthropic that (wisely) just moves the whole problem over to the world of prose and optimizes it from there.
All three are important steps indeed, but I think deserve finer delineation than ~'does it connect the agent to the real world more/better/stronger/truer'.
Ha, totally agree on the distaste for intelligence as a single dimension. I will also say that 'harness engineering' is gonna end up being an outdated term for 'the rest of AI' over time, MMW. A more (in)famous voice beating this same drum is Gary Marcus (I know!) under the term 'neurosymbolic' (?). Again, you have great intuitions here... I think it might help to consider how the human capacity for memory is simultaneously mundane and profound, at different levels of analysis. I think this situation is similar: we're not gonna need to invent Memory 2.0 (and can't, probably?), but there's a long list of human-specific heuristics, control planes, and other neural machines of some vague character that must exist, only a teeny tiny portion of which have been explored by "harness" engineering as of yet (for the best, probably...)TL;DR: We're not through the Kuhnian paradigm shift just yet -- the new episteme has far from penetrated all the subfields of cognitive science, IMHO. Predicting the landing point feels a little pointless, for that both that reason and an even bigger one: if RSI ends up being realistic (which it very likely is for our 2026 human computers, to some significant extent), this is all just the anteshock anyway. As "the singularity" implies, that kind of exponential shift could really take us anywhere (or nowhere, forever).
P.S. Never done this before, but fuck it: I'm currently seeking exciting remote work ASAP -- if you found this interesting, please consider this my cover letter. Sorry mods if against the rules, but, y'know... one-time exceptions for the singularity?
Hey thanks for sharing this. Was curious did you find the more you trained the model the more perf improved, or did it start plateauing. For example, let's say you didn't spend 67 cents, but you spent 67 dollars do you think you would get major benefits from that?
Yeah I've reached much higher perf but
- it feels logarithmic (like most perf-compute graphs), and eventually plateaus. 44% @ 67 cents was a good stopping point for me
- more compute would require a lot of effort and dealing with new problems like training stability, cost of iterations/sweeps (didnt have the money to convincingly run larger iterations)
I've been interested in training a transformer from scratch for the same learning reasons. The GPU cost/availability seemed prohibitive to do anything useful but you seem to have flipped that on its head. I love your outside the box approach.
*Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho*
On specific tasks sure on "generic performance" whatever that means for you not really.
As of today yes I agree with you
in the future, for general perf, I am optimistic that someone will figure out an alphazero like approach (ilya/silver/sutton/carmack seem to be working on something like this)
Fyi, the link to rhabdomyolysis is broken on the homepage! The URL is repeated
>Personally I think today's frontier models can be beat by training from scratch
You mean for a specific usecase?
Also aren't frontier models trained 'from scratch'?
> You mean for a specific usecase?
Yepp
> Also aren't frontier models trained 'from scratch'?
The full sentence was supposed to be "training from scratch only on ARC data"
the point was you dont need large scale pretraining
As an aside, the anagram is so good!
haha thanks!
Thank you for answering these questions. Looking forward for the next write up about this.
nothing like legendary shrugging and keeping the mind open
[dead]
I think you’re asking the right questions, sample inefficiency is horrible in modern LLMs. Despite this, I saw your analysis:
> The biggest increases in scores were due to
Modern architecture (SwiGlu instead of GELU, RMSnorm not layernorm, etc.) More data diversity, better shuffling of data scaling up: 8 layers instead of 4
This is commonly called squeezing the lemon and is usually a bit of a last resort. You should be able to achieve near SoTa with your new method, before you squeeze any lemons. This is, because the old SoTa is typically not using new optimisers and thus your results will be distorted by a large margin.
In terms of sample efficiency I want to add two things:
Runtime per-puzzle fine tuning is a very good target that provides a LOT of information. People have not looked at evolutionary methods to harness induction since the 90ies - if I was to work on ARC ever again I’m fairly certain this is where I’d look.
Best of luck, padawan
I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop?
> Training on the eval puzzles is cheating / “training on test”
> No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.
> Also, ARC is a metalearning benchmark, so you’re supposed to learn from the eval puzzles.
> Jargon: ARC has a set of train puzzles and a set of eval puzzles. Each puzzle has example pairs and test pairs. A pair consists of an input grid + output grid.
> The ARC, the label is only the test pair’s output grid in an eval puzzle.
> These labels were not trained on. They are hidden. You can delete it beforehand if you wish
I think what I gather here is that the test comes with one batch of training problems, which everyone agrees you can train on. But maybe the eval problems also come with input/output examples (to help define the problem) and training on those is controversial? I can’t see why it would be controversial but is that the criticism?
The point of ARC is essentially an "IQ Test" for AI systems. It is meant to cover abstract reasoning capabilities of generally-intelligent systems like LLMs. What the author did here was build a system that only solves ARC problems.
The other tension is the fact that this score is on the public eval set. In machine learning, you typically have 3 datasets: training, evaluation, and test. The training set is the dataset that's used to update the weights according to your loss function, you are "encoding" the patterns from the training set directly into your model. The eval set is what you use to track performance while training, it is NOT used to update model weights, but shows how well the model generalizes. The test set is a private holdout set that is only used when you're "done" developing your model. The difference between test and eval is information leakage: you can use performance against the eval set to modify your hyperparameters and model architecture to get better eval scores. So while the eval set doesn't directly update the weights, it can indirectly cause "overfitting" by tailoring your model to do well on the eval set. What you really want to see is the private test set performance, not the eval set. For all we know, this model could be ridiculously overfit on the eval set and perform poorly on the private test set.
2 different things are being called leakage here
1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, instead of being frozen after training
2) overfitting: agree that true measure is private set. It scores on private set roughly on par with TRM (a comparable model), obv with much lesser compute
--
also re LLMs: they do not follow this 3 split since (a) Incredibly hard to keep a pretrain dataset clean, (b) common in labs to benchmaxx during postraining (and known to do so on ARC)
> weight update during eval: this is a form of test time training and not really cheating.
Possibly "not really cheating", but it does make benchmark comparisons unfair - especially as the other models are unlikely to have their weights updated during the eval.
no, most models on kaggle are finetuning during test time, (including LLM based approaches)
Pure frontier LLMs dont, but thats because nobody knows how to make it work cleanly and at scale. Once someone makes it work, it will be deployed
I am very new to this but applying human intuition this still feels like cheating. Knowing all the question that will be on the exam and working on understanding them even if you are never given answers will obviously give you and edge.
what you describe would be cheating. My approach is the opposite. What I did was "You are born during the exam, given access to a training set and the questions then learn from scratch during the exam"
I put more details in the answer here: https://news.ycombinator.com/item?id=49525841
They have _not_ trained on the test set.
On the private test set, the right way to evaluate this type of model, is giving i it the test question Q, which it will first train to AR predict first, and then it will inference using the just-updated weights with Q as prompt, giving you back A, and then you compare A with A_true secretly.
I never said they did
It seems like an interesting strategy. Based on the author’s comment, they haven’t been at it for very long. So, I guess the folks who run the private test haven’t had a chance to get to it? It’d be interesting to hear how it does.
I'm currently 10th in the world on the private set on Kaggle. And iirc, at one point I was 4th
Can't comment more since its an ongoing competition
Does ARC measure "one shot learning"? I heard that the major unsolved problem in ML was developing systems that are good at dealing with novel problems.
Instantly one of my favorite comments this year. Nicely done.
What you gather is correct, assuming by "the test" you mean the ARC benchmark in general. It was controversial because people are used to LLMs which are frozen at train time, where the eval problems are usually not trained on for various reasons like fragility (basically porridgeraisin's ans which is great)
Here's another explanation. Take the train dataset and test dataset of a benchmark
Train: {x_i -> f(x_i)}, Test: {x_j -> f(x_j)}
As long as f(x_j) in the test set is hidden, there is no "training on test". In a normal benchmark, each x_i is a single datapoint. But in metalearning benchmarks like ARC, x_i is the puzzle itself that has a train set and the test questions within it, hence the confusion and controversy
I won't weigh in on whether it's "cheating" but it is definitely benchmaxxing
Yeah, unless the model is evaluated with blind hold outs, the benchmarks are misleading.
Basically, you have a bunch of Q,A pairs in the training dataset. Here, it was trained to next-word predict the question itself, as well as next-word predict the answer given the question as prompt. This is bog-standard, no one's complaining.
In the test dataset's Q,A pairs, it was only trained to next-word predict the question itself, and it was not given the answer at all.
It was then evaluated by seeing if it is able to output A_test given the Q_test as prompt.
What would be cheating is training it to produce A_test (given Q_test as prompt) as well, since then you can always make a model that scores 100% by just memorising Q_test, A_test pairs.
The complaints online mostly stem from not reading that properly and assuming they trained on Q_test,A_test instead of just Q_test. This is further because these days large LLMs are inadvertently trained on many benchmark solutions even unintentionally due to the massive scale of data and the infeasibility of auditing it all. But none of that is the case here.
The reason you want to train on Q_test is because in these AR transformer models, they learn useful composable encodings of Q by simply learning to next-word predict Q. So you enable the model to learn composable encodings of the test questions, so that it can hopefully "connect it" to an earlier train problem it had seen, and adapt the solution it had seen for that, much like humans do in school exams.
Without this step, you are making it difficult for the model to "connect" the test question to a train question it had seen earlier, and then it still has to adapt the solution. This way, you precompute that "this test question is like this train question" and then during the exam you only have to do the adapting the solution part after a simpler "retrieval" process.
You can just think of next-word training Q_test as a "retrieval" process.
This practice often used in continual learning or "test time training" is not yet useful in general real world ML tasks due to the differences in memory and compute requirements, and more so the general fragility of training large neural networks in a streaming realtime way (as opposed to large data, batched), versus inferencing from a static neural network. It is due to that fragility that I believe (correct me if I am wrong) this guy had to train on a batch of Q_tests. If you enforced that you will not provide Q_test_2 before they answer Q_test_1, the performance will drop.
While the increased compute and memory is difficult to solve inherently, there are various efforts being made to fix the fragility, especially in reinforcement learning where this is called "streaming RL", there is revival of interest as seen in RLC 2026.
[Note] Arc-AGI-1 doesn't have any actual english words or such, but it's simpler to pretend it was a basic Q&A benchmark to explain the above
Further question—the model produces an answer to the question, it sends the answer, and then gets graded. Does it get to know immediately how it did, or does it get the grade back at the end after answering all the questions?
If it is the former case, it would be possible to add the generated question/answer pair into the training set as well. Would that be considered fair? (Of course this is a moot point if the answers all get graded simultaneously at the end). Then the model could explore interesting strategies around what order to answer questions in.
In my uninformed opinion, the various permutations of question ordering/answer revealing all map to different real-world scenarios… and any of them could be interesting!
Nope, it never learns how it did on the questions.
During test time, you have to submit all the answers at once and you get the total score (so you dont even know which puzzles were solved)
It does not, if it gets the answer (or any information about them, even % of qns solved) and is able to adjust itself in response, then that is considered training on the test set and is wrong.
Sounds like a good day to be you, top 5 on Kaggle with a publication like this. It seems like you will be on a plane to SF shortly
Even cooler is his about me mention of saving his own life https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)
Crazy, considering rhabdo isn't that rare.
This was in India, where he describes the medical knowledge of providers as subpar at best.
Yeah but rhabdo is something literally any e.g. body builder, power lifter, etc could tell you about. Actually if somebody knows what hypertrophy is, they probably know what rhabdo is. It's a pretty normal and big concern in any sort of high intensity weight training.
I can't think of many ways that otherwise healthy and fit younger people can physically nearly kill themselves doing normal activity, so it kind of stands out - let alone it being not all that rare either. Rhabdo has even gone viral in the news like when a while back a couple of Chinese girls nearly killed themselves doing a social media 'squat challenge.' They did 1000, got rhabdo, didn't know what was happening, ended up in the ICU with kidney damage.
It also manifests in other ways too. For instance I had an elderly family member give himself rhabdo during a manic phase he was going through when he started going wild on construction and other physical tasks that were way beyond what his body was ready for.
Basically it's not some super obscure thing you'd expect only a good doctor, let alone a specialist, to know about.
People commonly knowing about Rhabdo is a much newer thing. I never heard people talk regularly about it all before CrossFit become popular
Yeah, before CrossFit it would be almost unheard of in the general population, except perhaps with the horseracing community.
Still, emergency doctors, especially in a hot place like India, should be aware of it.
there's a very large variance in doctors' abilities in India. At the very top they are close to the best in the world, esp with an insanely high workload.
but on an average, not great
Also, a lot of gymgoers and physical trainers I know hadn't heard of rhabdo either (and this is a relatively wealthy part of a tier 1 city)
things are changing for the better however
“What do you call a medical student who graduated at the bottom of their class?”
“Doctor.”
I studied biochem in undergrad and my classes were full of premed students.
I loved the subject and nerded out about the course material - I spent my time designing my own experiments around gene cloning that took several semesters to run. They were sharing last year's tests with their frat buddies and laughing at us nerds.
I've never looked at doctors the same way again after college. I looked up to them as a child, yet after seeing how the sausages were made, I started to doubt everything.
I frequently ask doctors, who spend all of ten minutes with me while the nurses do all the work, about the molecular specifics of what they're talking about. They talk down to me as if they're explaining to a child, yet they're frequently quite wrong. I'm not trying to sound superior to them, but I'm shocked they seem to care so little about the subject. It doesn't give me much hope about what they know and their abilities or competency.
I suspect surgeons and specialists are a different breed and aren't like this at all.
And to be clear, this isn't everyone. But it does seem to be the majority I've interacted with throughout my life.
When they act disgruntled at patient interaction, I detest that their profession tries to cap the number of med students per year. We should be letting in as many med students as we can take. We should let doctors from overseas immigrate and easily become practicing doctors here in the US. We should provide easy paths for nurses to become doctors.
The premed students in my university were chiefly concerned about money and prestige. They drove BMWs gifted to them by their parents and laughed at what I drove and how hard I studied. I had to put up with their bullying for years. I know not everyone who studies to become a doctor is like that, but it permanently skewed my view of their profession.
>When we should let doctors from overseas immigrate and easily become practicing doctors here in the US.
Lol, what does this has to do with anything regarding aptitude or curiosity!?
If we reduce the artificial shortage of doctors then the number of people entering the field specifically to make big money goes down.
> people entering the field specifically to make big money goes down...
This is a self defeating argument, because for it to work it requires that there is a larger number of people who love the field than the number of people who are in only for the $$$ (or else there will be even more scarcity of doctors)..
Which is not the case.
We have a surplus of people that want to become doctors and would work hard enough to do so. If we removed the artificial limits on residency, the supply of doctors would increase, which lowers wages somewhat, which lowers the number of people that want to become doctors. It reaches a new balance point with more doctors than before.
That itself is a good thing.
For it to also improve the motivations of doctors, all that needs to be true is that someone more motivated by money is more likely to be in the group that switches career.
It does not require the number of people that love the field to be a majority. Not that loving the field is a binary in the first place. People are a mix of motivations and cutoffs are arbitrary.
> It reaches a new balance point with more doctors than before.
But you just pull this "more doctors than before" out of your ass. There is no such assurance. If less people want to become doctors, then eventually there will be less doctors, not more.
I didn't pull it out of my ass. It's basic logic that if you remove a supply restriction then the supply goes up.
Let's attach fake numbers: Right now only 50 people can get residency each year, and 80 people starting college each year want to be doctors and would work hard enough. If you let everyone get residency, at first you'd make 80 doctors per year. This would decrease doctor pay and then only 75 people would want to be doctors, then 70. But if this ever dropped back near 50 for long, wages would spike back up, and the candidates would hit 80 again. It stabilizes in between, at 63.
With a bottleneck at X, and potential supply at Y, removing the bottleneck gets you a number between X and Y. All else equal, it can't stabilize below X. There will not 'eventually' be less doctors.
Alternatively don't even remove the bottleneck at first. Increase the residency bottleneck to 60. The number of potential doctors drops below 80, but it stays above 60, so now you have 60 doctors per year and the most money-motivated took a different major.
TL;DR: Let the number of residencies rise and the number of people that want to be doctors lower until they meet in the middle. Because residencies decide the actual number of doctors, meeting in the middle gives you more doctors.
> But if this ever dropped back near 50 for long..
There is no reason why it would bounce back at 50. What if it bounce back at around 20 and settle around 30? Then you have less doctors than before...
We already know that when the doctor production is 50, the number of potential doctors settles at 80.
Supply and demand means any lower doctor production causes even higher wages and that draws in even more potential doctors.
By what mechanism would it settle at 30? In particular, if it would settle at 30 despite an even harsher doctor shortage, why is it not already at or below 30?
To the parent posters credit, he's very honest that he developed a personal complex against doctors when he was a poor student. He just sees it as a way to lash out at American doctors, the irony being that foreign medical graduates leaving their families and communities to practice in America largely do so because they are exceptionally money motivated. That doesn't make them bad doctors, but there's certainly less likelihood they're doing it purely for the love of medicine or a desire to care for their communities.
I do agree with the sentiment though that the US needs to fund more residency slots as it's an asinine professional barrier, and that we would benefit from more physicians coming from more diverse financial backgrounds.
> certainly less likelihood they're doing it purely for the love of medicine or a desire to care for their communities.
Maybe self preservation?
In east europe, public hospitals will force doctors to work 36 hours shifts (overnight ER with theoretical sleep). Doctors have a full criminal liability for mall practise.
At the bottom of their class from the worst school in the country.
Well, I guess it would be nice if the graduation cutoff were above the level of “knows what rhabdo is”
Everyone and their dog who is on statins knows what rhabdo is. Bonkers!
I've been on statins for years, and I don't remember anyone talking to me about rhabdo. To be fair the education I received about my medications was a firehose of information after a heart attack and major heart surgery, so perhaps it's possible I missed a few things.
Not true, I am on statins and did not know.
I think is more concerning doctors didnt know about rhabdomyolysis...
> I agree that its rare to see to face problem sets in real life where every problem is given at once. Even if it is (like an exam), humans can usually only attempt one at a time
Just one small snippet that I thought was interesting. I would always read through ~the entire exam before starting. Both so that I could find the problems most approachable to me, but also because sometimes it helps me figure out the rest of the questions :-)
same! I'd often learn during the exam by solving an easier problem and then that would let me tackle a hard problem
> Ban offline training/pretraining. Models must train from scratch after submission Previously this was considered impossible so rule. My model shows this is possible Guarantees no synthetic data can be used It makes the comparison fair across differet models. Otherwise some models like LLMs can benchmaxx ARC by using ungodly amounts of offline training. (Since the benchmark has been around a long time, many ARC-like datasets have been created)
I'm not an ML researcher, so YMMV, but... how could a model learn to answer these ARC-AGI questions without training beforehand?
train from scratch only during the 12 hours allowed on Kaggle
Other competitions have implemented things like this before. Eg: OpenAI's Parameter Golf and Keller Jordan's Modded NanoGPT Speedrun
[dead]
how would you as a human know the answer to the arc-agi questions?
I spent years learning logic and doing puzzles. I don't think a baby or even average kid could solve these.
Predicate logic is trivially realized by linear transformations (I.e., matrices), and these matrices are easily discovered via gradient descent with appropriate reward functions.
> I don't think a baby or even average kid could solve these
The reward functions of a typical baby or kid is not 'get a huge dopamine boost when you solve a logic puzzle' (or whatever neurotransmitter, I don't know).
> Also, I’m not sure whether “general reasoning” even exists in the first place? Maybe humans are specialised too
I have been wondering the same. We are now exposed to so many stimuli, we are tricked into thinking this is the norm - to have a reasonable understanding about everything, unless specialization is called for.
> The mentioned approach is fundamentally flawed, since the inputs are used during pretraining constituting to a leakage, a universally recognized flaw of ML training.
I saw this on the community note for the last blog you wrote - anything to do here.
Not true. This is allowed in a metalearning context. Its called transductive learning and has existed since the 90s: https://en.wikipedia.org/wiki/Transduction_(machine_learning...
I address this in more detail in the blog
>Training on the eval puzzles is cheating / “training on test” No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.
I disagree, but i agree that training with answers is worse.
In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those questions. Sometimes it was the same exam sometimes they were 'slightly' different. In no occasions were the answers shared, and being math exams it wouldn't have made a difference, since solving the exercises is the actual training that allows replication of results and adaptation in testing.
It's a grey area for sure, but it's a quantitative matter, studying 20 different exams for 2 months is quite different than trying out 1 exam 1 week prior to an exam to verify all is well.
It's called teaching to the test, not teaching to the test and answers. In essence OP holds a naive version of what cheating is, and thus they think they are absolved, when actual cheating is much more nuanced. Many such cases.
Fwiw, the second uni I dropped out of was worse in that some students just used their phone during tests and talked with each other or googled. OP sounds like a student from Uni 1 claiming they don't do what Uni 2 students do.
And for reference, the exams I did pass I did by just reading the whole bibliography on my own, and doing exercises from the book if needed, I was passing with like 80-90%, never did I have to get a copy of a previous exam and study that, I think it's a ridiculous concept that has been normalized to meet an increasing societal pressure on everyone being an elite graduate (we can't ALL be elite), and if this repo is successful, it's because this attitude is so normalized that it's seeping into machine learning by diffusing the lines between training and testing set, and increasing the ratio between one and the other.
Hell, I'm not surprised that the software that the mass of test -studiers develop is software that studies tests. In the same manner that the software that cheaters develop is software that cheats Guardrails and breaks ToSes
> In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those question
Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"
What you described is cheating because more time than the exam permits. ARC was designed specifically to avoid this. The exam in question (kaggle competition) is 12hrs long with 4xL4s. I trained on 1.5hrs with a single 5090 (which converted to 4xL4s is slightly longer, but still within 12hrs).
(There's also access to experts who know the answer, which kaggle bans by banning the internet)
Lucas describes it well here (and his original tweet up the thread): https://x.com/giffmana/status/2002128356901597509
--
Your arguments btw support my work over the LLMs more. LLMs today are postrained with a large amount of synthetic ARC data. (Exactly the "teach to test" criticism). Thats why they perform so well on ARC. Base models still are terrible at ARC-2
I didn't phrase that well and cant edit, so clarifying:
What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"
>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"
>What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"
This is getting into metaphorgotten territory, I'd have to do a deep dive to really understand whether it is "cheating" or, more formally, a methodology that encourages overfitting.
But for what is worth, taking the textbook to the exam (regardless of whether you were born there or not), would be cheating in a test as well. Although it is possible that in the model training context it does not lead to overfitting, it certainly doesn't preclude it.
>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"
It is comforting that someone agrees, to this day this is considered a gray-area (or not even that) by students, graduates and those that look up to the University of Buenos Aires institution.
well not if its an open textbook exam
in non metaphor terms: In many ML situations you can carry the train set with you test time. Eg: KNN, SVM, replay buffers, etc
this is one such case
--
the separate overfitting concern is fair, look at private holdout performance for that. it performs on par with TRM (a comparable model) in the private set, ofc with far less compute
I think you can do whatever you want with your train set, what would be concerning is training with the test set.
I don't think overfitting is separate, the consequence of putting the benchmarks in your training set is not that you "cheat" by breaking some moral code, it's that it breaks the purpose of the benchmark and trains your model to be good at that benchmark only, instead of being generally useful.
> Increases in LLM scores are now mainly driven by post training (evidence in next section) and are probably a function of amount of synthetic data. They are learning to solve ARC tasks, not learn general abstract reasoning
Agreed and that's for any benchmark. Private tests are better but you still have to trust the provider to not log and use them for training.
That's why I like when a new set of tests like a new ARC-AGI version is published, that's where you can see which of the models abstracted to more general capabilities instead of being focused on the previous tasks. Most models completely fail new ARC-AGI tests.
The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. You hit a ceiling very fast and investing into more compute will give you diminishing results. So yes, you can train a custom model to do somewhat decently on a specific set of tasks but then what?
Then nothing - that's awesome. People think that LLMs are the know-all do-all solution to every problem now.
Putting solutions in terms of cents is a great way to potentially win over some ai boosters imo. There are other ways to solve hard problems.
> The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results.
I think thats unfair. Perf-compute is often logarithmic and will always saturate . Reaching the plateau faster is valuable as it often leads to better peaks (held true here and also look at modded nanogpt)
And more compute increases the perf (after dealing with other scaling problems)
> The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results.
Nothing about saying that it cost 67 cents implies that it will. Knowing only that it costs 67 cents you also have no reasonable basis for extrapolation. It doesn't indicate a trend whatsoever.
I don’t have any kind of ML background but I have always thought of sample efficiency as the great unsolved problem of AI. We humans have unbelievably good sample efficiency; often we can durably learn something on just a single example or two. This is the main area in which LLMs are vastly, vastly behind us.
the caveat is that we are not learning those small number of samples from scratch, since we're coming in with a large amount of training already, much of it from before we were even born
I've seen a few videos about people having their girlfriend/dad/etc play very simple video games for the very first time. It's a really great example of this. The most basic, simple, mechanics are difficult for them, and take surprising repetition to remember, because they have zero framework around any of it. It's really similar to watching a genetic algorithm learn to play Mario, but on fast forward.
No, the unsolved problem of AI is continuous learning. We never stop learning, we don't have a "training phase". You are always updating your world model even when you sleep. Also more quality training data does lead to greater learning efficiency as you have more priors to work with.
>We never stop learning, we don't have a "training phase".
We kind of do, if you count critical periods in childhood.
Probable mechanism: "What we found is a rapid increase in GABA in children, associated with learning, ..." [1]
[1] https://www.brown.edu/news/2022-11-15/children-learning
Why not both?
How does it perform on ARC-AGI-3?
There was this a few weeks ago:
"Schema Harness Achieves ~99% on Arc‑AGI‑3 Public" https://news.ycombinator.com/item?id=48938163
>> Schema, the harness we introduce today, reaches 99% on the ARC_AGI_3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol
What does that do with 5.6 Luna instead of the expensive models?
What of 'schema' would improve the performance of mdlARC?
mdlARC: https://github.com/mvakde/mdlARC
There's an updated ARC-AGI-1 chart with 5.6 Luna in each thinking level in this video from last week: "A New Architecture [..] | MOONSHOTS " https://youtube.com/watch?v=qQfUbo7Ldc0&t=2m5s
its not gonna do well on ARC-3 without some significant changes and effort
The new arch in that video is kinda misleading. Didn't really compare against proper baselines
Yeah, but still, how does that agent perform if paired with this inexpensively trained transformer instead of the more expensive foundation models?
Isn't this cheating? Or rather, are frontier agents only looking at one question at a time? If I understand correctly, you're looking at all the examples of the exam questions. If the exam was adjusted so that you can only look at one question at a time, you won't get 44% anymore.
Why would that be cheating? That's what humans do when they learn, they look for the signals and patterns that reduce the possible set of answers so they can converge on the solution and narrow the search space.
I mean, I'm interested to know if the frontier models also get to see all questions at once. Then it's more fair game than if they just see one question at a time.
They look at each problem individually.
As the ARC AGI guidelines [1] state, 'a core design principle of ARC-AGI is that the test taker must not know what the test will be.'
[1] https://arcprize.org/policy#dataset-security
Really impressive and creative research. I wonder if the leading labs do anything similar with their models? It doenst look like the open source labs do?
It means the questions including their answers are dependent. Ie, theres a data generating process for them, that the model uncovers. Like a KNN is known to have near Bayes accuracy as k/n to 0, n to infty, k to infty. The data reveals the dgp.
I suspect if you feed unrelated or even garbage questions into the eval set, it would reduce the performance.
I know that this is Transformer, and not LLM per se. But isn't this the same idea that PaulG said the other day, and many of the comments criticized him?
https://news.ycombinator.com/item?id=49412396
Yeah but I posted this work back in Dec-Feb lol
My above comment wasn't aimed at you. Thanks for the response though! What you did is phenomenal, and I wasn't taking a dig at you in any way.
What I intended in my comment was that, may be starting from scratch (like you did many months before, and what PaulG suggested recently) is the way to go for future job prospects and startups. Most of the comments in the thread I posted was negative for PaulG saying that.
Yeah agree with pg completely. Multiple people I know did this
So this achieves 44%, I'm wondering: To consider yourself a reasonably competent human, what % to achieve on ARC-AGI-1? Thanks in advance to anyone that knows, can answer!
@mods I am not sure the editorialized title is better! (It is a bit clickbaity, since this is a transformer that does not compete with LLMs at all outside the arc-agi-1 benchmark)
Thanks for motivating me to work a bit on non-LLM things again :)
Just wanted to say that the rhabdo part mentioned on your site was really impressive! Speaking as a medical doctor and full stack engineer myself.
thanks! was incredibly scary when it happened
Nice to see arc-agi-1 framed this way — I'd been circling the same idea without the right words.
Nice. It took me an hour just to understand what this competition is about :)
"I don’t understand why others didn’t figure this out"
- how about we allot the possibility that so many of presumed ML experts don't have any clue what they be doing, and are eventually API bitches, nothing more.
Could be that many presumed ML experts don't even know how to train on the evaluation set
I'm so sorry, in how many?
[flagged]
[flagged]
Is the author only running their model against one benchmark? I don't think anyone finds that difficult to achieve, the difficulty comes when you want to make the model not benchmaxxed to a specific benchmark, and generalize so it can solve problems not part of the training data, but seems this model is specifically for not this? How useful is that?
If you just wanted to pass these specific tasks in this specific benchmark, and wanted to do so cheaply, I'm sure a non-LLM-based approach would yield better results for even cheaper, since what the author's model does, seem to basically be "solve ARC puzzles", not a general LLM or "coding" LLM.
I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture. A lot of people have sort of forgot that machine learning is more than just LLMs these days.
I found it to be a very interesting angle.
> He is showing computers can solve these issues, without using an LLM architecture.
Isn't it a LLM he's building though? My very point is that this particular use case could be solved better without building a LLM, now you claim he is not? The description of what he's doing surely makes it sound like it's a (very small) LLM, and personally I'm still on the "if it quacks like a duck" train in life.
> A lot of people have sort of forgot that machine learning is more than just LLMs these days.
Yeah, which I guess if you make my previous comment more concise, is exactly what I state too.
Nowhere does he say he built an llm. Hes using a transformer, not an llm.
> Nowhere does he say he built an llm. Hes using a transformer, not an llm.
Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results".
Obviously it's not a Large Language Model, but to me this looks more like a LLM than not, given the architecture he's chosen. But again, maybe I misunderstand?
Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs.
When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks
And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles
Out of interest, would you call BERT an LLM? It’s pre trained but not particularly large.
AFAIK, the “large” qualifier came when transformers allowed to scale the size of language models compared to the recurrent models that where in fashion before. And although BERT isn't large by today's standard, it was large enough for the time.
idk the definition is fuzzy. thats why people use the "modern" qualifier to talk about decoder-only style and this is also not clean since you now have reasoning models which are separate
It’s neither large nor language-based. ARC-AGI-1 is grid-based and nonverbal.
Use of a transformer is not necessary or sufficient to qualify as an LLM.
A LLM should at the very least be a language model, i.e. be able to take human-readable text as input or produce it as output. Transformers are used for plenty of tasks that don't involve language, for example object detection or blind source separation, where the models aren't called LMs; and on the other hand there are some LLM architectures that exclusively use linear attention variants and aren't really transformers anymore.
Transformer solves a Seq2Seq problem just like RNNs. All Seq2Seq problems need not involve a language. In this case teaching on ARC puzzles doesn't mean what he trained is now trained on a language which will be English(or any other language) in this case. So, does his training successfully models "English as a language" -> No. This implies it is not "Large" and has not modeled any "language".
Just to be clear: It was well known that you can reach such scores with small models and without an LLM if you train on the task. The author highlights those models himself - e.g. HRM/TRM.
The novelty is more that it works with such a plain transformer and low compute price.
The whole point of his model is to optimize for a very specific benchmark.
BUT, he does not use labels when training, so the model does not know the answers.
> The whole point of his model is to optimize for a very specific benchmark.
But benchmaxxing is what we generally try to avoid for training, as there is no point really for it. We used to call it "overfitting", now you're saying this person does it intentionally? Why?
There are plenty of applications where a machine learning system needs to optimize for a very limited data set that is still intractable by linear logic systems of reasonable scale and complexity. It’s interesting, because he is using the legos of LLMs to build highly specialized machine learning systems, which is a very pragmatic approach. Obviously a lot of other ways to achieve similar goals, but it’s cool to see someone back porting the modern tools towards older style optimizations.
Also, the complexity of the task he is using occupies an interesting middle ground of ultra high dimensionality (for a “simple” problem) while being limited in width to a narrow set of solves- a space where one would be tempted to imagine you would need a much more capable system.
Overfitting, as well as the specific instances I've seen of the word benchmaxxing, involve knowing the answers and training to those answers. That did not happen here. The model is limited in scope, which means it's not being scored on generic intelligence, but neither is it defective and terrible at solving new problems inside its scope, like you get with overfitting.
Why not? There is $700k reward for the next iteration of this benchmark https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2...
I would not call this overfitting, it's finetuning for specific task where you have a benchmark.
[dead]