Does this give sensible analysis if you don't record your thinking out loud?
I've been using Elogram (https://elogram.gg/, made by a friend), which gives interpretability by showing how difficulty of a position changes with elo.
No it does not help the quality of the analysis from what I saw, but it makes the analysis much more interesting, because it can challenge my wrong judgement and answer the questions that I asked myself out loud.
Very interesting project, is Maia used in this platform?
It integrates an LLM of your choice with StockFish to analyze games. Back then, the results weren't very good and I abandoned the project but a lot has changed (the LLM's got better and I did too) so maybe it is time to take another stab at it.
The early GPTs were supposedly reasonable at chess but it indeed is not a very useful skill to have for an LLM (Elo 1800 IIRC) which is stronger than most players and for them it could be a useful tool.
Even at higher levels, I think there could be use. The Dvoretsky example on my GitHub is real but it’s not a guarantee you get it and it could just be reiterating from training. What didn’t work for me was forcing it to look at lines in detail, but maybe with larger context and better engineering, you can get the LLM to extract themes from the search tree. With compare and contrast, it could maybe figure out why a move works in one variant and not in another and produce a coherent explanation of this nuance.
I’m not trying this yet because it would take a lot of tokens and I don’t have the time.
As a chess player I feel this is fascinating although there are plenty of analysis options already on lichess on chess.com. Still, sometimes it is more about learning and curiosity. Well done.
Just a note that this is not novel. Automatic analysis of chess games built on traditional engines has existed for a long time. Without using stockfish as a sidekick, this wouldn’t be interesting because of the poor play of LLMs, however having an LLM enhance the commentary is interesting and I suspect has already been done by chess.com for years.
This what I think too. I also think there are much more refined approaches than the one I tried here. From a cursory look, I saw there are both scientific litterature on the subject of mixing llms and tools like stockfish and some dedicated closed platforms that put that into action.
Let's make clear that I did not spend much time on this project. Ideally I would have tried to put other models into the mix, like maybe Maia to better see the game from a "real player" eyes and pinpoint where expected move and stockfish moves differ.
Anyway, to me, it's good enough to be usable and shared.
You could also download my free chess app that hooks up to Claude for real time game analysis or chat about the game.
https://pwalessi.itch.io/chess-buddy
Nice use of a skill for something concrete. I've been happier with AI helpers that spit out a short, checkable report than ones that try to be a whole coach.
How does it work? Does it just use stockfish under the hood?
I just remember the times when I tried to use an AI model to play chess and half of the time it either hung a piece or made an illegal move.
Was a bit to overzealous... Just read the readme. Looks pretty cool. Maybe I'll give it a try and see how it compares to the chess.com analysis (although I have to say that I never really use it).
You only need to spend a couple of minutes talking to an LLM about chess to realize it has no understanding of the game. It will produce reasonable sounding explanations for things that are completely wrong. It will make illegal moves. It will hallucinate pieces on squares even when the entire PGN is in the context. There are no prompts you can give it to fix this. I don't really understand how you could create this skill without noticing this.
Absolutely it will. You have to run them in an environment without tools because today they just turn on stockfish if you ask on claude.ai.
These models are strong at math and coding because they have been heavily trained for it in post-training. That is where a majority of the gains have come from in the past two years. That has absolutely no impact on chess ability. Heavily train a deep learning model for chess and you get Leela, an absolutely monster. Don't train them for chess and I doubt its even 1200.
It is possible to do post-training on a base LLM and get a strength equivalent to a strong amateur but people haven't done that at any meaningful scale recently, so hard to say what the ceiling might be with the largest models. Probably pretty high but it would be expensive and still worse than Leela.
The only interesting question is could it explain chess to you - and I haven't seen evidence that it can. ChessGPT at least - while purportedly 1800 strength, can't explain strategic themes from the first chapter of beginner strategy books accurately or explain why specific moves are bad (positionally), it just knows they are.
Yep, they are quite bad without stockfish. You can test it with the playchess skill in my repo. Maybe 1200/1300, who knows? Still I do think that they can, with enough time, explore multiple variations where they confront their naivety to stockfish and build up a compact picture on why move Y should have been played instead of move X. That was my intuition when building this skill.
Have you actually tested this with quiet positions, where understanding needs to be more about different strategic or positional choices? They can explain concrete tactical lines with stockfish analysis at hand, but so can an analysis board. In my experience, even with fairly deep and accurate engine analysis, they cannot do this accurately with quiet positions. I tried to build a product around this idea and I'm convinced that even if it is possible it would require hundreds of thousands - or millions - spent on post-training to even confirm that possibility.
Good point. I had to tweak a bit the system to get more positional analysis because this is something I wanted too get. I think it's still shying away on this aspect. When it does dwell on it, I feel that when the output describe what should be the plan of both camps, it's quite convincing but only a much stronger player (or me with stockfish) could really assess this.
Anyway I am with you that deep positional appraisal is very hard and we should not expect too much from this set of skills on that front.
Thx for tips. I just saw there are some courses on chess.com, that will be nice to compare claude understanding with Silman commentary as the gold standard.
Edit: I am pushing a small attempt on this position. I fed this one as a screenshot https://www.chess.com/lessons/roots-of-positional-understand... . I forgot to tell that it was black to move but that was fine. The analysis quality seems ok. What puzzles me is that it recognized the Carlsbad structure and talked a bit the plans around the minority attack, not only the correct explanations on g6 and why it should be played. This is where we benefit from positional stuff in the training data.
Not by hand, no, the code was generated with claude code. The readme too, but with some extra efforts to avoid the awful ai generated readme.
It took multiple sessions to get to this result. At first I only generated annotated pgns and standalone html page inspired by lichess. The video generation was the cherry on top, it took few iterations too to fix issues and add markers and arrows. I only use consume the generated video these days, for the moment.
Neat! I've been slopping together various games to have AI play them, balatro, poker, blackjack, it's just so fun. Was just thinking about chess, wondering if, if I gave them an unrestricted sandbox, they'd start using existing solvers or not. They're surprisingly good at doing cardcounting, by the way.
Writing skills files like this one don't seem to be the future of computing or AI, not by the quality, nor by the cost (15$ per usage?).
Also, you seem to have written the actual SKILL.md prompts themselves with AI? I don't know what to say that's insane, at least write the prompts? This idea of asking chatgpt to write the prompts for you is beyond lazy. Then presenting it as a project or tool of value to share to others is delusional.
No need to apologize. I like reading HN for comments that don't beat around the bush.
It's true that it's costly. I never tried to optimize it. In a way I feel this hacky project doesn't deserve its place on the front page. It's just me hacking around on Claude Code to get something I like. Take it as a proof of concept if you will. I'd be happy to see lighter alternatives.
> Then presenting it as a project or tool of value to share to others is delusional
Here I see a gap in your reasoning. Lazy and costly, sure. But useless, I'm not so sure. In my world, a vibecoded tool can be useful enough to be shared, despite the risks. Claude Code itself is almost entirely written by Claude, according to its creator. Yet I use it every day.
"The fact that it reflects on my own thinking during the game makes it interesting from a teaching point of view, so I thought it was worth sharing." As AI the premium sycophantic scammer does
"but for me it is a much more pleasant and memorable experience than clicking around Stockfish branches"
And SO much less effective than doing the harder, more tedious feeling work
The jury is out for the effectiveness, it's hard to debate this subject. From my cognitive background I know very well how important the generative effect is for learning. Maybe I'll add features that leverage generative/testing effect one day. Anyway, clicking on stockfish branches can be quite a passive activity too if done badly. I don't know how to do that well to be honest. My goal is often to just to understand what I missed, full stop.
For the sycophancy, I can say I did not feel that at all. When stockfish says your move suck, claude would have a hard time saying the opposite (no "you are absolutely right" when I am not).
That was exactly my estimation when I played "raw" against Opus. But Opus with stockfish as a tool and much time available can, from what I experienced, generate good comments.
Does this give sensible analysis if you don't record your thinking out loud?
I've been using Elogram (https://elogram.gg/, made by a friend), which gives interpretability by showing how difficulty of a position changes with elo.
No it does not help the quality of the analysis from what I saw, but it makes the analysis much more interesting, because it can challenge my wrong judgement and answer the questions that I asked myself out loud.
Very interesting project, is Maia used in this platform?
Already a year ago I vibe coded Babelfish: https://github.com/bobjansen/Babelfish
It integrates an LLM of your choice with StockFish to analyze games. Back then, the results weren't very good and I abandoned the project but a lot has changed (the LLM's got better and I did too) so maybe it is time to take another stab at it.
LLMs have gotten better at things they've been heavily post-trained for; its really post-training that has gotten better.
The early GPTs were supposedly reasonable at chess but it indeed is not a very useful skill to have for an LLM (Elo 1800 IIRC) which is stronger than most players and for them it could be a useful tool.
Even at higher levels, I think there could be use. The Dvoretsky example on my GitHub is real but it’s not a guarantee you get it and it could just be reiterating from training. What didn’t work for me was forcing it to look at lines in detail, but maybe with larger context and better engineering, you can get the LLM to extract themes from the search tree. With compare and contrast, it could maybe figure out why a move works in one variant and not in another and produce a coherent explanation of this nuance.
I’m not trying this yet because it would take a lot of tokens and I don’t have the time.
1800 was ChessGPT, it was post-trained on chess.
As a chess player I feel this is fascinating although there are plenty of analysis options already on lichess on chess.com. Still, sometimes it is more about learning and curiosity. Well done.
Just a note that this is not novel. Automatic analysis of chess games built on traditional engines has existed for a long time. Without using stockfish as a sidekick, this wouldn’t be interesting because of the poor play of LLMs, however having an LLM enhance the commentary is interesting and I suspect has already been done by chess.com for years.
I'm an ok chess player (roughly OP level) and slightly more experienced claude code user.
I tried to use claude to analyze my games using stockfish - this is not a problem claude is good at.
I spent few hours (however admittedly around opus 4.8) with claude on this and it was unable to use stockfish in any useful way.
I think if the skill created by OP works then it's a real contribution. I'll give it a try.
Thanks for trying it out! I'd be happy to see how it goes for you and compare our implementation notes if the analysis you got is not too bad.
There is a massive gap between computer analysis and teaching though. And its larger the less you know about chess.
Also dont forget that stockfish and agents are not alternatives. An agent can use stockfish to help players learn.
This what I think too. I also think there are much more refined approaches than the one I tried here. From a cursory look, I saw there are both scientific litterature on the subject of mixing llms and tools like stockfish and some dedicated closed platforms that put that into action.
Let's make clear that I did not spend much time on this project. Ideally I would have tried to put other models into the mix, like maybe Maia to better see the game from a "real player" eyes and pinpoint where expected move and stockfish moves differ.
Anyway, to me, it's good enough to be usable and shared.
You could also download my free chess app that hooks up to Claude for real time game analysis or chat about the game. https://pwalessi.itch.io/chess-buddy
Looks really cool, would love to try it but I'm on Mac so would appreciate Mac version.
I don’t have a Mac handy but I’ll see if I can build for Mac in Godot without a Mac
Thanks for sharing! So you gave stockfish to Claude too? Did you try other techniques?
Yeah I give the board position and stockfish’s analysis to Claude. The opening name too.
You say chess and I’m here to shamelessly promote my “coin chess” version:)
https://chesspoly.com/
This needs some balancing. The fact you can buy a bishop on turn 1 and check their king to disrupt their economy is overpowered
Thanks for playing both. Did you by chance try to play vs computer on medium or hard level?
Is not it quite easy to defend against this attack simply putting a pawn before kind?
Yeah, just have it a go and it's way too easy to grab a bishop from the start and just use checks to get them in a bad spot
Nice use of a skill for something concrete. I've been happier with AI helpers that spit out a short, checkable report than ones that try to be a whole coach.
How does it work? Does it just use stockfish under the hood? I just remember the times when I tried to use an AI model to play chess and half of the time it either hung a piece or made an illegal move.
Was a bit to overzealous... Just read the readme. Looks pretty cool. Maybe I'll give it a try and see how it compares to the chess.com analysis (although I have to say that I never really use it).
You only need to spend a couple of minutes talking to an LLM about chess to realize it has no understanding of the game. It will produce reasonable sounding explanations for things that are completely wrong. It will make illegal moves. It will hallucinate pieces on squares even when the entire PGN is in the context. There are no prompts you can give it to fix this. I don't really understand how you could create this skill without noticing this.
Will it make illegal moves? I have doubts regarding this, at least for the strongest models like Opus5.5 or Astra.
Absolutely it will. You have to run them in an environment without tools because today they just turn on stockfish if you ask on claude.ai.
These models are strong at math and coding because they have been heavily trained for it in post-training. That is where a majority of the gains have come from in the past two years. That has absolutely no impact on chess ability. Heavily train a deep learning model for chess and you get Leela, an absolutely monster. Don't train them for chess and I doubt its even 1200.
It is possible to do post-training on a base LLM and get a strength equivalent to a strong amateur but people haven't done that at any meaningful scale recently, so hard to say what the ceiling might be with the largest models. Probably pretty high but it would be expensive and still worse than Leela.
The only interesting question is could it explain chess to you - and I haven't seen evidence that it can. ChessGPT at least - while purportedly 1800 strength, can't explain strategic themes from the first chapter of beginner strategy books accurately or explain why specific moves are bad (positionally), it just knows they are.
https://github.com/waterhorse1/ChessGPT
Yep, they are quite bad without stockfish. You can test it with the playchess skill in my repo. Maybe 1200/1300, who knows? Still I do think that they can, with enough time, explore multiple variations where they confront their naivety to stockfish and build up a compact picture on why move Y should have been played instead of move X. That was my intuition when building this skill.
Have you actually tested this with quiet positions, where understanding needs to be more about different strategic or positional choices? They can explain concrete tactical lines with stockfish analysis at hand, but so can an analysis board. In my experience, even with fairly deep and accurate engine analysis, they cannot do this accurately with quiet positions. I tried to build a product around this idea and I'm convinced that even if it is possible it would require hundreds of thousands - or millions - spent on post-training to even confirm that possibility.
Good point. I had to tweak a bit the system to get more positional analysis because this is something I wanted too get. I think it's still shying away on this aspect. When it does dwell on it, I feel that when the output describe what should be the plan of both camps, it's quite convincing but only a much stronger player (or me with stockfish) could really assess this.
Anyway I am with you that deep positional appraisal is very hard and we should not expect too much from this set of skills on that front.
When I was testing I just gave it positional exercises from Silman.
Thx for tips. I just saw there are some courses on chess.com, that will be nice to compare claude understanding with Silman commentary as the gold standard.
Edit: I am pushing a small attempt on this position. I fed this one as a screenshot https://www.chess.com/lessons/roots-of-positional-understand... . I forgot to tell that it was black to move but that was fine. The analysis quality seems ok. What puzzles me is that it recognized the Carlsbad structure and talked a bit the plans around the minority attack, not only the correct explanations on g6 and why it should be played. This is where we benefit from positional stuff in the training data.
Very cool work!
Did you write the skills (text, and code) all by hand, or are those prompt outputs?
Not by hand, no, the code was generated with claude code. The readme too, but with some extra efforts to avoid the awful ai generated readme.
It took multiple sessions to get to this result. At first I only generated annotated pgns and standalone html page inspired by lichess. The video generation was the cherry on top, it took few iterations too to fix issues and add markers and arrows. I only use consume the generated video these days, for the moment.
Makes sense, good use of the tools at hand. And the SKILL.md files, also generated similarly?
Yes.
Nice. The best part about this kind of development using models is when you have them write instructions for themselves / other models :).
I am very interested in this, but it looks like the video link gives 404
Weird, the video is supposed to be embedded. On my chrome desktop it displays properly. EDIT: fixed
Neat! I've been slopping together various games to have AI play them, balatro, poker, blackjack, it's just so fun. Was just thinking about chess, wondering if, if I gave them an unrestricted sandbox, they'd start using existing solvers or not. They're surprisingly good at doing cardcounting, by the way.
I find it funny that a video game (Balatro) is lumped with poker and blackjack.
It is :) Mostly because it's easy to interface with!
Writing skills files like this one don't seem to be the future of computing or AI, not by the quality, nor by the cost (15$ per usage?).
Also, you seem to have written the actual SKILL.md prompts themselves with AI? I don't know what to say that's insane, at least write the prompts? This idea of asking chatgpt to write the prompts for you is beyond lazy. Then presenting it as a project or tool of value to share to others is delusional.
Sorry for being harsh
No need to apologize. I like reading HN for comments that don't beat around the bush.
It's true that it's costly. I never tried to optimize it. In a way I feel this hacky project doesn't deserve its place on the front page. It's just me hacking around on Claude Code to get something I like. Take it as a proof of concept if you will. I'd be happy to see lighter alternatives.
> Then presenting it as a project or tool of value to share to others is delusional
Here I see a gap in your reasoning. Lazy and costly, sure. But useless, I'm not so sure. In my world, a vibecoded tool can be useful enough to be shared, despite the risks. Claude Code itself is almost entirely written by Claude, according to its creator. Yet I use it every day.
"The result is not perfect" as LLMs are
"The fact that it reflects on my own thinking during the game makes it interesting from a teaching point of view, so I thought it was worth sharing." As AI the premium sycophantic scammer does
"but for me it is a much more pleasant and memorable experience than clicking around Stockfish branches"
And SO much less effective than doing the harder, more tedious feeling work
The jury is out for the effectiveness, it's hard to debate this subject. From my cognitive background I know very well how important the generative effect is for learning. Maybe I'll add features that leverage generative/testing effect one day. Anyway, clicking on stockfish branches can be quite a passive activity too if done badly. I don't know how to do that well to be honest. My goal is often to just to understand what I missed, full stop.
For the sycophancy, I can say I did not feel that at all. When stockfish says your move suck, claude would have a hard time saying the opposite (no "you are absolutely right" when I am not).
Generative AI is really good at making you feel like you understand a lot without actually making you understand anything in-depth
Because lengthy difficult cognitively intensive labor is required for the brain to actually change its structure and connections
Very true. I like to think AI somehow optimizes for "efficient vagueness", which is bad news for our brain.
Nonetheless, as many do, I often ask AI to explain me stuff. I know it's not perfect, but it's convenient, it's a trade-off to make.
The elo of opus 5 models is 1300 or so. I wouldn't take chess lessons from a 1300.
That was exactly my estimation when I played "raw" against Opus. But Opus with stockfish as a tool and much time available can, from what I experienced, generate good comments.
Well yeah, of course. But also, analysing with stockfish is _also_ useless, unless you're a super GM.