There’s been an interesting co-evolution that I’ve been experiencing with Claude Code. I’ll ask it to do a task, I’ll watch what it’s doing (often lots of find and grep and ripgrep) and then after the task is complete I’ll ask it if there are any tools that would’ve made the job easier. This has led to tools like fzf and others (notmuch for indexing email, for example). I’ve then taken those tools and figured out how to work them into my own workflow, both CLI and Emacs.
We’ve also collaborated on some Python tooling that takes a rather slow data format that I often have to process and analyze, indexed the whole corpus, and for analysis I can do (or Claude Code can) a single-pass conversion to Parquet which is then queryable with DuckDB. That tool has dramatically improved my turnaround time on one-off analysis tasks and as a Python CLI tool using Typer, the interface is also nicely discoverable for LLM harnesses to work with.
I wonder if you put job classifications along a spreadsheet's leftmost column (e.g. search in file, search in directory), and data types along a spreadsheet's topmost row (e.g. JSON file, macOS-based filesystem), would you end up finding gaps in the intersection of the columns and rows make these new tools you've created less surprising? For example, to "search a file" on a "macOS-based filesystem", you would naturally gravitate towards `grep`. Or to search the contents of a JSON file, you would maybe use `grep`, or perhaps `jq` and a well crafted query. Well, as you expand out the job classifications and data types (or some other abstraction), you start to realize where we don't have tools today. And then Claude can go and create those, perhaps even proposing a faster alternative!
I had a similar intuition that we could definitely optimize token efficiency and performance for non-general tasks.
For example, I was handling a lot of infra operations, many of which follow fixed workflows like locating a specific log in certain time range, identify the issue, and apply the appropriate mitigation steps. Since these workflows are entirely command-line based, I added a hook to Claude Code that monitors the commands I repeatedly run, including those invoked by Claude Code itself, and compiles them into reusable scripts(with skills as entries). It can work well.
The takeaway is that we should spend more time reviewing our own work patterns and workflows.
When you use “we” hopefully you are referring to someone on your team. Just be careful, the first step down the rabbit hole of AI psychosis is humanization of LLM.
Tangental, but my LSP config breaks every few months. I don't bother to fix it anymore, I open an LLM in ~/dotfiles and complain until it works again, usually in a few minutes.
What's interesting is observing how much work this takes. (it gives me much more empathy toward my past self; how was a clumsy human supposed to know and reason about these things!?, especially when I hadn't touched the configs since a few months prior and had forgotten them almost entirely).
Most often there's 10-20 very small programs all working together to give the desired experience. The amount of minutes and tokens required to solve these seemingly simple problems like "My LSP isn't working" is sometimes much more than expected.
I have the exact same anecdote; it makes me wonder if this is a common enough use case that a small language model could be trained and run locally for these kinds of “configuration bullshit problems”.
I feel like a very large percentage of my Claude usage ends up having it automate configuration shit, because historically that has been the part of software engineering I have always hated.
I bet if you look back, you were using a lot less effort in the past than what you see the LLM doing now. They are pretty bad at taking a straight line to the solution.
I mentioned on the main thread, but thought it would be topical here too. Claude Code ships LSP support but only if packaged within a plugin. So there are hoops you have to jump through to get it rather than getting it out of the box. But it might be long term more reliable than ad hoc fixes?
With the skill installed GPT 5.6 usually automatically uses it, and it reduces code exploration time and token usage significantly, for example by just printing the types and functions in a file without bodies, and only expanding when needed.
(note: it also has editing functionality, which doesn't work so well, since the models are heavily tilted towards common editing tools in post training)
Interesting tool, it seems that dedicated skills for these tools are important. Even explicit system-prompt instructions—such as “you must use tool X for code exploration”—often fail to behave as expected.
Another tricky part is that how to evaluate customized toolsets properly and ensure they continue to work perfectly as the underlying model/harness evolves.
Very good tool: I see the go support is not released. Will you be cutting a new release? Will take it for a spin. IMO you can specialize this as a read only AST/search tool, as you said, let models use regular edit tool.
Also some sample commands runs and output, would be useful in the README, for anyone passing by.
Setting up an LSP relies on 2 requirements to properly work:
(a) The LSP's tools being competently built & consistent, and
(b) the LLM using it having been properly trained to use LSPs in general.
Using a native tool like grep has the same 2 assumptions, but
(a) is satisfied due to ossification of grep's core features (a good thing), and
(b) is extra-satisfied because the LLM can be trained to properly use grep specifically, and not "20th variation of grep wrapped behind an LSP, but just different enough to throw curveballs".
On top of that, grep is almost always present in default Linux environments, so its presence is assumed & can be relied upon when needed.
> On a noisy TypeScript repository, it improved F1 by 0.246 and used 12% fewer tokens. The useful predictor was lexical noise, not whether the language had strong static types.
What is "lexical noise", which appears to be a key term in the argument, and why is it not defined? Does it mean that an identifier could show up in, say, a docstring in a non-structured way where a semantic tool would not identify it?
Makes sense to me that LLMs, which are language based, would match better with a tool that searches at a language lever rather than at a deeper structured level.
What has worked well for me is to have a SKILL that instructs to use the LSP more often than not.
LSP works best when using dependencies that are already compiled locally, but if all source is available, yeah... I still don't have a solid answer on which one is best.
But again, for already compiled dependencies (think Java bytecode), without LSP configs, the agent is likely going to attempt to extract binaries from JAR files, use grep and javap, and potentially attempt to decompile the .class files.
Anyone noticed how Gemini in antigravity often devolve into catting scripts into a python file and the script is just a search and replace against an existing file. So and edit operation becomes
The author uses their own harness, I'd want to see if what they are seeing is related to the harness, they should at the minimum also use CC which has direct LSP support.
Their metric is token economy, many of us don't pay per token and capability in terms of difficulty and quality are a more important metrics for the kinds of work I do. I'd want those tested as well.
Just a few days ago I was pondering why an Emacs like environment that provides such powerful tools to examine and manipulate texts is not being used more prominently than say VSCode where you need to be build many features or use plugins or rely on system utilities.
Humans don't use command line lsp tools - why should agents use them?
I use lsp when it is integrated in the IDE - but I haven't seen yet any usable command line tools - even if theoretically it would be very useful to type 'lsp-python -replace-function-name foo bar'.
As a guy who started in lisp I always find LSP to be a weird uncanny valley situation. It's like 99% of a REPL, but it will always have corner cases where that 1% difference is important.
Not only Lisp, but Javascript and Ruby (AKA dynamic languages) devs also rely heavily in REPLs, for me at least the reason is that you don't know what value a variable holts, so you need to inspect them somehow, with typed languages you have less reasons to reach such tool.
iPython was famous for a while but I think its popularity has waned, but yes as you point out really any dynamic language can do a repl; it's just that it's mostly the lisp/scheme and forth families that have leaned so heavily into the idea.
It's the same reason people like symbolic debuggers: you can run your program as many lines at a time as you want and stop and inspect the state whenever you need to. I haven't really used jupyter but I understand the workflow there is very similar.
TBH I frequently have a hard time getting Claude to use LSP at all. I'm not sure if it just breaks or what. It's like 10 items down on my list of random crap to figure out eventually.
It's good, and my agent is using it just fine (explicitly told it to prefer it over just grepping randomly). I use it in Swift, Go, and Node projects. If you use worktrees, make sure to run `codegraph init` post-setup to avoid confusing the robot :)
how come we shifted away from tree-sitter? i still feel like if you're going to let a neural network program, let it generaye an AST. it doesnt matter if it gets the name wrong, as long as the code makes sense.
I’ve had good results with JetBrains AiChat in that really integrates in their IDEs - you can see it using ide tools and i generally found it spends less time thrashing about compared to Claude.
Which language and agent? I used Aider on a Python project which had string references (as opposed to module tree) and cross-repo references. LSP lookups were fast and I hated it when Aider used 2M tokens a day, but grep was more complete.
This article presents some evidence for why Grep might work better but I don’t think it does a great job of explaining why it gets chosen - is it something that was intentionally reinforced during training or was it just because LSP is harder to train on because it’s usually hidden behind some IDE interface
There’s no reason why you couldn’t write a search tool that e.g combines LSP and grep. Or ast-grep, for that matter. It feels like one of those things we haven’t spent much time investigating because grep is good enough
"is it something that was intentionally reinforced during training or was it just because LSP is harder to train on because it’s usually hidden behind some IDE interface"
that's also my doubt, it's much easier to train with grep while only a fraction of project can setup LSP properly.
I am thinking of where the initial training data came from. For example, Claude Code likely collected a substantial number of real-world coding trajectories through its CLI. However, trajectories involving tools such as LSP, MCP, or AST-grep were probably scarce in the dataset.
This lack of representation may also indirectly limit the effectiveness of subsequently generated synthetic data.
It is astonishing how easy it is to see the AI hand at work in the writing. Really, it is almost impossible not to see. I don’t get how these people feel it is appropriate to pass off slop like this and not even bother to edit it.
The author is not a native speaker. The research topic and methodology are manual, along with the first draft of the article, but he eventually did use AI to make the writing "better". It is either this or grammatical slips here and there, which may annoy the same group of people even more.
As a non-native speaker, I really hate how people use this as an excuse. LLMs are good at fixing grammatical errors, so I use them daily as spell checkers. Using LLMs for rewriting is just lazy.
Try:
I'm learning English as a second language. I will provide you with a text, and you have to print the corrected text in a code block so that I can copy it. Under it, there should be two lists: one containing the errors and brief descriptions of them, and another containing tips for improving the text. Do not give your opinion about the text. Focus on grammar.
Do you seriously read the article and not see that it is extremely low signal-to-noise? And full of non-sequiturs and strange unnecessary clarifications? And passed off as a research project.
“Claude, write an article about why agents use grep instead of LSP”
This would have saved everyone the pain of reading this.
As another user pointed out: “Training support is a hypothesis consistent with these results, not something this study proves.” is not a sentence a human would write, nor is it a sentence that a human should ever be made to read. I apologize for reproducing it; the article is chock full of these “gems”.
100%. This article is written by AI. The problem is it makes you doubt the whole thing: Did the person behind this do any work at all? Or is this just content marketing for an AI project - something that used to require time and effort - but now can be done by anyone with a click on a button.
Too bad. Because this is sort of an interesting subject.
Have you never read human writing before? Humans write all kinds of confusingly worded things all the time - it’s why we have editors even for writers who are the cream of the crop.
But also that sentence is entirely fine as it is to me, it’s pretty simple and clear isn’t it?
There’s been an interesting co-evolution that I’ve been experiencing with Claude Code. I’ll ask it to do a task, I’ll watch what it’s doing (often lots of find and grep and ripgrep) and then after the task is complete I’ll ask it if there are any tools that would’ve made the job easier. This has led to tools like fzf and others (notmuch for indexing email, for example). I’ve then taken those tools and figured out how to work them into my own workflow, both CLI and Emacs.
We’ve also collaborated on some Python tooling that takes a rather slow data format that I often have to process and analyze, indexed the whole corpus, and for analysis I can do (or Claude Code can) a single-pass conversion to Parquet which is then queryable with DuckDB. That tool has dramatically improved my turnaround time on one-off analysis tasks and as a Python CLI tool using Typer, the interface is also nicely discoverable for LLM harnesses to work with.
I wonder if you put job classifications along a spreadsheet's leftmost column (e.g. search in file, search in directory), and data types along a spreadsheet's topmost row (e.g. JSON file, macOS-based filesystem), would you end up finding gaps in the intersection of the columns and rows make these new tools you've created less surprising? For example, to "search a file" on a "macOS-based filesystem", you would naturally gravitate towards `grep`. Or to search the contents of a JSON file, you would maybe use `grep`, or perhaps `jq` and a well crafted query. Well, as you expand out the job classifications and data types (or some other abstraction), you start to realize where we don't have tools today. And then Claude can go and create those, perhaps even proposing a faster alternative!
> Or to search the contents of a JSON file, you would maybe use `grep`, or perhaps `jq` and a well crafted query
Try https://github.com/tomnomnom/gron.
I had a similar intuition that we could definitely optimize token efficiency and performance for non-general tasks.
For example, I was handling a lot of infra operations, many of which follow fixed workflows like locating a specific log in certain time range, identify the issue, and apply the appropriate mitigation steps. Since these workflows are entirely command-line based, I added a hook to Claude Code that monitors the commands I repeatedly run, including those invoked by Claude Code itself, and compiles them into reusable scripts(with skills as entries). It can work well.
The takeaway is that we should spend more time reviewing our own work patterns and workflows.
You should disclose that you are the co-author of this paper.
When you use “we” hopefully you are referring to someone on your team. Just be careful, the first step down the rabbit hole of AI psychosis is humanization of LLM.
It’s me and my plucky gang of agents!
Nah as long as you don't give them first-names, I think you should be ok. also:
Pretty pleaser please people don't get your agents registered as direct-reports in the org-chart with HR!
But how else am I supposed to collect their salary?
Is that how you got into this state?
Tangental, but my LSP config breaks every few months. I don't bother to fix it anymore, I open an LLM in ~/dotfiles and complain until it works again, usually in a few minutes.
What's interesting is observing how much work this takes. (it gives me much more empathy toward my past self; how was a clumsy human supposed to know and reason about these things!?, especially when I hadn't touched the configs since a few months prior and had forgotten them almost entirely).
Most often there's 10-20 very small programs all working together to give the desired experience. The amount of minutes and tokens required to solve these seemingly simple problems like "My LSP isn't working" is sometimes much more than expected.
I have the exact same anecdote; it makes me wonder if this is a common enough use case that a small language model could be trained and run locally for these kinds of “configuration bullshit problems”.
I feel like a very large percentage of my Claude usage ends up having it automate configuration shit, because historically that has been the part of software engineering I have always hated.
it's not small but DS4F is cheap enough to have a go on this, it works wonderfully
I bet if you look back, you were using a lot less effort in the past than what you see the LLM doing now. They are pretty bad at taking a straight line to the solution.
This seems like the LLM is vastly overcomplicating things. My entire Neovim config is a single 500 line init.lua and this is enough for LSP to work.
LLMs love overcomplicating things so that is to be expected.
My nvim LSP config is 20 lines and mostly consists of "if you see X language, use Y program as LSP".
I also have been using the same LSP config for approximately 7 years.
I don't get the pain.
I mentioned on the main thread, but thought it would be topical here too. Claude Code ships LSP support but only if packaged within a plugin. So there are hoops you have to jump through to get it rather than getting it out of the box. But it might be long term more reliable than ad hoc fixes?
I've had great success with a tool I wrote that can print sparse ASTs for code.
https://github.com/theduke/smartedit
With the skill installed GPT 5.6 usually automatically uses it, and it reduces code exploration time and token usage significantly, for example by just printing the types and functions in a file without bodies, and only expanding when needed.
(note: it also has editing functionality, which doesn't work so well, since the models are heavily tilted towards common editing tools in post training)
Interesting tool, it seems that dedicated skills for these tools are important. Even explicit system-prompt instructions—such as “you must use tool X for code exploration”—often fail to behave as expected.
Another tricky part is that how to evaluate customized toolsets properly and ensure they continue to work perfectly as the underlying model/harness evolves.
Very good tool: I see the go support is not released. Will you be cutting a new release? Will take it for a spin. IMO you can specialize this as a read only AST/search tool, as you said, let models use regular edit tool.
Also some sample commands runs and output, would be useful in the README, for anyone passing by.
Yeah, I'll do a new release with a whole bunch of local fixes later today.
This look interesting. I have a large mono repo at working ill got it a go.
Setting up an LSP relies on 2 requirements to properly work:
(a) The LSP's tools being competently built & consistent, and
(b) the LLM using it having been properly trained to use LSPs in general.
Using a native tool like grep has the same 2 assumptions, but
(a) is satisfied due to ossification of grep's core features (a good thing), and
(b) is extra-satisfied because the LLM can be trained to properly use grep specifically, and not "20th variation of grep wrapped behind an LSP, but just different enough to throw curveballs".
On top of that, grep is almost always present in default Linux environments, so its presence is assumed & can be relied upon when needed.
> On a noisy TypeScript repository, it improved F1 by 0.246 and used 12% fewer tokens. The useful predictor was lexical noise, not whether the language had strong static types.
What is "lexical noise", which appears to be a key term in the argument, and why is it not defined? Does it mean that an identifier could show up in, say, a docstring in a non-structured way where a semantic tool would not identify it?
Makes sense to me that LLMs, which are language based, would match better with a tool that searches at a language lever rather than at a deeper structured level.
What has worked well for me is to have a SKILL that instructs to use the LSP more often than not.
LSP works best when using dependencies that are already compiled locally, but if all source is available, yeah... I still don't have a solid answer on which one is best.
But again, for already compiled dependencies (think Java bytecode), without LSP configs, the agent is likely going to attempt to extract binaries from JAR files, use grep and javap, and potentially attempt to decompile the .class files.
At the end of the first paragraph there is this sentence:
"Training support is a hypothesis consistent with these results, not something this study proves."
I understand it, but I find the wording very unnatural. To improve readability I would have written it in the active form:
"We cannot prove training explains this result, but this fits the data best."
The text seems LLM generated. Maybe it is mixed with some human editing because it is not too bad. But you still have typical LLM constructs.
Anyone noticed how Gemini in antigravity often devolve into catting scripts into a python file and the script is just a search and replace against an existing file. So and edit operation becomes
1. Use bash to stream code into a new file
2. Run file
3. Check the change step 2 did.
4. Delete file from step 1
I see no one has referenced the written by ai paper itself https://arxiv.org/abs/2608.13568
https://github.com/agentconnect-md/lsp-vs-grep-token-study
The author uses their own harness, I'd want to see if what they are seeing is related to the harness, they should at the minimum also use CC which has direct LSP support.
Their metric is token economy, many of us don't pay per token and capability in terms of difficulty and quality are a more important metrics for the kinds of work I do. I'd want those tested as well.
I wonder if the experiments were done before https://github.com/anthropics/claude-code/issues/30948 was closed.
There still seems to be a lot of issues with lsp/pyright, and might contribute to why grep at least gives a more stable and predictable performance (https://github.com/anthropics/claude-code/issues?q=is%3Aissu...)
it's done before that.
Also, I hope we do experiments for other agents like codex.
Just a few days ago I was pondering why an Emacs like environment that provides such powerful tools to examine and manipulate texts is not being used more prominently than say VSCode where you need to be build many features or use plugins or rely on system utilities.
It's CapEx vs OpEx.
Humans don't use command line lsp tools - why should agents use them?
I use lsp when it is integrated in the IDE - but I haven't seen yet any usable command line tools - even if theoretically it would be very useful to type 'lsp-python -replace-function-name foo bar'.
Claude Code ships LSP support, but only as part of a plugin. I think they’re currently the only vendor to fully support it.
As a guy who started in lisp I always find LSP to be a weird uncanny valley situation. It's like 99% of a REPL, but it will always have corner cases where that 1% difference is important.
I still don't understand what is about REPL that lisp people find particularly useful.
Do you recommend any resources explaining the workflow?
Not only Lisp, but Javascript and Ruby (AKA dynamic languages) devs also rely heavily in REPLs, for me at least the reason is that you don't know what value a variable holts, so you need to inspect them somehow, with typed languages you have less reasons to reach such tool.
iPython was famous for a while but I think its popularity has waned, but yes as you point out really any dynamic language can do a repl; it's just that it's mostly the lisp/scheme and forth families that have leaned so heavily into the idea.
Not only dynamic languages! Java has `jshell` for instance
It's the same reason people like symbolic debuggers: you can run your program as many lines at a time as you want and stop and inspect the state whenever you need to. I haven't really used jupyter but I understand the workflow there is very similar.
TBH I frequently have a hard time getting Claude to use LSP at all. I'm not sure if it just breaks or what. It's like 10 items down on my list of random crap to figure out eventually.
anyone had experience with https://github.com/colbymchenry/codegraph?
It's good, and my agent is using it just fine (explicitly told it to prefer it over just grepping randomly). I use it in Swift, Go, and Node projects. If you use worktrees, make sure to run `codegraph init` post-setup to avoid confusing the robot :)
how come we shifted away from tree-sitter? i still feel like if you're going to let a neural network program, let it generaye an AST. it doesnt matter if it gets the name wrong, as long as the code makes sense.
I’ve had good results with JetBrains AiChat in that really integrates in their IDEs - you can see it using ide tools and i generally found it spends less time thrashing about compared to Claude.
This has not been my experience at all.
Which language and agent? I used Aider on a Python project which had string references (as opposed to module tree) and cross-repo references. LSP lookups were fast and I hated it when Aider used 2M tokens a day, but grep was more complete.
I'd wager it's because most people use grep over an language server
I don't see why someone needs an LLM as an LSP when code is structured data.
This article presents some evidence for why Grep might work better but I don’t think it does a great job of explaining why it gets chosen - is it something that was intentionally reinforced during training or was it just because LSP is harder to train on because it’s usually hidden behind some IDE interface
There’s no reason why you couldn’t write a search tool that e.g combines LSP and grep. Or ast-grep, for that matter. It feels like one of those things we haven’t spent much time investigating because grep is good enough
"is it something that was intentionally reinforced during training or was it just because LSP is harder to train on because it’s usually hidden behind some IDE interface"
that's also my doubt, it's much easier to train with grep while only a fraction of project can setup LSP properly.
I am thinking of where the initial training data came from. For example, Claude Code likely collected a substantial number of real-world coding trajectories through its CLI. However, trajectories involving tools such as LSP, MCP, or AST-grep were probably scarce in the dataset.
This lack of representation may also indirectly limit the effectiveness of subsequently generated synthetic data.
Training a model on ast-grep would be a huge intelligence and performance boost, I think.
I use serena as an MCP for all LSPs and it works great. She does installs hooks into Claude reminding to use LSP tools.
It is astonishing how easy it is to see the AI hand at work in the writing. Really, it is almost impossible not to see. I don’t get how these people feel it is appropriate to pass off slop like this and not even bother to edit it.
Just painful to read.
The author is not a native speaker. The research topic and methodology are manual, along with the first draft of the article, but he eventually did use AI to make the writing "better". It is either this or grammatical slips here and there, which may annoy the same group of people even more.
Hope this clarifies things.
I think it's extremely unlikely that people who are bothered by LLMisms in writing would be more bothered by human grammar slips.
I'd much prefer grammar slips than a LLM-ified post.
> The author is not a native speaker.
As a non-native speaker, I really hate how people use this as an excuse. LLMs are good at fixing grammatical errors, so I use them daily as spell checkers. Using LLMs for rewriting is just lazy.
Try: I'm learning English as a second language. I will provide you with a text, and you have to print the corrected text in a code block so that I can copy it. Under it, there should be two lists: one containing the errors and brief descriptions of them, and another containing tips for improving the text. Do not give your opinion about the text. Focus on grammar.
Format: [TEXT] C: [CONTEXT]
Shut up. Just shut up.
> Avoid generic tangents.
> Please don't post shallow dismissals
> Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage.
See: Hacker News Guidelines
> Please don't post shallow dismissals
I’ll point out you also violate this guideline.
Do you seriously read the article and not see that it is extremely low signal-to-noise? And full of non-sequiturs and strange unnecessary clarifications? And passed off as a research project.
“Claude, write an article about why agents use grep instead of LSP”
This would have saved everyone the pain of reading this.
As another user pointed out: “Training support is a hypothesis consistent with these results, not something this study proves.” is not a sentence a human would write, nor is it a sentence that a human should ever be made to read. I apologize for reproducing it; the article is chock full of these “gems”.
100%. This article is written by AI. The problem is it makes you doubt the whole thing: Did the person behind this do any work at all? Or is this just content marketing for an AI project - something that used to require time and effort - but now can be done by anyone with a click on a button.
Too bad. Because this is sort of an interesting subject.
Have you never read human writing before? Humans write all kinds of confusingly worded things all the time - it’s why we have editors even for writers who are the cream of the crop.
But also that sentence is entirely fine as it is to me, it’s pretty simple and clear isn’t it?
>Shut up. Just shut up.
Try to remain civil, even when you have Big Feelings.