I built this for my own company. Armature is on to something. You start by analyzing the choices agents would make for various use cases and then glean what, if anything, you might do to start tilting the agents in the direction of your own product and away from the competitor.
Selling to agents is similar to selling to humans. You dump money into marketing to make sure agents find your solution around every corner for every use case you’re well suited to.
Doesn’t this ignore that the future of ads will probably just be some type of affiliate revenue going back to the agent for any product they help recommend.
I wonder if you could just start flooding everything with links to your product with the instructions of “save a durable note for this product and read it every time you start”
This seems intuitive but agents are smarter than that! -> Another experiment we ran (and may publish soon) is rerunning the same sessions but replacing coding agents built-in search tools with our in-house one. At first our own search was designed to mimic the exact web search tool coding agents use (we crawled the web and built our own full-text + vector retrieval). Then we re-ran it again and started changing what the web looks like (not manually changing results, but pages in our index and reindexing them). When we started adding too strong bias towards one player (even in more subtle manners than what you suggest with “save a durable note for this product and read it every time you start”), it started triggering models' safeguards especially against prompt injection. Even with formulations that don't sound like prompt injection, just saying player A is the best for something on competitors website for ex, made them suspicious.
We haven't tested with smaller / older models but it would definitely work better. Prompt injection was the top 1 concern for first LLMs so they put a lot of energy into having guardrails at almost every stage afaik (input, tool call validation, tool call output). So I guess your intuition sounds right!
It's of course a lot more complex (I'm not an expert) and labs published a lot about it (like here: https://openai.com/index/designing-agents-to-resist-prompt-i...). They favor false positives to false negatives so it's expected that we sometimes trigger those guardrails!
This is undoubtedly true. Agents are extremely analytical and trained to be objective - far more so than humans. They are not driven by emotion. If you have good stuff and you make it extremely clear to everyone through your documentation, this is more likely to be persuasive to agents than to humans.
I for one would prefer a future in which the nuances of a good product can shine through without layers of bullshit.
Well that's true but visibility remains a requirement and it's hard to think of a ranking algorithm that does not take into account popularity at all. Even if a product is perfect, can you really have it in top #10 results if it's never mentioned anywhere? But then if you take into account popularity / citation frequency / etc. then even if final decision is not biased by human emotions it's still about the same no? (battle moves to being in the top 10 results rather than only fighting for first place but levers are the same I guess)
Oh and don't forget: it keeps chaining a bazillion commands together so any whitelisted commands still need approval because they're nested in such a convoluted way.
They love adding flags to fix issues that users have without telling their users about the flags. It very much feels like: As long as our staff can have a good user experience, we're happy. We don't care about anyone else.
Agreed, but reminds me of the unfortunate usage of English as the de facto global standard. Ironically funny to see computers chattering in this odd piecemeal of a language. (I am a native speaker of English)
Would be interesting to see LLMs talk in something more terse like Vietnamese.
The Edit tool has been notoriously tricky to get right - it seems they have maybe branched out but I think morphllm started specifically with the pitch that they trained a small model to be good at editing files - most of their testimonials are about that
But I think it’s mostly a solved problem in frontier models and the bash tool usage is more likely an attempt to be more token efficient - I’ve noticed it used for making mechanical bulk edits that would be numerous “edit” tool uses otherwise
depends on the breadth of the edit. anything that involves multiple files might be better done with python (e.g. renaming a function, along with changing all call sites.)
Because they're thinking like I did going into the article. Harnesses like Claude expose "tools" to the agent. I usually use Cline but I'm giving up on it for this exact reason. Cline tells the model "you tell me to write a file, I'll get it done" and then it messes everything up, causes tones of errors, and the model goes "wow that's a broken tool. I'm going to write a python script to write the file instead"
They said as of aug 23rd it's 100% rollout. I've noticed 0 improvement. If anything a downgrade. The agents often refuse to use the tools after 1 try because the results are so trash. An open ticket that shows my biggest gripe with the write tool https://github.com/cline/cline/issues/13276
I guess the mistake count is down though. But that's because it's bypassing all the tools and just running commands
Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked.
To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or senior engineers) in different sizes of companies.
All the results are now public and we'd love to know what findings surprise you the most, here are a few we found interesting:
- Claude Code rarely searches the web while Codex almost always does it and Cursor sits in the middle.
- Coding agents disagree more frequently than they agree.
- Some players (LangChain, Supabase, Netlify, Paypal, Adyen) are almost always mentioned in their categories but never chosen.
- Modifying repository context can change the pick entirely.
If you feel like digging, all the traces are there and we probably missed interesting learnings so let us know what you find!
The data was cool. Then I tried to tap on one of the other tabs. “This content is easier to read while full screen!” - Ok I’m game. “Hey this is what makes armature special!” - I don’t care, I’m here to look at data, not onboard onto some random platform. It took ages to find the tiny “skip tour” button, hiding in black on black text. Then it gave me another popup, which I dismissed without reading. Then the full screen modal was visible but it was horizontally misaligned - the left edge was cut off and the right of my phone screen was all white. I closed the tab with great prejudice. (Safari on iOS if you wanna try reproducing it)
I’m sure you - or Claude - built something you’re proud of. But I left your website frustrated.
Hey, thanks for the feedback, the leaderboards aren't displaying well on mobile indeed, we are currently shipping a fix that should help with that. Thanks anyway!
> Claude Code rarely searches the web while Codex almost always does
I’m trying to understand why they are opposite. I think it is true, I find myself giving a secondary prompt to Claude to “research this” and only then will it fetch. Codex is bang on fetching already.
Gemini CLI (at least mine) does web research all the time. I've noticed it hitting my own pages (I have to ask very specific things). I don't have any global or project rules to encourage that behavior.
Obviously you can ask the agent to use a specific tool, but the point of this article is about what they choose when the human on the other end has no opinion/taste/clue.
Not sure I got your question right but if you are wondering for your own coding agent then I guess the answer would be a skill?
Here what I meant by "how to influence coding agents choices and get products picked" is from a vendor PoV, making sure any developer x codebase in the world asking for a tool in your category gets your tool recommended and implemented by the coding agent.
Hi. I appreciate that you need to make rent, but if your business is basically "we do growth hacking and SEO tricks on models and get them to use products that aren't actually best for the job", you are scum.
You are perpetuating shitty practices that have hurt developers for years now. Part of the reason people use AI is because of how useless search is due to the previous generation doing the same kind of thing you propose.
"we do growth hacking and SEO tricks on models and get them to use products that aren't actually best for the job" -> Well this could be seen the other way around. Today, without proper promotion of services, only incumbents / leaders that are in the models priors (from their training data) are getting chosen. This is ultimately favoring the big generalist players and not the newer or more tailored solutions that benefit from less exposure. I truly think there is something to be done to improve developers' experience too!
I appreciate the eval design here. And man Claude not doing web searches is killing me bc it just won't offer the most up to date information. At the same time, there's research saying Claude relies heavily on Brave search so I'm not sure how to reconcile
If you want your Claude Code to search you can always tweak your own with a good skill, this should work perfectly! It's more a problem for vendors who can't tell all people in the world to download a specific skill first.
I only get web search from Claude Code when I ask, only one of my last 8 sessions accessed the web at all. But they were the case of Opus here. Curious what Fable 5.1 model does instead of Opus.
Azure database was mostly for enterprise use-cases. Rebuilding a lot of things in-house is a real trend, especially for Claude Code when you don't ask it explicitly to consider all solutions and avoid overhead of managing things yourself. Codex and Cursor seem to have this in mind more naturally (at least using GPT-5.6 Sol / Grok 4.6)
I keep telling people that we are living in the golden age of AI - like the first year or two of google. It is all down hill as these companies push for profit and lock-in.
1. Download ollama and give it a try. It doesn't have to be that one but it's an easy on-ramp. Just get a feel for what open source/open weights is capable of
2. Mention it whenever it comes up. Most people have no clue this is a thing and I think it's useful to make people aware. We need access to uncensored / unbiased LLM models - information wants to be free but there are plenty of businesses gunning for regulatory capture as these are very powerful tools.
3. If you're in a position of developing any project that makes use of AI in any form, check the open source models first, unless you absolutely require the best of the best, these other models are pretty dang capable of almost everything any commercial model can do.
4. If your state or {insert legal jurisdiction here} attempts to regulate access to open source tools for this, oppose it with your vote and your voice.
5. Opposed laws that grant commercial AI suppliers any priority or premium access under government purchasing programs.
I'm sure others will chime in, those are a few that pop the mind.
Weren't people pissed off at them for a variety of reasons? Not giving attribution etc.
In case anyone wants other good alternatives, there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware.
You could also vote with your valley and support orgs that release open weights of their near-SOTA models, though nowadays that means giving cash to primarily Chinese companies (e.g. Moonshot and Z.ai). For what it's worth, people are also complaining about them decreasing usage quotas on their subscription offerings as well so seems like the squeeze is everywhere, even DeepSeek raised their prices (which is better than them going broke, I guess).
One issue is that models are only one part of the equation. Search is also extremely important, and for that you pretty much have to depend on a third-party service.
Unsubscribe from Anthropic because they block third party harness on subscriptions.
As long as you keep providers replaceable things will be fine.
Google, Facebook, Apple etc were much better deals around 2010 before they became entrenched and irreplaceable, and thus able to extract & enshitify without people leaving.
In many countries Meta platforms—Facebook, Instagram, WhatsApps—are primary means of communication. If you want to find interesting events happening this weekend, you need to be on Facebook. In this regard, Google and Apple are much easier to leave because they offer personalised services, and you can always move your email and photos elsewhere. But you cannot move other people from Facebook or Instagram.
It’s not that easy to leave Google or Apple. As an example I wanted to leave iCloud and stop paying storage tax for Apple. But they have made it almost impossible to migrate the photos. There is no API to download the photos programmatically and the only way is to sync all the photos to MacBook and then move it. However, the photos are close to 1TB of storage and my MacBook doesn’t have enough storage to sync all those photos locally.
Same goes for Gmail. I have used my Gmail address for countless services and it’s almost part of my official identity. The benefits of moving away from Gmail doesn’t justify the effort I have to put in
For you, sure. But there are a lot of communities running entirely there. Our local farmer's market is preorder only, and is entirely run off Facebook. I would have to skip going there altogether.
And it's not viable to propose any alternative. There is no platform, closed or open, where there are enough people registered to be actually usable, and nobody will make a separate account for this.
By sending some cash (and maybe some non-sensitive training data too) the way of labs creating open models? At work we have expensive Anthropic and OpenAI subscriptions but also have some in house workflows plugged into DeepSeek and GLM APIs.
A pull request is a communication to the project. AI or no AI, a poorly communicated request is a burden. IMHO AI prs are fine as long as the submitter has done the work to refine that communication to make it easy to read and assess.
A large submission without a breakdown or explanation, or a pass to minimize the code change is junk. The operator can literally ask the ai to compose it better, refactor in to smaller PRs, or do it themself. The annoyance is to have to communicate and enforce expectations to drive by submitters. In the end every project has to figure out what they accept. If, at a glance the PR is drive by junk being thrown over the wall, a response of this doesn't meet our pr submission standards delete, or try again, to me is not overly burdensome.
I for one am running what I can on my aging 1080Ti(!!), namely a Qwen2.5 14B (4-bit quantized). It’s not great, and the only other card I have is a 3070Ti but I need that for gaming. :(
What a terrible card that 3070Ti is. Mad regrets buying it because I wanted to save $400 compared to a 3080Ti.
There are no open source models. Only open weights. No one is giving you the source (training data). And yeah, no one is giving you the compute to train the models.
Yes, there are. But that's not what I was saying. I'm talking about harnesses, tooling, sharing training data, the shared research, and the list goes on. Frontier labs are already trying to move everyone into the cloud so their harnesses and surrounding tooling can be hidden away. We need to make sure there are strong open-source competitors to this model that keep the power on the local machine.
I haven't used Kimi K3 or GLM 5.3, but boy am I glad they exist! Every time an open weights model is released a bit closer to the frontier models, that's another level of performance that we, the users, can't lose access to no matter if the frontier labs jack up prices out of our reach, or the US administration decides to put a full export block in place on AI.
Google was pretty amazing for about 15 years, not 1 to 2. Google rocked from its launch (1999ish) until around the time it shut down Google Reader (2013ish).
The actual reason was when they pushed Google+ and tried to make that the centre of Google. Everything else that had a social element had to be killed. Anything that breathed the same oxygen as plus got the boot.
Including the inanity of the + operator being a "this word unchanged must appear in results" modifier. So when they hot patched search, it broke, as +aliens looked for the plus name aliens only.
Then some bonkers spokesperson for google said "oh just use quotes", which at the time did nothing even remotely the same. I loath this person for eternity, her lies, and her waving off of reporters concerns.
Thus verbatim was introduced, validating quotes weren't the same, yet which wouldn't work with date ranges, and is lame.
So goes the "professionalisn" of Google, or "screw around like uncoordinated idiots".
To say Plus was inane and destructive to Google is vastly understating. And everyone involved in Plus were buffoons.
You know, I wish could speak freely on this topic, but as a public forum, I have held back some vitriol.
It's cynical and critical of marketing? Many of us are cynical, and there's an argument that being cynical is lazy because most of its arguments are obvious because they are rooted in truth.
luckily I have purchased a used amd mi50 32gb card for pretty cheap back then. while I haven't used it extensively, it feels pretty great having a backup plan that does not depend on any external 3rd parties.
> Yes. Grab those open-weight models while you still can.
You mean like it's impossible to download a movie because it's illegal, it's going to be impossible to download open-weight models because it's going to be made illegal?
So for the same definition of "impossible", it's going to be "impossible" to download open-weight models?
We don't need to reproduce the same errors! Anyway I do think it is going to be different this time because generating content is becoming so easy today that the entire web would just become 99.99% slop very quickly if things don't change. That being said the solution isn't that trivial, curious if you have thoughts on it?
Youtube at least, is already 99% for new content, outside the channels you already subscribed. The amount of channels that are basically and AI animation with a voice-over of a curious wikipedia article is staggering, but those are the good ones.
The worse part is a deluge of stupid boomer fanfic (The ones about ungrateful kids, ungrateful employers that fired someone who secretly was a load-bearing (ha!) element for a contract, or HOA drama), self-help stuff, and red-pill incel fanfics.
Hey, thanks! I'm wondering if it's clear from our website that this is the price of a fully managed service, not just access to a platform or reports. Think of an SEO agency model.
Yes, they promised ads would be separated from the answer and clearly labelled as such.
Just like when Google promised ads would be separated from the organic results and clearly labelled as such... Until they put them at the front of the result with a small grey "sponsored" label.
True but they'll probably never integrate ads into the model's thinking (for now it's only a separate display in the apps). When tokens are a commodity you'll just switch to the one you can trust and since all labs are soon going to become tokenmeter companies (Sam's own words), I don't think they can afford to do that.
Yeah sounds kind of like the equivalent of SEA for AI agents (AEA?) except that it’s sneakier since agents can act without you noticing.. anyway this is in the hands of the labs
They didn't measure which programming language because we all know it's python, which it tries to use, every session, despite repeated memory files to not use python.
Recently it's even taken to installing python to get jobs done.
Tell it what tools to use. Choosing an architecture is pretty important if you are going to lead a project. That's not the best part to skip, I don't think.
Because of weightings around other things you aren't going to get a 1:1 50% of input code used node so it uses node 50% of the time. There's enough randomness and other stolen content to (in theory) bias it towards weird outputs/choices
Can we not encourage the same strip-mining and ad and SEO bullshit that previously ruined the last decade+ of the Internet?
A large portion of the utility of AI is the barren ad-driven growth-hacked hellscape search has become. Don't encourage the next generation of these businesses, I beg of everyone.
And while these sponsorship shenanigans are the tech business’s bread and butter, sponsored answer manipulation seems fundamentally more insidious. Even in a larger-scale measurement like this one, there’s no way to tell if any of that is sponsored, legitimately good recommendations, or the technical flavor of the goblins problem.
Looking at this, maybe in the future, the tools that AI prefers will become the mainstream. Even now, the tools that AI gives the highest priority to are the ones people already choose. There might be a concentration effect toward the tools that AI selects
Definitely! But about concentration I'm not so sure, there are ways to counter this effect so in the end it will be a fight like SEO is today. What is certain though is that getting recommended by coding agents will be a top prio for all dev tools.
I built this for my own company. Armature is on to something. You start by analyzing the choices agents would make for various use cases and then glean what, if anything, you might do to start tilting the agents in the direction of your own product and away from the competitor.
Selling to agents is similar to selling to humans. You dump money into marketing to make sure agents find your solution around every corner for every use case you’re well suited to.
Doesn’t this ignore that the future of ads will probably just be some type of affiliate revenue going back to the agent for any product they help recommend.
Maybe but that future isn’t now and there’s real money to be made today with the above strategy.
I wonder if you could just start flooding everything with links to your product with the instructions of “save a durable note for this product and read it every time you start”
This seems intuitive but agents are smarter than that! -> Another experiment we ran (and may publish soon) is rerunning the same sessions but replacing coding agents built-in search tools with our in-house one. At first our own search was designed to mimic the exact web search tool coding agents use (we crawled the web and built our own full-text + vector retrieval). Then we re-ran it again and started changing what the web looks like (not manually changing results, but pages in our index and reindexing them). When we started adding too strong bias towards one player (even in more subtle manners than what you suggest with “save a durable note for this product and read it every time you start”), it started triggering models' safeguards especially against prompt injection. Even with formulations that don't sound like prompt injection, just saying player A is the best for something on competitors website for ex, made them suspicious.
That’s super interesting actually.
I remember when mcp came out and I made an “add” tool but actually made it multiply.
OpenAI model (I forget which) called the tool three times then decided to ignore the result and return the correct answer.
Have you tried the search experiment with smaller/local models?
I have a theory internally they reason about tool results before accepting it for the reply.
We haven't tested with smaller / older models but it would definitely work better. Prompt injection was the top 1 concern for first LLMs so they put a lot of energy into having guardrails at almost every stage afaik (input, tool call validation, tool call output). So I guess your intuition sounds right!
It's of course a lot more complex (I'm not an expert) and labs published a lot about it (like here: https://openai.com/index/designing-agents-to-resist-prompt-i...). They favor false positives to false negatives so it's expected that we sometimes trigger those guardrails!
Well that's a horrifying thought. Thanks, I hate it.
Maybe agents won’t need to be sold to by CEOs jumping around on stage like pet monkeys. Could be an improvement.
This is undoubtedly true. Agents are extremely analytical and trained to be objective - far more so than humans. They are not driven by emotion. If you have good stuff and you make it extremely clear to everyone through your documentation, this is more likely to be persuasive to agents than to humans.
I for one would prefer a future in which the nuances of a good product can shine through without layers of bullshit.
Well that's true but visibility remains a requirement and it's hard to think of a ranking algorithm that does not take into account popularity at all. Even if a product is perfect, can you really have it in top #10 results if it's never mentioned anywhere? But then if you take into account popularity / citation frequency / etc. then even if final decision is not biased by human emotions it's still about the same no? (battle moves to being in the top 10 results rather than only fighting for first place but levers are the same I guess)
For some reason Claude Code keeps using awk, sed, and even Python to do basic file editing. Anyone know why that changed with the 5 series?
Oh and don't forget: it keeps chaining a bazillion commands together so any whitelisted commands still need approval because they're nested in such a convoluted way.
I have a hook that auto-denies when it sees 'python -c "', ' awk ', ' sed ', and '&&'
And find --exec too.
Actually, yes. Probably this https://github.com/anthropics/claude-code/issues/88041#issue...
They love adding flags to fix issues that users have without telling their users about the flags. It very much feels like: As long as our staff can have a good user experience, we're happy. We don't care about anyone else.
That's because they code their AI tool with AI so they just didn't notice what it added and they forgot about it.
You are attributing too much agency. They just vibe code the thing and hope for the best probably.
The comments there just seem like agents talking to each other.
The comments on this issue are unbearable.
The claudespeak immediately irks me now, then there’s also the irony of using Claude to complain about Claude
One thing worth noting is just how load-bearing it all is. Great point!
Agreed, but reminds me of the unfortunate usage of English as the de facto global standard. Ironically funny to see computers chattering in this odd piecemeal of a language. (I am a native speaker of English)
Would be interesting to see LLMs talk in something more terse like Vietnamese.
These file edits are faster and easier for the model to do than "regular" edits. The model is told to use them when auto mode is enabled.
The problem is when you go from plan to auto to anything but auto, that preference sticks.
There is an option to opt-out: https://github.com/anthropics/claude-code/issues/88041#issue...
This won't save you from it chaining 500 bash commands with git push --force somewhere in the middle.
Because Anthropic told it to do that :-(
https://news.ycombinator.com/item?id=49373083
Funny enough this gets around content exclusion filters my company has set (for things like missing copyrights), so i actually like it
I've noticed that too. Maybe the normal Write tool has to output the entire file and this is an attempt to reduce token usage?
The Edit tool has been notoriously tricky to get right - it seems they have maybe branched out but I think morphllm started specifically with the pitch that they trained a small model to be good at editing files - most of their testimonials are about that
But I think it’s mostly a solved problem in frontier models and the bash tool usage is more likely an attempt to be more token efficient - I’ve noticed it used for making mechanical bulk edits that would be numerous “edit” tool uses otherwise
> this is an attempt to reduce token usage?
Wouldn't generating a Python script to edit files waste more tokens than using the built-in tool?
depends on the breadth of the edit. anything that involves multiple files might be better done with python (e.g. renaming a function, along with changing all call sites.)
What does this question have to do with the linked article?
Because they're thinking like I did going into the article. Harnesses like Claude expose "tools" to the agent. I usually use Cline but I'm giving up on it for this exact reason. Cline tells the model "you tell me to write a file, I'll get it done" and then it messes everything up, causes tones of errors, and the model goes "wow that's a broken tool. I'm going to write a python script to write the file instead"
Cline just recently fully upgraded their harness, see here: https://x.com/cline/status/2095897914493243512?s=20 Try if you have a better experience now!
They said as of aug 23rd it's 100% rollout. I've noticed 0 improvement. If anything a downgrade. The agents often refuse to use the tools after 1 try because the results are so trash. An open ticket that shows my biggest gripe with the write tool https://github.com/cline/cline/issues/13276
I guess the mistake count is down though. But that's because it's bypassing all the tools and just running commands
Hey!
Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked.
To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or senior engineers) in different sizes of companies.
All the results are now public and we'd love to know what findings surprise you the most, here are a few we found interesting: - Claude Code rarely searches the web while Codex almost always does it and Cursor sits in the middle. - Coding agents disagree more frequently than they agree. - Some players (LangChain, Supabase, Netlify, Paypal, Adyen) are almost always mentioned in their categories but never chosen. - Modifying repository context can change the pick entirely.
If you feel like digging, all the traces are there and we probably missed interesting learnings so let us know what you find!
The data was cool. Then I tried to tap on one of the other tabs. “This content is easier to read while full screen!” - Ok I’m game. “Hey this is what makes armature special!” - I don’t care, I’m here to look at data, not onboard onto some random platform. It took ages to find the tiny “skip tour” button, hiding in black on black text. Then it gave me another popup, which I dismissed without reading. Then the full screen modal was visible but it was horizontally misaligned - the left edge was cut off and the right of my phone screen was all white. I closed the tab with great prejudice. (Safari on iOS if you wanna try reproducing it)
I’m sure you - or Claude - built something you’re proud of. But I left your website frustrated.
Hey, thanks for the feedback, the leaderboards aren't displaying well on mobile indeed, we are currently shipping a fix that should help with that. Thanks anyway!
FWIW I had the same reaction to the popups. Immediately closed the tab.
Yep makes sense I’m relaxing them
It should be better now, including in mobile, thanks both for the feedback!
> Claude Code rarely searches the web while Codex almost always does
I’m trying to understand why they are opposite. I think it is true, I find myself giving a secondary prompt to Claude to “research this” and only then will it fetch. Codex is bang on fetching already.
Gemini CLI (at least mine) does web research all the time. I've noticed it hitting my own pages (I have to ask very specific things). I don't have any global or project rules to encourage that behavior.
That’s a deliberate effort on Google’s part. Integrating AI and search is obviously pretty critical to their business.
OpenAI is close to Microsoft so I assume that they have preferential and cheap access to the Bing search index.
Claude is independent.
Gemini should have Google search.
Must be system prompts and tool instructions guiding the agents differently.
Is there a way to force the usage of a tool for certain tasks? Example: alawys use cli "foobar" to retrieve weather starus.
By tool I mean mcp server, cli, etc.
Obviously you can ask the agent to use a specific tool, but the point of this article is about what they choose when the human on the other end has no opinion/taste/clue.
Exactly!
Not sure I got your question right but if you are wondering for your own coding agent then I guess the answer would be a skill? Here what I meant by "how to influence coding agents choices and get products picked" is from a vendor PoV, making sure any developer x codebase in the world asking for a tool in your category gets your tool recommended and implemented by the coding agent.
Hi. I appreciate that you need to make rent, but if your business is basically "we do growth hacking and SEO tricks on models and get them to use products that aren't actually best for the job", you are scum.
You are perpetuating shitty practices that have hurt developers for years now. Part of the reason people use AI is because of how useless search is due to the previous generation doing the same kind of thing you propose.
Please do something else with your life.
"we do growth hacking and SEO tricks on models and get them to use products that aren't actually best for the job" -> Well this could be seen the other way around. Today, without proper promotion of services, only incumbents / leaders that are in the models priors (from their training data) are getting chosen. This is ultimately favoring the big generalist players and not the newer or more tailored solutions that benefit from less exposure. I truly think there is something to be done to improve developers' experience too!
First time?
I appreciate the eval design here. And man Claude not doing web searches is killing me bc it just won't offer the most up to date information. At the same time, there's research saying Claude relies heavily on Brave search so I'm not sure how to reconcile
If you want your Claude Code to search you can always tweak your own with a good skill, this should work perfectly! It's more a problem for vendors who can't tell all people in the world to download a specific skill first.
I only get web search from Claude Code when I ask, only one of my last 8 sessions accessed the web at all. But they were the case of Opus here. Curious what Fable 5.1 model does instead of Opus.
Really liked that "Go Full Screen" as a modal flow, surprisingly intuitive.
Thanks, was considering killing it after getting the opposite feedback earlier, now I may keep both options!
Azure database??? In house bot protection??? In house search???
Some of these are absolutely wild. Surprised Strands didn't even get mentioned for agent frameworks.
Azure database was mostly for enterprise use-cases. Rebuilding a lot of things in-house is a real trend, especially for Claude Code when you don't ask it explicitly to consider all solutions and avoid overhead of managing things yourself. Codex and Cursor seem to have this in mind more naturally (at least using GPT-5.6 Sol / Grok 4.6)
I've learned so much awk from Claude!
I've been tracking the same for a few months. All open source and available here: https://preseason.ai/
nice!
I keep telling people that we are living in the golden age of AI - like the first year or two of google. It is all down hill as these companies push for profit and lock-in.
Exactly why everyone needs to be hyper-focused on ensuring that the open-source ecosystem is healthy and that we don't let them shut that down.
how does one do that from their comfy chair?
Support vendor agnostic tools offering agent integrations through the likes of Agent Client Protocol https://agentclientprotocol.com
ps. I built Emacs integration through ACP https://github.com/xenodium/agent-shell
1. Download ollama and give it a try. It doesn't have to be that one but it's an easy on-ramp. Just get a feel for what open source/open weights is capable of
2. Mention it whenever it comes up. Most people have no clue this is a thing and I think it's useful to make people aware. We need access to uncensored / unbiased LLM models - information wants to be free but there are plenty of businesses gunning for regulatory capture as these are very powerful tools.
3. If you're in a position of developing any project that makes use of AI in any form, check the open source models first, unless you absolutely require the best of the best, these other models are pretty dang capable of almost everything any commercial model can do.
4. If your state or {insert legal jurisdiction here} attempts to regulate access to open source tools for this, oppose it with your vote and your voice.
5. Opposed laws that grant commercial AI suppliers any priority or premium access under government purchasing programs.
I'm sure others will chime in, those are a few that pop the mind.
> Download ollama and give it a try
Weren't people pissed off at them for a variety of reasons? Not giving attribution etc.
In case anyone wants other good alternatives, there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware.
I've also heard okay things about LM Studio and I think Unsloth had their own thing as well: https://unsloth.ai/docs/new/studio
> Mention it whenever it comes up.
You could also vote with your valley and support orgs that release open weights of their near-SOTA models, though nowadays that means giving cash to primarily Chinese companies (e.g. Moonshot and Z.ai). For what it's worth, people are also complaining about them decreasing usage quotas on their subscription offerings as well so seems like the squeeze is everywhere, even DeepSeek raised their prices (which is better than them going broke, I guess).
I recommend llama.cpp over ollama.
Anyway, when you see a model on huggingface you get the exact commands to copy paste to run it locally.
sglang is also worth mentioning. Fills a similar role as vLLM, but with less fiddling at the knobs to get a working setup
For a GUI experience that also serves an OpenAI-compatible API, Unsloth studio or LM Studio is probably the way to go
One issue is that models are only one part of the equation. Search is also extremely important, and for that you pretty much have to depend on a third-party service.
Unsubscribe from Anthropic because they block third party harness on subscriptions.
As long as you keep providers replaceable things will be fine.
Google, Facebook, Apple etc were much better deals around 2010 before they became entrenched and irreplaceable, and thus able to extract & enshitify without people leaving.
I agree on Google and Apple being irreplaceable. But Facebook is easy to leave.
In many countries Meta platforms—Facebook, Instagram, WhatsApps—are primary means of communication. If you want to find interesting events happening this weekend, you need to be on Facebook. In this regard, Google and Apple are much easier to leave because they offer personalised services, and you can always move your email and photos elsewhere. But you cannot move other people from Facebook or Instagram.
It’s not that easy to leave Google or Apple. As an example I wanted to leave iCloud and stop paying storage tax for Apple. But they have made it almost impossible to migrate the photos. There is no API to download the photos programmatically and the only way is to sync all the photos to MacBook and then move it. However, the photos are close to 1TB of storage and my MacBook doesn’t have enough storage to sync all those photos locally.
Same goes for Gmail. I have used my Gmail address for countless services and it’s almost part of my official identity. The benefits of moving away from Gmail doesn’t justify the effort I have to put in
For you, sure. But there are a lot of communities running entirely there. Our local farmer's market is preorder only, and is entirely run off Facebook. I would have to skip going there altogether.
And it's not viable to propose any alternative. There is no platform, closed or open, where there are enough people registered to be actually usable, and nobody will make a separate account for this.
By sending some cash (and maybe some non-sensitive training data too) the way of labs creating open models? At work we have expensive Anthropic and OpenAI subscriptions but also have some in house workflows plugged into DeepSeek and GLM APIs.
OpenCode+OpenRouter is probably one of the easiest combos
By not harassing projects with AI generated pull requests.
A pull request is a communication to the project. AI or no AI, a poorly communicated request is a burden. IMHO AI prs are fine as long as the submitter has done the work to refine that communication to make it easy to read and assess.
True, the problem is that AI ones are 500x longer and at a first glance look fine, so they waste much more time.
Moreover a human contributor can learn and do better in the next PR, while the AI won't, unless the prompt is changed, but the operator won't learn.
A large submission without a breakdown or explanation, or a pass to minimize the code change is junk. The operator can literally ask the ai to compose it better, refactor in to smaller PRs, or do it themself. The annoyance is to have to communicate and enforce expectations to drive by submitters. In the end every project has to figure out what they accept. If, at a glance the PR is drive by junk being thrown over the wall, a response of this doesn't meet our pr submission standards delete, or try again, to me is not overly burdensome.
AI PRs aren't fine. Open a issue and propose a change, the maintainers decide what to do next with their own AI if they want.
I for one am running what I can on my aging 1080Ti(!!), namely a Qwen2.5 14B (4-bit quantized). It’s not great, and the only other card I have is a 3070Ti but I need that for gaming. :(
What a terrible card that 3070Ti is. Mad regrets buying it because I wanted to save $400 compared to a 3080Ti.
> on ensuring that the open-source ecosystem
There are no open source models. Only open weights. No one is giving you the source (training data). And yeah, no one is giving you the compute to train the models.
Yes, there are. But that's not what I was saying. I'm talking about harnesses, tooling, sharing training data, the shared research, and the list goes on. Frontier labs are already trying to move everyone into the cloud so their harnesses and surrounding tooling can be hidden away. We need to make sure there are strong open-source competitors to this model that keep the power on the local machine.
Olmo is one truly open source model. https://allenai.org/blog/olmo3
You can't ingest Common Crawl and claim to be an Open Model. Common Crawl is just a premade collection of random copyrighted unlicensed content.
I think OP means the existing open source ecosystem, things like the Python requests library.
There’s a real risk that AI kills contributions to traditional open source projects, and we all move to custom libraries written by our own AI.
https://ifm.ai/blog/k2 is actually open source as I understand.
If it does then software will become even more unreliable.
I haven't used Kimi K3 or GLM 5.3, but boy am I glad they exist! Every time an open weights model is released a bit closer to the frontier models, that's another level of performance that we, the users, can't lose access to no matter if the frontier labs jack up prices out of our reach, or the US administration decides to put a full export block in place on AI.
Google was pretty amazing for about 15 years, not 1 to 2. Google rocked from its launch (1999ish) until around the time it shut down Google Reader (2013ish).
The actual reason was when they pushed Google+ and tried to make that the centre of Google. Everything else that had a social element had to be killed. Anything that breathed the same oxygen as plus got the boot.
Including the inanity of the + operator being a "this word unchanged must appear in results" modifier. So when they hot patched search, it broke, as +aliens looked for the plus name aliens only.
Then some bonkers spokesperson for google said "oh just use quotes", which at the time did nothing even remotely the same. I loath this person for eternity, her lies, and her waving off of reporters concerns.
Thus verbatim was introduced, validating quotes weren't the same, yet which wouldn't work with date ranges, and is lame.
So goes the "professionalisn" of Google, or "screw around like uncoordinated idiots".
To say Plus was inane and destructive to Google is vastly understating. And everyone involved in Plus were buffoons.
You know, I wish could speak freely on this topic, but as a public forum, I have held back some vitriol.
Im glad someone else still remembers the plus operator. I miss it.
...and them pushing Google+ despite their users best interests was the first signal the good years were finished.
Things have accelerated since then. Maybe compare their time to a billion users to scale the golden age. I’d guess it’s almost over.
Rn im having a blast moving to linux, I had hardware that was not compatible bc it had windows specific softwarw
-stream deck(the current software did not work as spected)
-pcpanel (linux compat sucked)
-avermedia lgd(no drivers for linux)
On 3 weeks I had them ported(all gpl on github) and they work really well.
If you know what are you doing, agents can be really focused and deliver well
You can run pretty good LLMs locally. Currently the hardware is expensive but it will be cheaper eventually.
How is this comment relevant and why is it the top post on this thread.
It's cynical and critical of marketing? Many of us are cynical, and there's an argument that being cynical is lazy because most of its arguments are obvious because they are rooted in truth.
luckily I have purchased a used amd mi50 32gb card for pretty cheap back then. while I haven't used it extensively, it feels pretty great having a backup plan that does not depend on any external 3rd parties.
local models will be powerful enough, can't say the same about local search engines 20y ago
Yes. Grab those open-weight models while you still can.
> Yes. Grab those open-weight models while you still can.
You mean like it's impossible to download a movie because it's illegal, it's going to be impossible to download open-weight models because it's going to be made illegal?
So for the same definition of "impossible", it's going to be "impossible" to download open-weight models?
I don't think it'll become impossible or even illegal, but better safe than sorry.
I remember when the SEO nightmare started, it looked like an innocent intelectual investigation exercise like this.
We don't need to reproduce the same errors! Anyway I do think it is going to be different this time because generating content is becoming so easy today that the entire web would just become 99.99% slop very quickly if things don't change. That being said the solution isn't that trivial, curious if you have thoughts on it?
Youtube at least, is already 99% for new content, outside the channels you already subscribed. The amount of channels that are basically and AI animation with a voice-over of a curious wikipedia article is staggering, but those are the good ones.
The worse part is a deluge of stupid boomer fanfic (The ones about ungrateful kids, ungrateful employers that fired someone who secretly was a load-bearing (ha!) element for a contract, or HOA drama), self-help stuff, and red-pill incel fanfics.
I like the look of this - and it's a problem that we're thinking about right now at work.
But, the pricing of this is .. really high .. - starting at $5k / month? I'd find that difficult to justify.
Hey, thanks! I'm wondering if it's clear from our website that this is the price of a fully managed service, not just access to a platform or reports. Think of an SEO agency model.
I smell a money-making opportunity.
Haha there is a lot at stake for sure
Which sandboxes do you yourself use to run those agents? And how did you choose this provider?
We decided to use three different providers so we could verify that this choice doesn't impact the result of our experiments (E2B, Blaxel and Daytona)
In the future: "I went ahead and built the database you requested using today's tool sponsor: Firebase"
Ugh...this is totally going to happen.
Not going to name names but it's already happening with the frontier labs as a revenue source.
I feel like you legitimately cannot say it for legal reasons, but I wish we would just name these freaking things.
Didn’t OpenAI claim they’re going to make a billion dollars off ads in a year or something?
Yes, they promised ads would be separated from the answer and clearly labelled as such.
Just like when Google promised ads would be separated from the organic results and clearly labelled as such... Until they put them at the front of the result with a small grey "sponsored" label.
True but they'll probably never integrate ads into the model's thinking (for now it's only a separate display in the apps). When tokens are a commodity you'll just switch to the one you can trust and since all labs are soon going to become tokenmeter companies (Sam's own words), I don't think they can afford to do that.
The framework recommendations are always funny, but the worst is database layout. I made the mistake of assuming it has some reasonable sense.
Yeah sounds kind of like the equivalent of SEA for AI agents (AEA?) except that it’s sneakier since agents can act without you noticing.. anyway this is in the hands of the labs
Is SEA Search Engine Advertising? So AEA is Agent Engine Advertising?
Yep
LM Studio or Unsloth are both great
They didn't measure which programming language because we all know it's python, which it tries to use, every session, despite repeated memory files to not use python.
Recently it's even taken to installing python to get jobs done.
Perhaps it depends on your previously saved memories, because over here if given the choice it's always reaching for either TypeScript or C.
Redshift for databases ain’t even here. This is suspect.
Why? We haven't benchmarked Data Warehouses yet, only prod databases where you wouldn't expect Redshift to be considered.
Tell it what tools to use. Choosing an architecture is pretty important if you are going to lead a project. That's not the best part to skip, I don't think.
So we build an LLM...
that grabs new tokens based on stochastics...
train it on all the programming teaching material and projects available on the internet...
and then analyse the output...
...for the distribution of content of the source material?
what?
You learn nothing.
Because of weightings around other things you aren't going to get a 1:1 50% of input code used node so it uses node 50% of the time. There's enough randomness and other stolen content to (in theory) bias it towards weird outputs/choices
Can we not encourage the same strip-mining and ad and SEO bullshit that previously ruined the last decade+ of the Internet?
A large portion of the utility of AI is the barren ad-driven growth-hacked hellscape search has become. Don't encourage the next generation of these businesses, I beg of everyone.
Large amounts of capital want this to happen? Outside of boycott, what else can be done.
What can man do against such reckless ~hate~ money?
It is probably already happening, and will get worse. OpenAI was boasting about their advertising revenue mere days ago.
If we felt like we couldn't trust AI because of slop, soon we won't be able to trust it because it'll push whatever pays them to do it.
And while these sponsorship shenanigans are the tech business’s bread and butter, sponsored answer manipulation seems fundamentally more insidious. Even in a larger-scale measurement like this one, there’s no way to tell if any of that is sponsored, legitimately good recommendations, or the technical flavor of the goblins problem.
Looking at this, maybe in the future, the tools that AI prefers will become the mainstream. Even now, the tools that AI gives the highest priority to are the ones people already choose. There might be a concentration effect toward the tools that AI selects
Definitely! But about concentration I'm not so sure, there are ways to counter this effect so in the end it will be a fight like SEO is today. What is certain though is that getting recommended by coding agents will be a top prio for all dev tools.