I hope the future of AI isn't this sort, where companies provide the user/customer with an interface to an AI that can do things for the user... I would much prefer that companies instead provide an interface FOR an AI, and the user brings their own AI which connects to that interface.
In other words, provide my AI with tools, instead of providing me an AI that uses your tools.
That way, my AI can bring all the context it needs, and I can bring all of the settings and knowledge about what I want with me. I don't want a fractured world of tons of AIs i interact with where I have to explain all the fundamental information about what I want and how I work every time.
This also has the benefit of sidestepping the issue the essay is talking about. You provide a consistent tool, and the AI weirdness is not your issue anymore. You don't have to worry about solving for all the weird ways people prompt the AI, or the ways they break.
There's no interop between video conferences services, no BYO-video-conferencing app, why would this be any different? It took government regulation to allow telephones other than those provided by the phone company to be plugged into the phone network. Can't control you or usage if they just provide dumb pipes/interfaces that anyone's tool and interact with.
All of those browser and computer use agents (available from OpenAI, Anthropic, Meta, and X.ai already) represent an interoperability workaround already.
I think this is basically the motivation behind MCP servers, so you're in good company.
But I've spoken with many people at companies who've decided to add an in-UI agent to their apps, and I don't think this trend will persist. Absolutely AI/agents will increasingly become a major component of interfaces, but the current common incarnation of "Clippy for X" feels like a kludged bridge between an app that was not designed from first principles for agents (and often was poorly designed for humans) and the impulse to be "AI native".
From my experience, some of the most successful niches for this sort of UI so far have been in apps that were already well suited to it. I'm thinking specifically about analytics/dashboarding/"this is a portal for you to query things easily" software. They already started with a lot of the elements you want: visible provenance of the agents actions via the queries it writes, a malleable interface that you already expect to be customizable and ephemeral, and most importantly, "navigation" that is genuinely difficult for many users (in the sense that "navigating" can mean "querying specific data"). The agent provides a ton of value to users and its actions are intuitive and legible.
But the agents you see right now in a lot of apps that do things like navigate you to the right page by... sending you a link, which you could have clicked from the navbar? Or worse, which hijack your navigation and throw you on a page you're unfamiliar with, and where you have no sense of place or how to make your way to/from? I don't think they're particularly long for this world.
IMO it's a fundamental tension. AI does not fit well inside a product. It works best outside of it. But that means the product - not its functionality, but the business around it - becomes irrelevant. Which none of the product vendors want.
Also the motivation behind CLIs and SDKs, right? And the way agents write Python regularly, these days, I am hopeful what can be offloaded to deterministic systems will be.
A better internet is one in which LLMs enable everyone to ride their own custom Chromium (or whatever) flavor and the "frontend" of > 90% of sites - especially e-commerce - is generated specific to the user given their configuration of their personal LLM intermediary.
Companies will increasingly not ship frontends anymore, maybe just themed components that most users will ignore. "Pages" naturally become synthetic amalgamations of the various things you are interested in from various sources. Your preferred UI/UX comes for free courtesy of your LLM intermediary.
Everything is an API. Regulation will probably be needed to enforce this on the data access side given the incentives for companies to resist despite the benefit to the consumer. A relatively easy solution is to make it part of the requirements to take payments online, which is already highly regulated. The goal would be that essentially to operate anything approaching e-commerce online you must allow consumer AI intermediaries sufficient access that they could independently construct whatever your current frontend is from the publicly available API/MCP endpoints.
The tricky part of engineering around privacy/personas, account creation, etc. will be relatively easy following such state intervention. Take payments - virtual credit cards are already a useful way to protect data and mitigate risks when dealing with multiple vendors. This sort of API is a natural fit for a world in which companies are not allowed to box you into their horrible UIs, most of which are simply lazy attempts at copying the most profitable anti patterns of their competitors. Let your personal agent generate a unique card for each vendor behind the scenes. Why should you have to care?
How much better the internet would be if every UI/UX was in a meaningful sense, your own.
I feel like this is a bit too-optimistically describing an outcome which is unlikely, given how consumers have been slowly losing the same war for decades already.
The elephant in the room and the really "tricky part" involves powerful advertising and copyright-holding companies. They will (continue to) interpret that kind of user-agency as an existential threat. Their businesses revolve around deciding and knowing what you actually see and do with your computer, and their toolkit isn't just technical, but also economic and political.
That's why we already have problems with DRM, "felony contempt of business model" under the DMCA, browser fingerprinting, and everybody surrendering to arbitrary JS code controlling their computer. We capitulate not just to see an article or talk to a friend, but also to avoid our technical details getting into a secret un-appeal-able "suspicious" blacklist.
Some companies are taking the opportunity to use the MCP server to send messages to the agent that the user won't necessarily see to protest. Agents are generally incredibly gullible and a huge opportunity for this type of thing.
That basically describes the Google shopping ribbon that appears with searches for products. When Google was good, you'd say a website doesn't exist if you can't find it in the 1st page of Google. The shopping ribbon provides offers from websites that aren't on the first page of my search result. Although I don't think you can pay without visiting the seller's website, this means that these are effectively stores that are more visible through the APIs they use to talk to Google's recommender engine. These suggestions are also no doubt personalized.
This also describes adsense, or Amazon or AliExpress, so it is clearly a successful model. If there is much left of other models. The only thing we have to hopeful about is that the interface is standardized so commodified SWE can be used to make your own Amazon or AliExpress.
I'm currently building something like this, basically converting every service to a standardized API that you can consume in a structured manner. Not focused on AI, but focused on normal users with special requirements on usability.
Basically the ad blocking approach has failed, and one needs to recognize that an allow-list based approach is needed when consuming internet-based offerings. Pick the cherries out of each website, leave the trash behind, and then build a custom per-website GUI around the things that you actually want to consume.
You shouldn't need LLM and AI to do such things. Even if such things might be useful to some people, I might want to use the API without LLM/AI and just to read the API documentation and then to make my own, and some other people might want to do similarly. In the specific case of e-commerce, I had been making my own specification of a file format for e-commerce (which is independent of the protocol), which is not intended to use with LLM/AI, but instead is intended to avoid many kind of dishonest business while also being flexible and that you could make and use your own software or some other implementaiton instead of being forced to use their UI/UX.
We could have a sort of “Representational State Transfer” using HTTP to represent the state of resources in both a human readable and machine readable format.
I don’t think that will happen. It sounds good, and I would like it if things operated that way - but from the company perspective how do they, for example, have a tool call that gives the customer a discount, without it getting used when it shouldn’t?
Companies want ai to replace human customer service decision making, which means it can’t just be an api that an external agent can interact with, because it needs private knowledge of company processes and access to capabilities that are abusable.
But we’re already at the point where if you manage to talk to a human, mostly you end up speaking to someone with no actual power to resolve your issue - so i think basically the future is just going to suck
> from the company perspective how do they, for example, have a tool call that gives the customer a discount, without it getting used when it shouldn’t?
I might be misunderstanding your point but I think you're in agreement with the gp, as in they are arguing for a locked down interface behind which the AI sits, narrowed to use within whatever particular operations the user can access, as opposed to a wide open AI interface with access to any operation (in principle).
So in your example there's no way to ask the AI for any old discount because the interface for doing so is locked down to just the discounts you can access, although the AI may be able to apply a discount you didn't ask for dynamically, if allowed, to give the customer a better experience.
Of course both versions can be implemented securely, it's just probably in general a smaller attack surface if you provide a tighter interface first.
> have a tool call that gives the customer a discount, without it getting used when it shouldn’t?
without comment on the rest of your post, a lot of discounts are just dark patterns, buy 2 for $10 or one for $6 is one of the oldest dark patterns around, and a lot of people fall for it, when they really didn't need that much whatever it was they were buying. In fact there are laws outlawing these practices coming into force for various types of products (alcohol, sugar) .
Other types of discounts like loyalty are more vague personally I think are a minor dark pattern. Negotiated discount on bulk purchases b2b are a different class, but it could be argued that you should only get the discount that the seller saves on logistics.
I'm just pointing this out because its very normalized, 'signup and get 3 months free', 'buy 2 get one free' etc, people dont bat an eyelid at, the same as advertising , but they are all just the earliest versions of dopamine hacking and dark patterns that consensus now is coming around to say probably isn't the best thing when scaled beyond a simple one to one interaction. Why are snapchat dopamine hacking with streaks but your local coffee shop stamping your card for your 10th coffee free different?
> how do they, for example, have a tool call that gives the customer a discount, without it getting used when it shouldn’t?
One possibility is that the company and the user both have AI agents, and that these are able to negotiate with each other. Then I talk to my local customized agent, which knows my preferences, and it explains the required context if the company’s agent does something that won’t make sense to me. But the company’s agent is still privy to the details required to offer discounts, say.
> provide the user/customer with an interface to an AI that can do things for the user
This is exactly what MCP is. But the reality is it will likely be about as popular as browser extensions and most normies will avoid.
Simplifying UX at the cost of personal control is inevitable because not everyone wants to think about tool selection and coordination. But maybe we can angle the future toward "tool bundles" that interoperate well or (if we are dreaming) mandate models remain accessible by any harness, which none of the players want but would be best for users and ecosystem development.
> where companies provide the user/customer with an interface to an AI that can do things for the user
I'm afraid it is the future. The companies will protect all documentation and IP by putting it behind AI agents, and will monitor (with another AIs) how is it being used.
It's gonna be a dark era for any knowledge in public domain.
At that point, companies could just provide an API and let users bring their own user agent, be it AI or traditional.
The problem is that history has shown that companies really do not want to do that. They will happily pay to maintain an inferior interface and go out of their way to disrupt third party clients.
If the interface is documented and can also be used without AI, and you can write your own software instead if wanted, then that will be more helpful, I think. (Someone who does want to use it with the AI can still do so, without forcing everyone else to also do.)
This is like asking LG to make TVs where you can bring your own video input. It just doesn't make economic sense when they can extract so much money by owning the glass.
100% agree, and I predict this is where the actual nearest big battle in our industry will occur.
AI subsumes products. Users want that. Vendors, do not. AI does not work well when shackled within confines of a product - it works better from outside, where it can treat slices of products as tools, and mash them together into ad-hoc solutions. Alas, products is how our industry makes money. Take arbitrary slice of problem space, slap a trademark on it, and shill to people (or VCs for funding). This disconnect makes AI an existential threat to a good chunk of software industry, and you can bet that companies (possibly including your own employers) won't go gently into the night.
MCPs are an aberration, the early stage of "AI adoption" where no one knew what they're doing but they knew they "have to do something with AI!". This age is now ending, and I expect the tensions will go high, as most vendors will get desperate to avoid their products getting obsoleted overnight.
Imagine if companies exposed access to their services via a discoverable uniform interface.
Each company would determine the range of actions a customer could perform in response to their request and current circumstances. A representation of state could be transferred between the company's server and whatever client the customer chooses to use. The customer would be free to interact with any of these services in a way that suited them via their chosen client.
Either that or we could kludge something together with MCP.
So I actually struggled with this recently (2 months ago). My "vision" was to have customers BYO agent and provide a secure MCP with an in-app (forced) approval gate for sensitive operations. But everytime I described to one of my clients, how they could bring their own ChatGPT, Claude, Gemini, w/e, I got blank stares.
So now I'm building (already have done so mostly) cheap ZDR models into the application. I'm trying to build them in a way that I would want to use them, so a less "I'll be your AI today!" help bubble, but who's kidding... At least they aren't just customer pacifiers that use the help system, but actually can act on behalf of the user.
After which I'll end up building the MCP, but for power users, hopefully leveraging much of the same work. It's worth noting that MCPs have way too much friction still. It's gotten a lot easier recently (very recently) to bring your own custom MCP with Claude, but ChatGPT is more work, and restrictive (specific account types or submitted MCP apps) for the end user and I don't even know what Google wants at this point... It's a bit messy. IMO they are all dropping the ball (except Claude, which has taken the best approach).
I think we might be in this interim state that requires us to build ALL of the UIs, to address ALL of the users (Human responsively and Clankers), and it's a bit painful. If I was "the user" myself, then I would just want a CLI w/ OAUTH with clear documentation, that an agent could use (how is this worse than MCP?). But I'm not my own customer. That being said... I probably WILL build this version to scratch my own itch and bet on the future.
This is an amazing article. The problems it describes are exactly what we found when building a production system that used LLMs to (most of the time) produce reliable results. Extensive tests are necessary, and stakeholders have no idea how their suggestions fail in production. They just see the handful of times they tried something and had it work, not the long tail of cursed results. (“How hard can it be? Why don’t you just…”)
We didn’t get to the point of self-built prompts, as the article suggests, but it’s an intriguing idea.
This guy is on the ball with the problem. Totally correct: Like my friend Coda says all the time: "the textual nature of prompts leads us to take the intentional stance towards systems which aren’t conscious, and thus miss the essential nature of their non-meaning."
I don't know if his solution (""We should all go insane building interlocking evaluation and optimization pipelines, instead.") would be the long-term solution. Instead perhaps something could be trained into the models, i.e., he is describing a process at inference time that could be done at training time. To make their weird errors less frequent / make them more human.
cf. https://arxiv.org/abs/2008.04071 "On Controllability of AI" However, as I said, you can't make it perfect but you can make it better. (You can't make humans fully aligned with human society's interest anyway, including the humans controlling the nukes.)
> I don't know if his solution (""We should all go insane building interlocking evaluation and optimization pipelines, instead.") would be the long-term solution.
TFA could explain this one part better I think. The whole process proposed is real with lots of stuff in the literature, but by definition NOT a long-term solution in the sense that this process actually has no end. None of the approaches can get you a static answer for a moving target/platform.
So the "interlocking pipelines" for eval/opt would not be some stepping stone you can throw away, and they aren't something you'd run periodically. They'd basically be always on forever and spending 10-100x on system complexity and on tokens. Unless of course you're ready to freeze everything else about the whole system forever (including the backend model, and the whole nature of the "average" context window, the plugins/other prompts in the mix, etc).
Are most people in position to freeze requirements/platform forever? Not really, because if they were they'd just build a fairly static system and probably have limited use for AI. Are most people in a position to just casually accept 100x complexity/cost? Not really, that's the "it's not yet webscale" kind of advice that sounds good but isn't necessarily reasonable for average use-case or average org. Since specializing your own locale for this is usually a mistake.. the likely future direction is eval/optimization as a service
One issue with this that we ran into is that it costs actual countable money to run the test suite, which is distinct from anything else I’m used to, so the notion that we’d do enough testing to generate a statistically significant gauge of performance - man, I know it’s correct, but I’m not sure my company will survive the process.
For these tests, why not tune the temperature and such to reduce the randomness and convert them to almost-always-succeeds vs almost-always-fails? Is it not the iteration count that drives up the cost?
Isn't the goal of the author to get reproducible behavior out of the agent though? I would thinking turning the temperature down would serve that production goal too.
1. If you turn the temperature down too far, the output is just bad and no amount of running prompts optimization will let you hill climb your way to good performance.
2. It’s not about determinism vs non-determinism. It’s about chaos. A perfectly deterministic model is still chaotic. Meaning that very small changes to the input result in very large changes to the output.
Turning temperature down doesn’t actually get you predictable or reproducible behavior across different inputs.
Thank you, I forgot about the chaotic aspect, too - that’s the other part that makes this brutal. The bot performs perfectly on your tests, but your customer abhors the Oxford comma, so you, your marketing team and your test suite can go to hell.
right, this is the whole problem - the stochastic behavior is both the goal and the problem. If you want your tests to match production, you need to get a reasonable sample size, which costs real money.
As awesome as this is, because as much as I want to create an agent for my customer that is predictable and "deterministic", I can't help but think of how wasteful, expensive and not-fun this is.
Good for the author that it's fun for them, but for me it seems like I am in that "monkey ladder banana" experimemt: doing something because others are and the customer is giving me bana... sorry, money for it, convinced it will help him (the money would 100% stop if I started looping prompt optimizations like this).
If I have so many tools and MCPs as I do currently, and with each the behavior regresses and changes wildly, it seems I should either merge tools and do more automations and come back to the prompt. (The alternative being training my own model?)
This was great! When you think about optimizing prompts with GEPA (or comparable methods and tools), do you consider each tool or skill separately? How do you think about the optimization of the system prompt for a large agentic system?
I imagine that you do a collection of passes to cover each overlapping set of what you want evaluated, but the system prompt makes all cases dependent on each other.
What I’ve done in the past is use the system prompt to extract subjective criteria for an LLM judge (like various system prompt statements that contribute to brand voice) and check individual traces with that for evaluation, but I’d like to move beyond including that in a prompt at all.
No idea how effective this is, but it sure looks a lot more like engineering than most of the 'prompt engineering' things I've seen in the past years. Kudos to the author for writing this concisely without aggrandizing his work.
> the textual nature of prompts leads us to take the intentional stance towards systems which aren’t conscious, and thus miss the essential nature of their non-meaning
I see LLMs as being capable of making useful distinctions and having a rich action space. They are widely used because their operation is useful, and that can only happen when semantics work well in practice. But useful things that pay for themselves don't need our "essential nature" blessing, they already have persistence by mutual entanglement with us.
Ohhh, i didn't realize there was more, from mobile it looks as if the article ends on the first image, and the images are like other articles (the first being a self link)
Am I the only one who waited to the end for, and was disappointed not to see, a peek into these optimized prompts? I so want to take a peek into that abyss, even if that risks the abyss looking back at me. I'm curious just how twisted they get relative to the original, in what alien ways.
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
>Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
>When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
>Don't be curmudgeonly. Thoughtful criticism is fine, but please don't be rigidly or generically negative.
Crafting a good prompt is what differentiates an expert's output from a beginner's. Of course you need good constraints too. But those good constraints are created precisely through good prompts.
I hope the future of AI isn't this sort, where companies provide the user/customer with an interface to an AI that can do things for the user... I would much prefer that companies instead provide an interface FOR an AI, and the user brings their own AI which connects to that interface.
In other words, provide my AI with tools, instead of providing me an AI that uses your tools.
That way, my AI can bring all the context it needs, and I can bring all of the settings and knowledge about what I want with me. I don't want a fractured world of tons of AIs i interact with where I have to explain all the fundamental information about what I want and how I work every time.
This also has the benefit of sidestepping the issue the essay is talking about. You provide a consistent tool, and the AI weirdness is not your issue anymore. You don't have to worry about solving for all the weird ways people prompt the AI, or the ways they break.
There's no interop between video conferences services, no BYO-video-conferencing app, why would this be any different? It took government regulation to allow telephones other than those provided by the phone company to be plugged into the phone network. Can't control you or usage if they just provide dumb pipes/interfaces that anyone's tool and interact with.
All of those browser and computer use agents (available from OpenAI, Anthropic, Meta, and X.ai already) represent an interoperability workaround already.
I think this is basically the motivation behind MCP servers, so you're in good company.
But I've spoken with many people at companies who've decided to add an in-UI agent to their apps, and I don't think this trend will persist. Absolutely AI/agents will increasingly become a major component of interfaces, but the current common incarnation of "Clippy for X" feels like a kludged bridge between an app that was not designed from first principles for agents (and often was poorly designed for humans) and the impulse to be "AI native".
From my experience, some of the most successful niches for this sort of UI so far have been in apps that were already well suited to it. I'm thinking specifically about analytics/dashboarding/"this is a portal for you to query things easily" software. They already started with a lot of the elements you want: visible provenance of the agents actions via the queries it writes, a malleable interface that you already expect to be customizable and ephemeral, and most importantly, "navigation" that is genuinely difficult for many users (in the sense that "navigating" can mean "querying specific data"). The agent provides a ton of value to users and its actions are intuitive and legible.
But the agents you see right now in a lot of apps that do things like navigate you to the right page by... sending you a link, which you could have clicked from the navbar? Or worse, which hijack your navigation and throw you on a page you're unfamiliar with, and where you have no sense of place or how to make your way to/from? I don't think they're particularly long for this world.
IMO it's a fundamental tension. AI does not fit well inside a product. It works best outside of it. But that means the product - not its functionality, but the business around it - becomes irrelevant. Which none of the product vendors want.
Also the motivation behind CLIs and SDKs, right? And the way agents write Python regularly, these days, I am hopeful what can be offloaded to deterministic systems will be.
A better internet is one in which LLMs enable everyone to ride their own custom Chromium (or whatever) flavor and the "frontend" of > 90% of sites - especially e-commerce - is generated specific to the user given their configuration of their personal LLM intermediary.
Companies will increasingly not ship frontends anymore, maybe just themed components that most users will ignore. "Pages" naturally become synthetic amalgamations of the various things you are interested in from various sources. Your preferred UI/UX comes for free courtesy of your LLM intermediary.
Everything is an API. Regulation will probably be needed to enforce this on the data access side given the incentives for companies to resist despite the benefit to the consumer. A relatively easy solution is to make it part of the requirements to take payments online, which is already highly regulated. The goal would be that essentially to operate anything approaching e-commerce online you must allow consumer AI intermediaries sufficient access that they could independently construct whatever your current frontend is from the publicly available API/MCP endpoints.
The tricky part of engineering around privacy/personas, account creation, etc. will be relatively easy following such state intervention. Take payments - virtual credit cards are already a useful way to protect data and mitigate risks when dealing with multiple vendors. This sort of API is a natural fit for a world in which companies are not allowed to box you into their horrible UIs, most of which are simply lazy attempts at copying the most profitable anti patterns of their competitors. Let your personal agent generate a unique card for each vendor behind the scenes. Why should you have to care?
How much better the internet would be if every UI/UX was in a meaningful sense, your own.
I feel like this is a bit too-optimistically describing an outcome which is unlikely, given how consumers have been slowly losing the same war for decades already.
The elephant in the room and the really "tricky part" involves powerful advertising and copyright-holding companies. They will (continue to) interpret that kind of user-agency as an existential threat. Their businesses revolve around deciding and knowing what you actually see and do with your computer, and their toolkit isn't just technical, but also economic and political.
That's why we already have problems with DRM, "felony contempt of business model" under the DMCA, browser fingerprinting, and everybody surrendering to arbitrary JS code controlling their computer. We capitulate not just to see an article or talk to a friend, but also to avoid our technical details getting into a secret un-appeal-able "suspicious" blacklist.
Not sure if companies would ever want this. They rely on their dark patterns and intentional manipulations to get customers to do what they want.
This is why I’ve thought that MCP and LLM access never made sense. You’re moving your users out of your brand and out of product.
Some companies are taking the opportunity to use the MCP server to send messages to the agent that the user won't necessarily see to protest. Agents are generally incredibly gullible and a huge opportunity for this type of thing.
That basically describes the Google shopping ribbon that appears with searches for products. When Google was good, you'd say a website doesn't exist if you can't find it in the 1st page of Google. The shopping ribbon provides offers from websites that aren't on the first page of my search result. Although I don't think you can pay without visiting the seller's website, this means that these are effectively stores that are more visible through the APIs they use to talk to Google's recommender engine. These suggestions are also no doubt personalized.
This also describes adsense, or Amazon or AliExpress, so it is clearly a successful model. If there is much left of other models. The only thing we have to hopeful about is that the interface is standardized so commodified SWE can be used to make your own Amazon or AliExpress.
I'm currently building something like this, basically converting every service to a standardized API that you can consume in a structured manner. Not focused on AI, but focused on normal users with special requirements on usability.
Basically the ad blocking approach has failed, and one needs to recognize that an allow-list based approach is needed when consuming internet-based offerings. Pick the cherries out of each website, leave the trash behind, and then build a custom per-website GUI around the things that you actually want to consume.
You shouldn't need LLM and AI to do such things. Even if such things might be useful to some people, I might want to use the API without LLM/AI and just to read the API documentation and then to make my own, and some other people might want to do similarly. In the specific case of e-commerce, I had been making my own specification of a file format for e-commerce (which is independent of the protocol), which is not intended to use with LLM/AI, but instead is intended to avoid many kind of dishonest business while also being flexible and that you could make and use your own software or some other implementaiton instead of being forced to use their UI/UX.
We could have a sort of “Representational State Transfer” using HTTP to represent the state of resources in both a human readable and machine readable format.
I don’t think that will happen. It sounds good, and I would like it if things operated that way - but from the company perspective how do they, for example, have a tool call that gives the customer a discount, without it getting used when it shouldn’t?
Companies want ai to replace human customer service decision making, which means it can’t just be an api that an external agent can interact with, because it needs private knowledge of company processes and access to capabilities that are abusable.
But we’re already at the point where if you manage to talk to a human, mostly you end up speaking to someone with no actual power to resolve your issue - so i think basically the future is just going to suck
> from the company perspective how do they, for example, have a tool call that gives the customer a discount, without it getting used when it shouldn’t?
I might be misunderstanding your point but I think you're in agreement with the gp, as in they are arguing for a locked down interface behind which the AI sits, narrowed to use within whatever particular operations the user can access, as opposed to a wide open AI interface with access to any operation (in principle).
So in your example there's no way to ask the AI for any old discount because the interface for doing so is locked down to just the discounts you can access, although the AI may be able to apply a discount you didn't ask for dynamically, if allowed, to give the customer a better experience.
Of course both versions can be implemented securely, it's just probably in general a smaller attack surface if you provide a tighter interface first.
> have a tool call that gives the customer a discount, without it getting used when it shouldn’t?
without comment on the rest of your post, a lot of discounts are just dark patterns, buy 2 for $10 or one for $6 is one of the oldest dark patterns around, and a lot of people fall for it, when they really didn't need that much whatever it was they were buying. In fact there are laws outlawing these practices coming into force for various types of products (alcohol, sugar) .
Other types of discounts like loyalty are more vague personally I think are a minor dark pattern. Negotiated discount on bulk purchases b2b are a different class, but it could be argued that you should only get the discount that the seller saves on logistics.
I'm just pointing this out because its very normalized, 'signup and get 3 months free', 'buy 2 get one free' etc, people dont bat an eyelid at, the same as advertising , but they are all just the earliest versions of dopamine hacking and dark patterns that consensus now is coming around to say probably isn't the best thing when scaled beyond a simple one to one interaction. Why are snapchat dopamine hacking with streaks but your local coffee shop stamping your card for your 10th coffee free different?
> how do they, for example, have a tool call that gives the customer a discount, without it getting used when it shouldn’t?
One possibility is that the company and the user both have AI agents, and that these are able to negotiate with each other. Then I talk to my local customized agent, which knows my preferences, and it explains the required context if the company’s agent does something that won’t make sense to me. But the company’s agent is still privy to the details required to offer discounts, say.
You prevent abuse of the API by models the same way you prevent abuse by humans: you have server side checks.
They can want anything they like, if customers want to use agents and they don't provide APIs they will lose out.
> provide the user/customer with an interface to an AI that can do things for the user
This is exactly what MCP is. But the reality is it will likely be about as popular as browser extensions and most normies will avoid.
Simplifying UX at the cost of personal control is inevitable because not everyone wants to think about tool selection and coordination. But maybe we can angle the future toward "tool bundles" that interoperate well or (if we are dreaming) mandate models remain accessible by any harness, which none of the players want but would be best for users and ecosystem development.
Companies will do whatever creates the most lock-in for the user and generates the most profit for themselves.
Ask yourself what's in their best interest as a business? That's probably what they'll do.
> where companies provide the user/customer with an interface to an AI that can do things for the user
I'm afraid it is the future. The companies will protect all documentation and IP by putting it behind AI agents, and will monitor (with another AIs) how is it being used.
It's gonna be a dark era for any knowledge in public domain.
At that point, companies could just provide an API and let users bring their own user agent, be it AI or traditional.
The problem is that history has shown that companies really do not want to do that. They will happily pay to maintain an inferior interface and go out of their way to disrupt third party clients.
If the interface is documented and can also be used without AI, and you can write your own software instead if wanted, then that will be more helpful, I think. (Someone who does want to use it with the AI can still do so, without forcing everyone else to also do.)
This is like asking LG to make TVs where you can bring your own video input. It just doesn't make economic sense when they can extract so much money by owning the glass.
100% agree, and I predict this is where the actual nearest big battle in our industry will occur.
AI subsumes products. Users want that. Vendors, do not. AI does not work well when shackled within confines of a product - it works better from outside, where it can treat slices of products as tools, and mash them together into ad-hoc solutions. Alas, products is how our industry makes money. Take arbitrary slice of problem space, slap a trademark on it, and shill to people (or VCs for funding). This disconnect makes AI an existential threat to a good chunk of software industry, and you can bet that companies (possibly including your own employers) won't go gently into the night.
MCPs are an aberration, the early stage of "AI adoption" where no one knew what they're doing but they knew they "have to do something with AI!". This age is now ending, and I expect the tensions will go high, as most vendors will get desperate to avoid their products getting obsoleted overnight.
Imagine if companies exposed access to their services via a discoverable uniform interface.
Each company would determine the range of actions a customer could perform in response to their request and current circumstances. A representation of state could be transferred between the company's server and whatever client the customer chooses to use. The customer would be free to interact with any of these services in a way that suited them via their chosen client.
Either that or we could kludge something together with MCP.
So I actually struggled with this recently (2 months ago). My "vision" was to have customers BYO agent and provide a secure MCP with an in-app (forced) approval gate for sensitive operations. But everytime I described to one of my clients, how they could bring their own ChatGPT, Claude, Gemini, w/e, I got blank stares.
So now I'm building (already have done so mostly) cheap ZDR models into the application. I'm trying to build them in a way that I would want to use them, so a less "I'll be your AI today!" help bubble, but who's kidding... At least they aren't just customer pacifiers that use the help system, but actually can act on behalf of the user.
After which I'll end up building the MCP, but for power users, hopefully leveraging much of the same work. It's worth noting that MCPs have way too much friction still. It's gotten a lot easier recently (very recently) to bring your own custom MCP with Claude, but ChatGPT is more work, and restrictive (specific account types or submitted MCP apps) for the end user and I don't even know what Google wants at this point... It's a bit messy. IMO they are all dropping the ball (except Claude, which has taken the best approach).
I think we might be in this interim state that requires us to build ALL of the UIs, to address ALL of the users (Human responsively and Clankers), and it's a bit painful. If I was "the user" myself, then I would just want a CLI w/ OAUTH with clear documentation, that an agent could use (how is this worse than MCP?). But I'm not my own customer. That being said... I probably WILL build this version to scratch my own itch and bet on the future.
This is an amazing article. The problems it describes are exactly what we found when building a production system that used LLMs to (most of the time) produce reliable results. Extensive tests are necessary, and stakeholders have no idea how their suggestions fail in production. They just see the handful of times they tried something and had it work, not the long tail of cursed results. (“How hard can it be? Why don’t you just…”)
We didn’t get to the point of self-built prompts, as the article suggests, but it’s an intriguing idea.
This guy is on the ball with the problem. Totally correct: Like my friend Coda says all the time: "the textual nature of prompts leads us to take the intentional stance towards systems which aren’t conscious, and thus miss the essential nature of their non-meaning."
I don't know if his solution (""We should all go insane building interlocking evaluation and optimization pipelines, instead.") would be the long-term solution. Instead perhaps something could be trained into the models, i.e., he is describing a process at inference time that could be done at training time. To make their weird errors less frequent / make them more human.
cf. https://arxiv.org/abs/2008.04071 "On Controllability of AI" However, as I said, you can't make it perfect but you can make it better. (You can't make humans fully aligned with human society's interest anyway, including the humans controlling the nukes.)
> I don't know if his solution (""We should all go insane building interlocking evaluation and optimization pipelines, instead.") would be the long-term solution.
TFA could explain this one part better I think. The whole process proposed is real with lots of stuff in the literature, but by definition NOT a long-term solution in the sense that this process actually has no end. None of the approaches can get you a static answer for a moving target/platform.
So the "interlocking pipelines" for eval/opt would not be some stepping stone you can throw away, and they aren't something you'd run periodically. They'd basically be always on forever and spending 10-100x on system complexity and on tokens. Unless of course you're ready to freeze everything else about the whole system forever (including the backend model, and the whole nature of the "average" context window, the plugins/other prompts in the mix, etc).
Are most people in position to freeze requirements/platform forever? Not really, because if they were they'd just build a fairly static system and probably have limited use for AI. Are most people in a position to just casually accept 100x complexity/cost? Not really, that's the "it's not yet webscale" kind of advice that sounds good but isn't necessarily reasonable for average use-case or average org. Since specializing your own locale for this is usually a mistake.. the likely future direction is eval/optimization as a service
One issue with this that we ran into is that it costs actual countable money to run the test suite, which is distinct from anything else I’m used to, so the notion that we’d do enough testing to generate a statistically significant gauge of performance - man, I know it’s correct, but I’m not sure my company will survive the process.
Maybe a better approach would be to stop trying to get a product out of a slot machine?
But I got real lucky once!
For these tests, why not tune the temperature and such to reduce the randomness and convert them to almost-always-succeeds vs almost-always-fails? Is it not the iteration count that drives up the cost?
If turn the temperature down for tests, they won’t match production behaviors. If you turn it down too far in production, the output will just be bad.
Isn't the goal of the author to get reproducible behavior out of the agent though? I would thinking turning the temperature down would serve that production goal too.
2 things.
1. If you turn the temperature down too far, the output is just bad and no amount of running prompts optimization will let you hill climb your way to good performance.
2. It’s not about determinism vs non-determinism. It’s about chaos. A perfectly deterministic model is still chaotic. Meaning that very small changes to the input result in very large changes to the output.
Turning temperature down doesn’t actually get you predictable or reproducible behavior across different inputs.
Thank you, I forgot about the chaotic aspect, too - that’s the other part that makes this brutal. The bot performs perfectly on your tests, but your customer abhors the Oxford comma, so you, your marketing team and your test suite can go to hell.
right, this is the whole problem - the stochastic behavior is both the goal and the problem. If you want your tests to match production, you need to get a reasonable sample size, which costs real money.
As awesome as this is, because as much as I want to create an agent for my customer that is predictable and "deterministic", I can't help but think of how wasteful, expensive and not-fun this is.
Good for the author that it's fun for them, but for me it seems like I am in that "monkey ladder banana" experimemt: doing something because others are and the customer is giving me bana... sorry, money for it, convinced it will help him (the money would 100% stop if I started looping prompt optimizations like this).
If I have so many tools and MCPs as I do currently, and with each the behavior regresses and changes wildly, it seems I should either merge tools and do more automations and come back to the prompt. (The alternative being training my own model?)
RLHF might help, you don't have to train a model from scratch.
This was great! When you think about optimizing prompts with GEPA (or comparable methods and tools), do you consider each tool or skill separately? How do you think about the optimization of the system prompt for a large agentic system?
I imagine that you do a collection of passes to cover each overlapping set of what you want evaluated, but the system prompt makes all cases dependent on each other.
What I’ve done in the past is use the system prompt to extract subjective criteria for an LLM judge (like various system prompt statements that contribute to brand voice) and check individual traces with that for evaluation, but I’d like to move beyond including that in a prompt at all.
[dead]
No idea how effective this is, but it sure looks a lot more like engineering than most of the 'prompt engineering' things I've seen in the past years. Kudos to the author for writing this concisely without aggrandizing his work.
> the textual nature of prompts leads us to take the intentional stance towards systems which aren’t conscious, and thus miss the essential nature of their non-meaning
I see LLMs as being capable of making useful distinctions and having a rich action space. They are widely used because their operation is useful, and that can only happen when semantics work well in practice. But useful things that pay for themselves don't need our "essential nature" blessing, they already have persistence by mutual entanglement with us.
Instead of anger, I think one of the few take aways from Peter Steinberg Mr Open Claw was his view on agentic empathy.
How would I try to compute things with the amount of context? But quickly it turns into a blame game, of at parts trying to give WAY too much context.
The format of this article makes it almost impossible to read.
I gave up after about 10 "pages".
It’s a talk, not an article.
Ohhh, i didn't realize there was more, from mobile it looks as if the article ends on the first image, and the images are like other articles (the first being a self link)
> That is currently working, but since the fix is fully deranged I expect it’ll be disturbed again at some point.
This one made me laugh out loud
Am I the only one who waited to the end for, and was disappointed not to see, a peek into these optimized prompts? I so want to take a peek into that abyss, even if that risks the abyss looking back at me. I'm curious just how twisted they get relative to the original, in what alien ways.
[flagged]
[flagged]
[flagged]
[dead]
TL;DR: you need to do... essentially supervised learning on your prompts? I mean, if I wanted to do ML, I'd already have been doing it ten years ago.
[dead]
[flagged]
[flagged]
From the HN guidelines:
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
>Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
>When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
>Don't be curmudgeonly. Thoughtful criticism is fine, but please don't be rigidly or generically negative.
[flagged]
your loss
Crafting a good prompt is what differentiates an expert's output from a beginner's. Of course you need good constraints too. But those good constraints are created precisely through good prompts.