>None of the three can easily be captured in code, but are trivial to capture as documentation.
Posts like this frustrate me. Not because of what they ask, but because of what they incorrectly assume. They assume that documentation can provide enough context, and that human knowledge is not needed.
Every bit of written documentation can and will be misinterpreted. And perfect clarity is impossible. A well-written ADR does not eliminate all ambiguity, because there is too much cultural context around the writing of the ADR that attempting to read it from some other cultural vantage point leads to bad assumptions. We can find this basic lesson from reading law (2nd, 14th amendments to the constitution), history (what did happen after Muhammad died?), philosophy (what in the world is Plato's cave talking about?), or theology (how should we translate Ephesians 5:22-33 and what does that mean) outside its original context with other people.
Just writing things down and thinking an AI is going to later perfectly understand what the intent of the author is... patently ridiculous. I do not intend to dismiss the idea that we should probably document more, but the idea that the AI can just take our documentation and competently understand all the decisions represented in them is ludicrous.
Having worked at enterprise with 100+ engineers across multiple teams with complex reporting lines, I definitely agree with this. There's a lot of nuance behind decisions that docs simply can't capture. I mean in theory you can probably write every considerations that led you to make a certain decision, e.g.:
- I'm writing this service even though team X has built the same thing, because my team lead doesn't trust team X since the last time we depended on their service 3 years ago, they had a major downtime that screwed us up big time
- This service is using AWS Lambda simply because I think it's cool, despite the fact that the company has a dedicated team running k8s stack with argocd, argo rollouts, KEDA, etc for the entire company
- Service Y is written in this particular way because it's a service that is shared with another team that came from a company that was acquired, and they wouldn't use it unless we write it this particular way, and making the top execs happy is more important than dealing with a small tech debt (this is probably true)
But no one is going to write these in their RFC. But Sarah knows.
> They assume that documentation can provide enough context, and that human knowledge is not needed.
It's funny actually, because I fully agree with your reasoning. The only part were we differ is whether that's assumed, or even implied.
No documentation means running fully on tribal knowledge, or institutional knowledge if you prefer. Even if you capture your intent, imperfect and incomplete, in as little as 2 paragraphs, you'll get durable recorded memory, and intent you'll be able to reference. It does not eliminate ambiguity, but it adds framing, direction, and friction.
The examples are great, and they serve really well to prove another point that I intentionally left out: writing is not a one-shot activity. Documentation is living and should be treated as such. Unless it receives proper care continuously, it will wither and die. That could very well be the topic of a future post!
Thank you for reading and for providing thoughtful feedback!
I can’t tell from the text if the first paragraph of your response is meant to be ironic or tongue-in-cheek, but it certainly works that way.
It feels hard to recognize that you agree with me from the following paragraph of your post
> This is similar to asking Sarah why the team went for A over B, but without the imperfections of human memory, and available to agents as well. When an agent reads it, the decision isn’t an arbitrary historical fact - it’s the conclusion of an argument the agent can now evaluate and extend.
I definitely agree that human knowledge is imperfect, and that documentation is needed. But I just cannot get around the fact that the following sentence, out of context, can refer to football, American football, and basketball : ‘I passed to the center, and we scored’ and without more context, each of those can have multiple meanings even in each of the sports I named.
const hello = ‘world’
Is valid in so many programming languages that need more context.
No, the agents don’t ask Sarah, but maybe they should. And maybe I’m on this soap box because the thing is that I write internal line of business software and the technical stuff is easy. But when Sarah is the inside sales manager or the industrial engineer a continent away who speaks a different language, the reality of adequately capturing context for decisions that others have made that our software has to implement, and in a context rich enough for documentation to be succinct, is enough to make me frustrated with articles like yours that hand-wave that into ‘trivial to capture in documentation.’
I do agree with you both, but we need a wider view of what documentation means. I'm spending time this days reading the OpenBSD code and while the latter is really well written, I spent even more time reading the mailing list archives, the revision notes, the BSD4.4 book, some OS textbooks (Three Easy Pieces is great BTW), and various datasheets (Intel, USB,...). I would have a much easier time if I could find a dev to be my mentor and walk me through it.
All of this to say, tribal knowledge is valuable and rarely well captured. Even experts organize conferences and lectures instead of just sending each other papers. Written words is better than nothing, but knowledge is better transferred from the person that has it. If you lose that person, the only way forward is to train another one to replace it (either intentionally or not).
Pulling histograms from my dictation software, it's apparent that I speak for sixty minutes at a time multiple times a day at 200-350 wpm aggregate just... trying to communicate all the nuance about a tiny, _tiny_ subject area, for an agent to implement or write in plans/docs... and then despite the best efforts of both me and LLMs, that can be and usually still _is_ misinterpreted.
Whether bulk or terse, highly precise with words or highly nuanced in how it's communicated... I don't think any of that gets you to a place where the documentation is a substitute for asking the person.
People are trying to do this with meetings as well and certainly they help but code that's written plus meetings plus an architect like person yammering on endlessly about the nuance that went into it still is often not enough to capture the detail in earnest – and especially not in a way that won't be misinterpreted.
Perhaps if AI becomes truly superhuman in all of the relevant areas to the point that it makes the decisions just as well as the person in the chair... _then_ we might solve this by having it instantly pattern recognize the solution and the why, but until and unless we reach that day, I think what you're saying is very true.
>Every bit of written documentation can and will be misinterpreted.
Yes, humans (and human languages) are flawed and lossy.
>A well-written ADR does not eliminate all ambiguity,
True: no docs can ever eliminate all ambiguity (on a decent sized project at least).
But this entire argument seems to be "letting perfect be the enemy of the good". Documentation doesn't have to be perfect or 100% unambiguous to be useful.
I agree. But that usefulness can be deceptive, especially when one thinks enough context has been captured to prevent misunderstandings. An LLM (or human, to be clear) can easily
go down the wrong path from their own assumptions, and now you have three problems.
My main complaint is that articles like this lead people to think they they can avoid those three problems, without understanding why there are three.
People would often ask me why I would just read the code for a project rather than ask whoever wrote it. At the end of 30 minutes talking to someone, I may not have learned anything. Or I may have learned the wrong thing because they wrote it 3 years ago and forgot about it.
At the end of 30 minutes I either understand how the project works at a high level or know enough to know I'm not going to be improving anything with this project, ever.
As the 'sarah' for an org I'm golden handcuffed to, avoid becoming the 'sarah' at all costs it will cost you so much in your career. I am a founding engineer self-taught from it/software security to full stack dev, and I pushed so we aimed to higher better engineers than myself. When they came in it was a full rewrite, a new abstraction and going down the same paths I learned the hard way not to go down. I was pushed out of enforcing hard-earned business logic, and we are still paying for it. Everywhere from not accepting the inherent complexity of the problem and over simplifying software to trying to get same features we had from 3 years ago. This then has made me the scapegoat for why the new engineers made xyz decision, and was the one in charge of fixing the shortcuts, bugs and workarounds. I have received promotions for this to be in charge of the veterans that didn't listen to me, whom still don't until it's a fire drill. Joke among colleges is I was the first 'agent' our company had, endless work, just enough authority to do current task, not enough respect/authority to solve the symptom. I understand this is also a failure of my office politics and am improving, however its hard to balance blunt productivity, slow careful positioning and letting people struggle with items I don't recommend until circle back.
This is such a toxic combination, becuse it requires significant people skills to get out of the "the kid who learned everything here and is grateful for it" and get proper respect as a professional. At some point the only option is changing jobs. I've seen companies matching your offer, finally realizing you actually have value in the market, but don't count on that, don't bluff.
I too ended up as the "go to guy", partly because I had a lot of enthusiamsm for my new job, and partly because the talent pool there wasn't very deep (or maybe they were smarter than me). It's fulfilling until it becomes unrewarding, I had to move on after almost 5 years. Still did consulting for them ocassionally for a couple more years.
That feels more familiar than I'd care to admit - definitely been there, done that. I think what most fail to realize is that it eventually turns into a ball and chain, decreasing your mobility within the organization. So the consequences of running an org on tribal knowledge and Sarahs is far worse, and direct, than that it won't work very well with agents.
It’s very easy to turn into the Sarah - or the Brent if you prefer the Phoenix Project analogy. As exciting as it might initially be to be the go-to person, it’s also, as you so elegantly put it, “endless work, just enough authority to do current task, not enough respect/authority to solve the symptom”.
Much appreciated! Thanks for the blog around ADR, I didn't know there was a full ecosystem for the approach and the AI tie in will help me sell the process
All organizations above a certain size have a Sarah. This I've learned first and second hand over decades (the second hand was a spouse whose job at one point was finding, interviewing, and collecting the knowledge of her's org's Sarahs).
Very, very few of these organizations have ever known, and fewer still have ever cared, about their Sarahs.
This isn't the end of Sarahs. Sarahs have never had their time or place beyond immediate teams, many of which have used Fight Club rules when it came to their Sarah: Never talk about Sarah, especially not to the boss. Other, non Fight Club rules: When Sarah is away, cover as best you can. Change jobs before Sarah retires. It is not the end, because the time of Sarahs never began.
So I agree with ";dr" comment, but it would apply had this been written by a human, by AI, by a super-intelligent shade of blue, or a small furry creature from Alpha Centauri.
the agent has the operator, and the operator has a sarah
i have spent a lot of time answering colleague's agent's questions as a sarah.
my thought has been to try to commit any of the answers to those questions into the relevant codebases so that they become findable.
same with anytime i have to give the agent extra context about some code. The end goal being that as much knowledge as possible is out of my head, and put nearby the code where its most useful
It requires surprisingly few lines of AGENTS.md to ensure before a line of code is patched, SPEC.md has the new target and a DEVLOG.md has a journal of why, then after the code tests OK, the commit carries not only the change but the motivation, and any DOCS.md is updated with new context.
It turns out wrapping the patch with these things makes the patch better. Plus, you have docs. All this goes in the PR.
Humans think docs cost/waste time. Agents not only don't care, but code better when doing SDLC (because once you get rid of the cult ceremonies, the core principles work). And so do humans.
Maybe Sarah can write it down, and maybe so can anyone that gets an answer from her, and maybe so can the LLM.
> We’ve all been in that 30-minute meeting that could just as well have been a two-paragraph ADR, if anyone would have bothered writing the decision down as it happened. Organizations learned to run on oral tradition because the alternative required discipline that was hard to reward, and not documenting it properly rarely turned into a real problem - at least not immediately. With agents, on the other hand, the full cost of a missing paper trail is paid every time the session terminates.
I just think this is entirely wrong. Oral tradition is valuable because it's flexible in a way that written tradition struggles to be. Just a handful of oral-tradition decisions I routinely see that could never be written up as persistent documentation:
* The CEO said X is our top priority, but we think Y is more important and we can do it without compromising too much on X, so we're going to do both.
* Team A has a track record of quality and success, so their decisions are subject to less review and receive more deference
* Team B is sloppy and makes a lot of bad calls, so we don't trust their judgments when doing so might lead to an outage for us.
All three of those should be written down. The first one, write down well justified but also open to learning context you don't have, and discuss that with the CEO.
The next two don't have to name names, but absolutely can name practices and privileges. Do these things, participate this way, get bonus. Do other things, no participation, and no bonus. Be written-clear about the expectations and consequences. And then hold teams accountable.
>None of the three can easily be captured in code, but are trivial to capture as documentation.
Posts like this frustrate me. Not because of what they ask, but because of what they incorrectly assume. They assume that documentation can provide enough context, and that human knowledge is not needed.
Every bit of written documentation can and will be misinterpreted. And perfect clarity is impossible. A well-written ADR does not eliminate all ambiguity, because there is too much cultural context around the writing of the ADR that attempting to read it from some other cultural vantage point leads to bad assumptions. We can find this basic lesson from reading law (2nd, 14th amendments to the constitution), history (what did happen after Muhammad died?), philosophy (what in the world is Plato's cave talking about?), or theology (how should we translate Ephesians 5:22-33 and what does that mean) outside its original context with other people.
Just writing things down and thinking an AI is going to later perfectly understand what the intent of the author is... patently ridiculous. I do not intend to dismiss the idea that we should probably document more, but the idea that the AI can just take our documentation and competently understand all the decisions represented in them is ludicrous.
Having worked at enterprise with 100+ engineers across multiple teams with complex reporting lines, I definitely agree with this. There's a lot of nuance behind decisions that docs simply can't capture. I mean in theory you can probably write every considerations that led you to make a certain decision, e.g.:
- I'm writing this service even though team X has built the same thing, because my team lead doesn't trust team X since the last time we depended on their service 3 years ago, they had a major downtime that screwed us up big time
- This service is using AWS Lambda simply because I think it's cool, despite the fact that the company has a dedicated team running k8s stack with argocd, argo rollouts, KEDA, etc for the entire company
- Service Y is written in this particular way because it's a service that is shared with another team that came from a company that was acquired, and they wouldn't use it unless we write it this particular way, and making the top execs happy is more important than dealing with a small tech debt (this is probably true)
But no one is going to write these in their RFC. But Sarah knows.
> They assume that documentation can provide enough context, and that human knowledge is not needed.
It's funny actually, because I fully agree with your reasoning. The only part were we differ is whether that's assumed, or even implied.
No documentation means running fully on tribal knowledge, or institutional knowledge if you prefer. Even if you capture your intent, imperfect and incomplete, in as little as 2 paragraphs, you'll get durable recorded memory, and intent you'll be able to reference. It does not eliminate ambiguity, but it adds framing, direction, and friction.
The examples are great, and they serve really well to prove another point that I intentionally left out: writing is not a one-shot activity. Documentation is living and should be treated as such. Unless it receives proper care continuously, it will wither and die. That could very well be the topic of a future post!
Thank you for reading and for providing thoughtful feedback!
I can’t tell from the text if the first paragraph of your response is meant to be ironic or tongue-in-cheek, but it certainly works that way.
It feels hard to recognize that you agree with me from the following paragraph of your post > This is similar to asking Sarah why the team went for A over B, but without the imperfections of human memory, and available to agents as well. When an agent reads it, the decision isn’t an arbitrary historical fact - it’s the conclusion of an argument the agent can now evaluate and extend.
I definitely agree that human knowledge is imperfect, and that documentation is needed. But I just cannot get around the fact that the following sentence, out of context, can refer to football, American football, and basketball : ‘I passed to the center, and we scored’ and without more context, each of those can have multiple meanings even in each of the sports I named.
const hello = ‘world’
Is valid in so many programming languages that need more context.
No, the agents don’t ask Sarah, but maybe they should. And maybe I’m on this soap box because the thing is that I write internal line of business software and the technical stuff is easy. But when Sarah is the inside sales manager or the industrial engineer a continent away who speaks a different language, the reality of adequately capturing context for decisions that others have made that our software has to implement, and in a context rich enough for documentation to be succinct, is enough to make me frustrated with articles like yours that hand-wave that into ‘trivial to capture in documentation.’
I do agree with you both, but we need a wider view of what documentation means. I'm spending time this days reading the OpenBSD code and while the latter is really well written, I spent even more time reading the mailing list archives, the revision notes, the BSD4.4 book, some OS textbooks (Three Easy Pieces is great BTW), and various datasheets (Intel, USB,...). I would have a much easier time if I could find a dev to be my mentor and walk me through it.
All of this to say, tribal knowledge is valuable and rarely well captured. Even experts organize conferences and lectures instead of just sending each other papers. Written words is better than nothing, but knowledge is better transferred from the person that has it. If you lose that person, the only way forward is to train another one to replace it (either intentionally or not).
Pulling histograms from my dictation software, it's apparent that I speak for sixty minutes at a time multiple times a day at 200-350 wpm aggregate just... trying to communicate all the nuance about a tiny, _tiny_ subject area, for an agent to implement or write in plans/docs... and then despite the best efforts of both me and LLMs, that can be and usually still _is_ misinterpreted.
Whether bulk or terse, highly precise with words or highly nuanced in how it's communicated... I don't think any of that gets you to a place where the documentation is a substitute for asking the person.
People are trying to do this with meetings as well and certainly they help but code that's written plus meetings plus an architect like person yammering on endlessly about the nuance that went into it still is often not enough to capture the detail in earnest – and especially not in a way that won't be misinterpreted.
Perhaps if AI becomes truly superhuman in all of the relevant areas to the point that it makes the decisions just as well as the person in the chair... _then_ we might solve this by having it instantly pattern recognize the solution and the why, but until and unless we reach that day, I think what you're saying is very true.
>Every bit of written documentation can and will be misinterpreted.
Yes, humans (and human languages) are flawed and lossy.
>A well-written ADR does not eliminate all ambiguity,
True: no docs can ever eliminate all ambiguity (on a decent sized project at least).
But this entire argument seems to be "letting perfect be the enemy of the good". Documentation doesn't have to be perfect or 100% unambiguous to be useful.
I agree. But that usefulness can be deceptive, especially when one thinks enough context has been captured to prevent misunderstandings. An LLM (or human, to be clear) can easily go down the wrong path from their own assumptions, and now you have three problems.
My main complaint is that articles like this lead people to think they they can avoid those three problems, without understanding why there are three.
People would often ask me why I would just read the code for a project rather than ask whoever wrote it. At the end of 30 minutes talking to someone, I may not have learned anything. Or I may have learned the wrong thing because they wrote it 3 years ago and forgot about it.
At the end of 30 minutes I either understand how the project works at a high level or know enough to know I'm not going to be improving anything with this project, ever.
As the 'sarah' for an org I'm golden handcuffed to, avoid becoming the 'sarah' at all costs it will cost you so much in your career. I am a founding engineer self-taught from it/software security to full stack dev, and I pushed so we aimed to higher better engineers than myself. When they came in it was a full rewrite, a new abstraction and going down the same paths I learned the hard way not to go down. I was pushed out of enforcing hard-earned business logic, and we are still paying for it. Everywhere from not accepting the inherent complexity of the problem and over simplifying software to trying to get same features we had from 3 years ago. This then has made me the scapegoat for why the new engineers made xyz decision, and was the one in charge of fixing the shortcuts, bugs and workarounds. I have received promotions for this to be in charge of the veterans that didn't listen to me, whom still don't until it's a fire drill. Joke among colleges is I was the first 'agent' our company had, endless work, just enough authority to do current task, not enough respect/authority to solve the symptom. I understand this is also a failure of my office politics and am improving, however its hard to balance blunt productivity, slow careful positioning and letting people struggle with items I don't recommend until circle back.
> founding engineer self-taught
This is such a toxic combination, becuse it requires significant people skills to get out of the "the kid who learned everything here and is grateful for it" and get proper respect as a professional. At some point the only option is changing jobs. I've seen companies matching your offer, finally realizing you actually have value in the market, but don't count on that, don't bluff.
I too ended up as the "go to guy", partly because I had a lot of enthusiamsm for my new job, and partly because the talent pool there wasn't very deep (or maybe they were smarter than me). It's fulfilling until it becomes unrewarding, I had to move on after almost 5 years. Still did consulting for them ocassionally for a couple more years.
That feels more familiar than I'd care to admit - definitely been there, done that. I think what most fail to realize is that it eventually turns into a ball and chain, decreasing your mobility within the organization. So the consequences of running an org on tribal knowledge and Sarahs is far worse, and direct, than that it won't work very well with agents.
I am Sarah, and it has been incredibly profitable for me: good ratings, promotions up to director-level IC, extra retention RSU grants.
If you have golden handcuffs and multiple promotions out of being Sarah, what is it costing you?
Author here.
That sounds exhausting to say the least.
It’s very easy to turn into the Sarah - or the Brent if you prefer the Phoenix Project analogy. As exciting as it might initially be to be the go-to person, it’s also, as you so elegantly put it, “endless work, just enough authority to do current task, not enough respect/authority to solve the symptom”.
Best wishes! I hope you manage to turn it around.
Much appreciated! Thanks for the blog around ADR, I didn't know there was a full ecosystem for the approach and the AI tie in will help me sell the process
All organizations above a certain size have a Sarah. This I've learned first and second hand over decades (the second hand was a spouse whose job at one point was finding, interviewing, and collecting the knowledge of her's org's Sarahs).
Very, very few of these organizations have ever known, and fewer still have ever cared, about their Sarahs.
This isn't the end of Sarahs. Sarahs have never had their time or place beyond immediate teams, many of which have used Fight Club rules when it came to their Sarah: Never talk about Sarah, especially not to the boss. Other, non Fight Club rules: When Sarah is away, cover as best you can. Change jobs before Sarah retires. It is not the end, because the time of Sarahs never began.
So I agree with ";dr" comment, but it would apply had this been written by a human, by AI, by a super-intelligent shade of blue, or a small furry creature from Alpha Centauri.
Simon should ask Sarah how to write an article without relying on an LLM
This article fails to mention Sarah burned out and is now a barista.
the agent still has a sarah.
the agent has the operator, and the operator has a sarah
i have spent a lot of time answering colleague's agent's questions as a sarah.
my thought has been to try to commit any of the answers to those questions into the relevant codebases so that they become findable.
same with anytime i have to give the agent extra context about some code. The end goal being that as much knowledge as possible is out of my head, and put nearby the code where its most useful
It's already quite feasible to record meetings and get AI summaries. As that works better, it will become more widespread.
It requires surprisingly few lines of AGENTS.md to ensure before a line of code is patched, SPEC.md has the new target and a DEVLOG.md has a journal of why, then after the code tests OK, the commit carries not only the change but the motivation, and any DOCS.md is updated with new context.
It turns out wrapping the patch with these things makes the patch better. Plus, you have docs. All this goes in the PR.
Humans think docs cost/waste time. Agents not only don't care, but code better when doing SDLC (because once you get rid of the cult ceremonies, the core principles work). And so do humans.
Maybe Sarah can write it down, and maybe so can anyone that gets an answer from her, and maybe so can the LLM.
This - oh, so much this!
Besides documentation, you can commit your chat sessions in git notes e.g. with https://usegitai.com/
> We’ve all been in that 30-minute meeting that could just as well have been a two-paragraph ADR, if anyone would have bothered writing the decision down as it happened. Organizations learned to run on oral tradition because the alternative required discipline that was hard to reward, and not documenting it properly rarely turned into a real problem - at least not immediately. With agents, on the other hand, the full cost of a missing paper trail is paid every time the session terminates.
I just think this is entirely wrong. Oral tradition is valuable because it's flexible in a way that written tradition struggles to be. Just a handful of oral-tradition decisions I routinely see that could never be written up as persistent documentation:
* The CEO said X is our top priority, but we think Y is more important and we can do it without compromising too much on X, so we're going to do both.
* Team A has a track record of quality and success, so their decisions are subject to less review and receive more deference
* Team B is sloppy and makes a lot of bad calls, so we don't trust their judgments when doing so might lead to an outage for us.
All three of those should be written down. The first one, write down well justified but also open to learning context you don't have, and discuss that with the CEO.
The next two don't have to name names, but absolutely can name practices and privileges. Do these things, participate this way, get bonus. Do other things, no participation, and no bonus. Be written-clear about the expectations and consequences. And then hold teams accountable.
[flagged]
ai;dr