This is ominous and very depressing given what we've recently learned / reconfirmed about LLMs sapping our ability to persist through difficult problems:
> There were 2 or 3 bugs that stumped me, and after 20 min or so of debugging I asked Claude for some advice. But most of the debugging was by hand!
Twenty whole minutes. Us old-timers (I am 39) are chortling.
I am not trying to knock the author specifically. But he was doing this for education, not for work. He should have spent more like 6 hours before desperately reaching for the LLM. I imagine after 1 hour he would have figured it out on his own.
You say this as if the LLM isn’t committing things it doesn’t even recognize as bugs if you don’t babysit it. I’d rather have a codebase with a few very well marked evil zones, rather than a codebase no one has read. All code contains demons and it’s good to have an understanding of their locations and relative power
Often when LLMs give me some command option or advice I haven’t seen before I try to independently verify it. And I’ve often been frustrated just how hard it is to find this info from the source documents.
Though a lot of the time this is more an inefficiency of the documentation and Google rather than something only LLMs could do.
YES. I don't know how many multi WEEK sessions of debugging I've been through in my career. Frustrating, but so many valuable lessons learned in the process. LLMs are absolutely causing us to lose something very important.
Having a tool that instantly searches through the first 50 pages of google and comes up with a reasonable solution is just speeding up what I would have done manually anyways.
Would I have learned more about (and around) the system I‘m building? Absolutely. I just prefer making my system work over anything else, so I don’t mind losing that.
The multi week debugging sessions weren't fun, but that doesn't mean they weren't valuable and important and a growth and learning opportunity that we now will no longer experience.
This does depend on who you are; If you're a senior with 10+ years of experience, it's a failure of your abilities to cut your losses or know when to seek help if you take far too long debugging something.
But for juniors, it's invaluable experience. And as a field we're already seeing problems resulting from the new generations of juniors being taught with modern web development, whose complexity is very obstructing of debugging.
I wish more was being invested in AI autocomplete workflows. That was a nice middle-ground.
But yeah my hunch is "the old way" - although not sure we can even call it that - is likely still on par with an "agentic" workflow if you view it through a wider lens. You retain much better knowledge of the codebase. You improve your understanding over coding concepts (active recall is far stronger than passive recognition).
So we’ve already grown nostalgic for the old days… skimming through an alien looking codebase, scratching your head trying to figure what crazy abstraction the last person who touched this code had in mind. Oh shit it was me? That made so much more sense back then… but it’s been 6 hours and I can’t figure out why this does not work anymore. So you read some docs but they are poorly written. So you find something on Google and try to hack that into your solution. But nope, now more stuff broke. There goes your day.
It's easy to take for granted lots of experience programming before the advent of LLMs. This seems like a good strategy to develop understanding of software engineering.
I remember writing BASIC on the Apple II back when it wasn't retro to do so!
Here’s how i do it: I create a lot of stuff using AI to the max, but I also spend the necessary of time on reviewing that the AI is producing code that passes my cognitive load standards. this involves some tokens spent on grooming code and documenting well. Most of this is effortless thanks to an AGENTS.md based on this: https://github.com/zakirullin/cognitive-load/blob/main/READM... but i have a good sense of catching when things are getting weird and i steer back.
Then, when credits run out. It’s show time! The code is neatly organized, abstractions make sense, comments are helpful so I have a solid ground to do some good old organic human coding. I make sure that when i’m approaching limits I’m asking the AI to set the stage.
I used to get frustrated when credits ran out because the AI was making something I would need to study to comprehend. Now I’m eager to the next “brain time hand-out”
It sounds weird but it’s a form of teamwork. I have the means to pay for a larger plan but i’d rather keep my brain active.
I did things the old way for 25 years and my carpal tunnels are wearing out. LLMs let me produce the same quality I always have with a lot less typing so not mad at that at all. I review and own every line I commit, and feel no desire to go back to the old way.
What scares the shit out of me are all these new CS grads that admit they have never coded anything more complex than basic class assignments by hand, and just let LLMs push straight to main for everything and they get hired as senior engineers.
It is like hiring an army of accountants that have never done math on paper and exclusively let turbotax do all the work.
If you have never written and maintained a complex project by hand, you should not be allowed to be involved in the development of production bound code.
But also, I feel this way about the industry long before LLMs. If you are not confident enough to run Linux on the computer in front of you, no senior sysadmin will hire you to go near their production systems.
Job one of everyone I mentor is to build Linux from scratch, and if you want an LLM build all the tools to run one locally for yourself. You will be way more capable and employable if you do not skip straight to using magic you do not understand.
> It is like hiring an army of accountants that have never done math on paper and exclusively let turbotax do all the work.
It's not though. It's fundamentally different because TurboTax will still work with clear deterministic algorithms. We need to see that the jump to AI is not a jump from hand written math to calculators. It's a jump from understanding how the math works to another world of depending on magic machines that spit out numbers that sort of work 90% of the time.
From what I remember, typical new C++ debugged code speed is about 20-25K lines per year, lines that are non-blank, non-comment and not completely verifiable by compiler. E.g., standalone bracket or comma or semicolon are not lines of code, function header is too not a line of code, but computation, conditions and loops are. This is from old IBM statistics, I learned about it circa 2007.
If we assume that there are 50 weeks per year, this gives us about 400-500 lines of code per week. Even at long average 65 chars per line, it goes not higher than 33K bytes per week. Your comment is about 1250 bytes long, if you write four such comments per day whole week, you would exceed that 33K bytes limit.
> If you have never written and maintained a complex project by hand, you should not be allowed to be involved in the development of production bound code.
So only the old hands allowed from now on, or how are we going to provide these learning opportunities at scale for new developers?
Junior developers have always been a lot less effective than senior developers. We will need new senior developers so we will need to train junior developers. Maybe we train them by forcing them to do things the hard way. The slow way. By hand. Because if we let them do things the fast way they are going to cause some serious damage.
The same way I learned 25 years ago still works today. Volunteer on open source projects.
Always happy to mentor people at stagex and hashbang (orgs I founded).
Also being a maintainer of an influential open source project goes on a resume, and helps you get seen in a crowded market while boosting your skills and making the world better. Win/win all around.
Even by pessimistic progress projections AI will be better than most at coding before this is a long term issue. And the output multiplier I'm seeing I suspect the number of SWEs needed to achieve the same task is going to start shrinking fast.
I don't think SWE is a promising career to get started in today.
But pro-AI posts never seem to pin themselves down on whether code checked in will be read and understood by a human. Perhaps a lot of engineers work in “vibe-codeable” domains, but a huge amount of domains deal with money, health, financial reporting, etc. Then there are domains those domains use as infrastructure (OS, cloud, databases, networking, etc.)
Even where it is non-critical, such as a social media site, whether that site runs and serves ads (and bills for them correctly) is critical for that company.
But you have to be good at SWE to be good at security engineering and sysadmin, and the demand there is skyrocketing.
We have a completely broken internet with almost nothing using memory encryption, deterministic builds, full source bootstrapping, secure enclaves, end to end encryption, remote attestation, hardware security auth, or proper code review.
Decades of human cognitive work to be done here even with LLM help because the LLMs were trained to keep doing things the old way unless we direct them to do otherwise from our own base of experience on cutting edge security research no models are trained on sufficiently.
> It is like hiring an army of accountants that have never done math on paper and exclusively let turbotax do all the work.
That is exactly been the situation for years. Once graduated accountants are not doing maths. They are using software (Exel, Xero etc.). They do need to know some basic formulas eg. NPV.
What they need to know is the law, current business practices etc.
I mean, that's the only way I code. I don't use llm's to do my work for me. I'm perfectly capable of solving any sort of problem on my own, and then I'll understand it well enough to explain it to someone later.
You should do what you want, and as a break it’s fine. But IMO right now the most leverage for most people is learning how to effectively manage agents. It’s really hard. Not many are truly good with it. It will be relevant for a long time.
Why would you think that? The landscape is fast-moving. Prompting tricks and "AI skills" of yesterday are already dated and sometimes actively counterproductive. The explicit goal of the companies working on the tech is to lower the barriers to entry and make it easier to use, building harnesses and doing refinement that align LLMs to an intuitive mode of interaction.
Do you think they'll fail? Do you think we've plateaued in terms of what using a computer looks like and your learnings for wrangling the agents of this year will be relevant for whatever the new hotness is next year? It's a strong claim that demands similarly strong argument to support.
I believe the point (which you seem to tacitly agree with) is that a young dev's time is much better spent reading and writing code "the old-fashioned way" vs chasing the new SOTA in AI-assisted development. A competent dev can basically master agentic development in a few months. But it takes years to become competent.
Not sure what you are using, but that’s easier said than done. I just set up an agent to ensure that my other agent would follow my coding guidelines by using hooks. The coding agent responded by switching to editing with `sed`, etc. to circumvent the hooks.
Claude Opus is going to give zero fucks about your attempts to manage it.
this is exactly right, I don't manage agents anymore (and have spent countless hours before learning how to do so, now this is a skill like my microsoft access skills (which were amazing back in the day...)
It is hard indeed. I find it really quite exhausting.
Personally, I feel like I have always been a very competent programmer. I'm embracing the new way of working, but it seems like quite a different skillset. I somewhat believe that it will be relevant for a long time, because there is an incredibly large gap in outcomes between members of my team using AI. I've had good results so far, but I'm keen to improve.
Yeah, it's really difficult to remember to tell it "make no mistakes". Typing a prompt is also really hard, especially when you have to remember the cli command to open the agent. Sometimes I even forget if I need to use "medium", "high", or "xhigh" for a task.
I see you got downvoted by I agree. I went through a massive valley of despair and turned back to hand crafting only to realize that for me coding was always a means to an end and I really didn't care at all about how I got there. Now I'm having a lot of fun building out all kinds of wonky projects.
This is ominous and very depressing given what we've recently learned / reconfirmed about LLMs sapping our ability to persist through difficult problems:
> There were 2 or 3 bugs that stumped me, and after 20 min or so of debugging I asked Claude for some advice. But most of the debugging was by hand!
Twenty whole minutes. Us old-timers (I am 39) are chortling.
I am not trying to knock the author specifically. But he was doing this for education, not for work. He should have spent more like 6 hours before desperately reaching for the LLM. I imagine after 1 hour he would have figured it out on his own.
Yep and after 6 hours don't reach for LLM, instead:
* Ask someone to come over and look
* Come back the next day, work on something else
* Add comment # KNOWN-ISSUE: ...., and move on and forget about it.
But year spent days on a bug at work before ha ha!
> Come back the next day, work on something else
This is a tried and true way of working on puzzles and other hard problems.
I generally have 2-4 important things in flight, so I find myself doing this a lot when I get stuck.
You say this as if the LLM isn’t committing things it doesn’t even recognize as bugs if you don’t babysit it. I’d rather have a codebase with a few very well marked evil zones, rather than a codebase no one has read. All code contains demons and it’s good to have an understanding of their locations and relative power
Now imagine someone else reading this and genuinely considering 20 minutes a long time to wait :-)
Often when LLMs give me some command option or advice I haven’t seen before I try to independently verify it. And I’ve often been frustrated just how hard it is to find this info from the source documents.
Though a lot of the time this is more an inefficiency of the documentation and Google rather than something only LLMs could do.
YES. I don't know how many multi WEEK sessions of debugging I've been through in my career. Frustrating, but so many valuable lessons learned in the process. LLMs are absolutely causing us to lose something very important.
I don’t miss multi week debugging sessions.
Having a tool that instantly searches through the first 50 pages of google and comes up with a reasonable solution is just speeding up what I would have done manually anyways.
Would I have learned more about (and around) the system I‘m building? Absolutely. I just prefer making my system work over anything else, so I don’t mind losing that.
The multi week debugging sessions weren't fun, but that doesn't mean they weren't valuable and important and a growth and learning opportunity that we now will no longer experience.
If I told someone I spent a week debugging a problem these days I think I would get laughed out of the call. Even a day might hit somw chuckles.
If you cant fix the bug just slop some code over it so its more hidden.
This is all gonna be fascinating in 5-10 years.
This does depend on who you are; If you're a senior with 10+ years of experience, it's a failure of your abilities to cut your losses or know when to seek help if you take far too long debugging something.
But for juniors, it's invaluable experience. And as a field we're already seeing problems resulting from the new generations of juniors being taught with modern web development, whose complexity is very obstructing of debugging.
This really does feel like a mass hysteria event. Bizarre to have to live through it.
I wish more was being invested in AI autocomplete workflows. That was a nice middle-ground.
But yeah my hunch is "the old way" - although not sure we can even call it that - is likely still on par with an "agentic" workflow if you view it through a wider lens. You retain much better knowledge of the codebase. You improve your understanding over coding concepts (active recall is far stronger than passive recognition).
Remember the old days of our youth, i.e. last week Monday, when we still wrote code by hand?
Well when the fashions change, the old ways are “old fashioned” but only literally
I can't tell if OP is satire... I've just seen so many unhiged takes that this article reads completely in line with the discourse...
So we’ve already grown nostalgic for the old days… skimming through an alien looking codebase, scratching your head trying to figure what crazy abstraction the last person who touched this code had in mind. Oh shit it was me? That made so much more sense back then… but it’s been 6 hours and I can’t figure out why this does not work anymore. So you read some docs but they are poorly written. So you find something on Google and try to hack that into your solution. But nope, now more stuff broke. There goes your day.
It's easy to take for granted lots of experience programming before the advent of LLMs. This seems like a good strategy to develop understanding of software engineering.
I remember writing BASIC on the Apple II back when it wasn't retro to do so!
This is awesome! I myself did a 12 weeks batch at RC (W1'24) and had an absolute blast. Happy coding! Stay curious.
fellow RC'er here - hi! I was Fall 2 '23.
I love being able to put my brain cells at lean, coq, haskell. All the fun stuff. And have my money job taken care of mostly with agents.
Not the point of the article but
> 15 years of Clojure experience
My God I’m old.
Here’s how i do it: I create a lot of stuff using AI to the max, but I also spend the necessary of time on reviewing that the AI is producing code that passes my cognitive load standards. this involves some tokens spent on grooming code and documenting well. Most of this is effortless thanks to an AGENTS.md based on this: https://github.com/zakirullin/cognitive-load/blob/main/READM... but i have a good sense of catching when things are getting weird and i steer back.
Then, when credits run out. It’s show time! The code is neatly organized, abstractions make sense, comments are helpful so I have a solid ground to do some good old organic human coding. I make sure that when i’m approaching limits I’m asking the AI to set the stage.
I used to get frustrated when credits ran out because the AI was making something I would need to study to comprehend. Now I’m eager to the next “brain time hand-out”
It sounds weird but it’s a form of teamwork. I have the means to pay for a larger plan but i’d rather keep my brain active.
I did things the old way for 25 years and my carpal tunnels are wearing out. LLMs let me produce the same quality I always have with a lot less typing so not mad at that at all. I review and own every line I commit, and feel no desire to go back to the old way.
What scares the shit out of me are all these new CS grads that admit they have never coded anything more complex than basic class assignments by hand, and just let LLMs push straight to main for everything and they get hired as senior engineers.
It is like hiring an army of accountants that have never done math on paper and exclusively let turbotax do all the work.
If you have never written and maintained a complex project by hand, you should not be allowed to be involved in the development of production bound code.
But also, I feel this way about the industry long before LLMs. If you are not confident enough to run Linux on the computer in front of you, no senior sysadmin will hire you to go near their production systems.
Job one of everyone I mentor is to build Linux from scratch, and if you want an LLM build all the tools to run one locally for yourself. You will be way more capable and employable if you do not skip straight to using magic you do not understand.
> It is like hiring an army of accountants that have never done math on paper and exclusively let turbotax do all the work.
It's not though. It's fundamentally different because TurboTax will still work with clear deterministic algorithms. We need to see that the jump to AI is not a jump from hand written math to calculators. It's a jump from understanding how the math works to another world of depending on magic machines that spit out numbers that sort of work 90% of the time.
From what I remember, typical new C++ debugged code speed is about 20-25K lines per year, lines that are non-blank, non-comment and not completely verifiable by compiler. E.g., standalone bracket or comma or semicolon are not lines of code, function header is too not a line of code, but computation, conditions and loops are. This is from old IBM statistics, I learned about it circa 2007.
If we assume that there are 50 weeks per year, this gives us about 400-500 lines of code per week. Even at long average 65 chars per line, it goes not higher than 33K bytes per week. Your comment is about 1250 bytes long, if you write four such comments per day whole week, you would exceed that 33K bytes limit.
I find this amusing.
> If you have never written and maintained a complex project by hand, you should not be allowed to be involved in the development of production bound code.
So only the old hands allowed from now on, or how are we going to provide these learning opportunities at scale for new developers?
Serious question.
Junior developers have always been a lot less effective than senior developers. We will need new senior developers so we will need to train junior developers. Maybe we train them by forcing them to do things the hard way. The slow way. By hand. Because if we let them do things the fast way they are going to cause some serious damage.
The same way I learned 25 years ago still works today. Volunteer on open source projects.
Always happy to mentor people at stagex and hashbang (orgs I founded).
Also being a maintainer of an influential open source project goes on a resume, and helps you get seen in a crowded market while boosting your skills and making the world better. Win/win all around.
Can't disagree, that's how I did it too :-)
Even by pessimistic progress projections AI will be better than most at coding before this is a long term issue. And the output multiplier I'm seeing I suspect the number of SWEs needed to achieve the same task is going to start shrinking fast.
I don't think SWE is a promising career to get started in today.
There’s certainly a lot of uncertainty.
But pro-AI posts never seem to pin themselves down on whether code checked in will be read and understood by a human. Perhaps a lot of engineers work in “vibe-codeable” domains, but a huge amount of domains deal with money, health, financial reporting, etc. Then there are domains those domains use as infrastructure (OS, cloud, databases, networking, etc.)
Even where it is non-critical, such as a social media site, whether that site runs and serves ads (and bills for them correctly) is critical for that company.
But you have to be good at SWE to be good at security engineering and sysadmin, and the demand there is skyrocketing.
We have a completely broken internet with almost nothing using memory encryption, deterministic builds, full source bootstrapping, secure enclaves, end to end encryption, remote attestation, hardware security auth, or proper code review.
Decades of human cognitive work to be done here even with LLM help because the LLMs were trained to keep doing things the old way unless we direct them to do otherwise from our own base of experience on cutting edge security research no models are trained on sufficiently.
> It is like hiring an army of accountants that have never done math on paper and exclusively let turbotax do all the work.
That is exactly been the situation for years. Once graduated accountants are not doing maths. They are using software (Exel, Xero etc.). They do need to know some basic formulas eg. NPV.
What they need to know is the law, current business practices etc.
I mean, that's the only way I code. I don't use llm's to do my work for me. I'm perfectly capable of solving any sort of problem on my own, and then I'll understand it well enough to explain it to someone later.
You should do what you want, and as a break it’s fine. But IMO right now the most leverage for most people is learning how to effectively manage agents. It’s really hard. Not many are truly good with it. It will be relevant for a long time.
> It will be relevant for a long time.
Why would you think that? The landscape is fast-moving. Prompting tricks and "AI skills" of yesterday are already dated and sometimes actively counterproductive. The explicit goal of the companies working on the tech is to lower the barriers to entry and make it easier to use, building harnesses and doing refinement that align LLMs to an intuitive mode of interaction.
Do you think they'll fail? Do you think we've plateaued in terms of what using a computer looks like and your learnings for wrangling the agents of this year will be relevant for whatever the new hotness is next year? It's a strong claim that demands similarly strong argument to support.
> It’s really hard
How? I just open multiple terminal panes, use git tree, and then basically it’s good old software dev practices. What am I missing?
You're probably significantly underselling the value of your own "good old software dev practices."
I believe the point (which you seem to tacitly agree with) is that a young dev's time is much better spent reading and writing code "the old-fashioned way" vs chasing the new SOTA in AI-assisted development. A competent dev can basically master agentic development in a few months. But it takes years to become competent.
The agents are already learning to manage agents, if it’s relevancy you’re looking for you might want to take up plumbing instead.
Not sure what you are using, but that’s easier said than done. I just set up an agent to ensure that my other agent would follow my coding guidelines by using hooks. The coding agent responded by switching to editing with `sed`, etc. to circumvent the hooks.
Claude Opus is going to give zero fucks about your attempts to manage it.
this is exactly right, I don't manage agents anymore (and have spent countless hours before learning how to do so, now this is a skill like my microsoft access skills (which were amazing back in the day...)
What has been most valuable for you?
It is hard indeed. I find it really quite exhausting.
Personally, I feel like I have always been a very competent programmer. I'm embracing the new way of working, but it seems like quite a different skillset. I somewhat believe that it will be relevant for a long time, because there is an incredibly large gap in outcomes between members of my team using AI. I've had good results so far, but I'm keen to improve.
For the average and mundane stuff, sure do whatever everyone is doing.
For the good stuff, there’s no alternative but to know and to have taste. Llms change nothing.
If they're so great, then we will end up somewhere where it's easy to pick up.
You will be relevant for 6 months until they manage themselves.
Yeah, it's really difficult to remember to tell it "make no mistakes". Typing a prompt is also really hard, especially when you have to remember the cli command to open the agent. Sometimes I even forget if I need to use "medium", "high", or "xhigh" for a task.
> It will be relevant for a long time.
Citation needed.
I see you got downvoted by I agree. I went through a massive valley of despair and turned back to hand crafting only to realize that for me coding was always a means to an end and I really didn't care at all about how I got there. Now I'm having a lot of fun building out all kinds of wonky projects.