I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change.
In fact, if it wasn't for the fact that it made making the actual changes I identified much easier (move these joins into a CTE etc) it would have been a detriment. Not only did I get sidetracked by a bunch of useless suggestions but I also had to put up with others dumping their raw AI output at me as if it was somehow a meaningful contribution.
The thing that makes it work really well is to make sure it has all the tooling to verify its hypotheses. If you allow it to run the full lifecycle in loops you will be surprised how well it works.
Built exactly this for work a few months ago. Claude has a read-only account to the dev DB and a bunch of python postgres tools, I point it at queries/views and ask it optimize, and it has to:
1) verify identical query results
2) run repeatedly to get average, worst, best, etc duration of runs
Sped up so many legacy things that none of us were ever going to bother with.
It comes from C. Northcote Parkinson's work, where he calls it "The Law of Triviality". The idea is that people clog up decision-making by introducing a swarm of issues that have little relative impact or importance for the project at hand, leading to a massive misallocation of attention. His example is a $10 million construction project for a Nuclear Reactor where as much time is spent debating the materials to be used for a $3000 bike shed as on the actual reactor buildout. The law of triviality says "the time spent on any item of the agenda will be in inverse proportion to the sum [of money] involved."
Source: C. Northcote Parkinson, "Parkinson's Law and Other Studies in Administration" (1957)
Thanks. I know what bikeshedding is. I’m asking what cyandees specifically means. The normal definition of bikeshedding doesn’t seem to make any sense in his sentence.
I’m assuming he’s using it to mean “cruft” or “low value features” but maybe there’s some meaning I’m missing.
Just like humans, AI needs a "reality" to check things against. If we don't provide that in an automated way, somehow, it will always make mistakes and we'd have to point it out. Since, at that point, we're just mediating with what reality gives us and what an AI should do with it.
AI doesn't have enough senses yet. It's trapped in a box.
Very few problems really have "all the tooling to verify its hypotheses" though. even if you want to construct such an harness.
Also let me ask you why we need better and better and models if what we have already can produce good output with 'all the tooling to verify its hypotheses'
We don't. You could freeze model development today and we would be able to use our current models to effectively optimize existing code for years to come.
Yes, sort of. Not that there is anything wrong with that. Fuzzing yields tons of trash results, the occasional fleck of gold and the even more rare nugget. Then you analyze how it found the flakes and nuggets update your fuzzing harness. Every so often you get to write a paper or give a talk about a breakthrough to share meaningful improvements.
Current AI isn't super effective at making the breakthroughs, but it sure is effective at democratizing the ones you can point it at.
The same tooling that you'd use. AI is not magic, it cannot know exactly how a program will run under your exact circumstances and data. Therefore, if it can use a profiler, test suite or application, or be able to read logs, just like you would, it then can make changes, evaluate them, and iterate. Personally, I've created test MCP servers that will perform actions like submit orders, have the AI capture and analyze PerfViews and Concurrency Visualizer, write out tracing data to logs... all the same stuff I was doing by hand for years.
Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD!
I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.
It bothers me that you have to explicitly state this to the agent. Makes me think what else is missing from that file which also needs to be explicitly stated, but I don't know what don't know. "Do a good job"?
This is where the harness shines. It's why products like Cursor have been growing like crazy. If you drop an extremely intelligent person into a problem they've never seen before, the initial context and knowledge-sharing just to get them up to speed is immense. They'll make a lot of mistakes until they're ready to fix your problem. The harness can leapfrog most of that. There is a booming industry in harnesses with no saturation in sight. We're not even close to maximising the utility of current models, let alone the amazing improvements which are happening week to week.
Will, it depends on the AI. Anthropic used to have a lecture-length system prompt for their models to explain this stuff—part of the secret sauce for Claude Code—and famously found that the 5 series models no longer need it.
As usual, if you use anything but the best model available I’m going to state that the better ones do better. If you do use the best model available, then I’ll just mention that Fable still has limits and still needs some guidance.
One thing it does not do is deliberately build tests which test nothing at all, or which restate the code under test. I mention this because certain other models absolutely would.
> Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD!
Tests onli validate the presence of bugs, not their abscence (Djikstra).
I'll also add that tests look at outputs and don't care how those outputs are derived. E.g. code filtering the entire db in memory will be fine in tests.
Haven't you ever written tests that assert that certain things happen or do not happen along the way? We use mocks all the time to, say, make sure that the code doesn't load the entire db in memory. Then it's there as a regression if somebody down the line wants to do that exact thing for some reason.
"Beware of bugs in the above code; I have only proved it correct, not tried it." - Donald Knuth
Tests prove the things you thought of worked, and it often isn't hard to find the likely edge cases such that you have reasonable confidence everything works. You will be wrong from time to time, but not that often. You can prove code correct, but if the proof is wrong (common when a human is doing it), or the spec is wrong (most people have no clue how to write a comprehensive spec) it can still be wrong.
If the agent can write the tests its supposed to pass and we are worried that agent produces quite a lot of slop (which is why we are doing the tests), then what is the defense against test slop? Test the tests?
Doing a pass where you just ask the AI to sanity-check the existing tests (against rules like “test against the spec, not the implementation” can also help.
In this case, you need benchmarks, in addition to regular tests. You also need concise instructions (or better yet, following your language / ecosystem norms if applicable) to how to run this stuff
I'm convinced this is why there's a bifurcation in old-hand/new-hand results: the old hands include the steps they'd do and the tools they'd use, and the new-hands never learned those tools.
Which model are you using? There is monumental difference between models. Even between "frontier models". When people tell these stories, it would be great to also add which model you were using.
As an example Opus 5.0 is in completely different class compared to Cursor Grok 4.5 even if the benchmarks don't show such massive difference. Not even talking about regular stuff like Sonnet or Composer or stuff like that.
Yeah. If you need something to dig deep, you need to try Fable (optionally in /goal mode).
For performance testing, I wrote isolated testbeds that try to impair the system in realistic ways (latency/jitter/bandwidth limit on logical WAN hops when load testing), and Fable is happy to send a bunch of agents at it and iterate until it gets the results it's looking for.
I think that if you are used to Sonnet medium or something, this will surprise you, but models like Fable and Sol on high/xhigh will really dig deep until they meet your goal. (I mostly use this for bug hunting and not perf, but ... I think it can do perf if you set it up right.)
1. Ask Claude Code or coding agent to research the internet, documentation, and Github for examples and learning working with similar problems.
2. Make the ten best examples of solving the problem based on the research.
3. Run each against the database enough times (100, 1000, or 10,000,000 depending) to empirically prove which is optimized. You can set other criteria based on your knowledge.
4. ???
5. Profit
The point is, it is cheap to research combing through 1,000s of examples and learnings and test the best and most relevant.
The mistake there is to point it at code to figure out performance optimizations.
The place to find them would be performance profiles, query plans, telemetry. The guidance for perf still applies, measure before and after change.
The issue is that the code often does not contain the information to do a perf optimization. Eg. you can't tell your cache size, the volumes of data in your DB or the latency of your network through just the text. Should you provide this context, you can get better results.
> The place to find them would be performance profiles, query plans, telemetry.
Tbh, once this information is available (which is the tricky part), in 99% of cases there's really no AI needed to analyze the data, since the 'low hanging fruits' will usually stand out anyway. And once you get into the area where optimization hotspots are no longer obvious, you're already deep in the diminishing returns area and optimizations for one use case or hardware configuration may degrade performance on others. That's my experience anyway.
> there's really no AI needed once this information is available
Most people are not arguing that problems are too tricky for a human to solve once presented with it, but AI can look at 1000 things at the same time across a whole code base and then just come back with the results a human can review.
True, but at least IME, once you get beyond fixing the 'obvious bugs' (how many there are depends on the initial quality of the code base - but I agree that AI is really useful to find those), you'll get into a murky grey area of "maybe false positives" which require a lot of time to validate. As a result you sink a lot of time creating and tweaking code-base-specific rules to try filtering out false positives, and IMHO this is exactly the tipping point where the whole thing becomes pointless because it becomes a bureaucratic monster.
E.g. a good code analysis tool needs to work predictably at button press on any code base.
Still better than nothing of course, e.g. I actually think bug scanning / code analsysis is indeed the one area where LLMs are actually useful, but it suffers from the same 'diminishing returns' problem as traditional approaches, if not worse (e.g. still no silver bullet, but a mostly useful additional tool in the toolbox).
It really depends on the context. On a pure computational code, it can work really well.
I recently optimized some code asking Claude to "make it faster" 2 different programs. For each, it wrote a benchmark, gathered initial data. Emitted some hypothesis and measured data around them with profilers, then did some changes (behind feature flags), checked the output was identical in either side and through profiling that the right code path was taken, and then benchmarked both.
And it did that for multiple successive changes in both codebases. Some changes were purely algorithmic (better complexity), some were related to tradeoffs between memory and computation (caching intermediate results better), some were about creating less intermediate objects to relieve the memory pressure, some where about a better memory layout to improve data locality.
The annoying part is that some of the code that was optimized was generated by Claude in the first place, so it's a bit frustrating that it didn't do those in the first place. But I guess, my iterative approach to making those tools didn't work on big enough data sets at first where it would have been an issue.
Yup, I do this regularly now. Whenever I think that something takes longer than it should I just tell it to set up a testing harness to measure each part of the process.
Once I have the baseline I'll just set a goal to improve performance by 10x. Works better than it has any right to. Have to be careful to exclude changes that massively increase complexity for tiny gains after its done, but the big improvements are often sensible choices that I'd also make as an engineer (i.e. more efficient data representation, caching and memoization where it matters, parallel processing, etc.)
A self validating loop goes a long way with getting AI to optimize stuff. We had some builds that used to take a long time (15-20~ minutes), that i just asked Claude to reduce the build time. It brought it down to 2 minutes and under 10 seconds for incremental builds.
Turns out folks had been copy pasting some setup code which took 2 minutes to run, and only needed to be run once per environment. Claude refactored the code to get it to run once, and since we had tests which validated the build artifacts, it was able to roll back if its change had broke the build.
I've consistently found that I see performance issues that the AI misses. It often says "that's not going to be what improves performance, it's noise" and then I get it to do it and it's like a global 30% throughput win lol. I think a lot of performance guidance it'll be trained on is shit - I see devs consistently misunderstand performance too and downplay the impact of anything other than "IO".
Which model/agent/harness tool did you use? I've found what you describe was my exact experience some ~6-8 months ago, but since about a month or so the game has completely changed. Using 5.6 Sol with highest reasoning setting in Codex or Fable in Code, the models come up with a list of possible improvements from static analysis (ranked by complexity/benefit), write and run their own custom profilers and deliver significant performance improvements with barely any input needed from my side. So this is no longer a model issue, it's a user toolchain issue.
Very hard to say anything definitive on this because it's a moving target, but last time I tried models still had a distinct sense of "consistently good, sometimes great at micro, bad at macro". Similar to how, even for relatively pedestrian CRUD, they'll do code that's objectively fine at the function/file/class level but can still make a mess if you don't supervise them at least at a high-level.
Thats fine but then Fable 5 should have requested this information instead of blundering along. So why didn't it? An expert human asked to do the same task would have surely asked for the additional data.
That's what I add at the end of my prompts: "Ask me questions on anything that's not clear related to this task" or similar. Both the CLI and the VSCode plugin have a nice interface designed for this, asking the user questions.
That's like saying Fable 5 should have told me how to prompt it to solve the Riemann hypothesis like Terence Tao would. The skill of the user still matters.
I've had a completely different experience. I usually know the problematic part and then ask the AI to optimize that exact part with benchmarks, telling it to try various different approaches, and it very often gains massive performance increases on various tasks, with me of course steering it and giving it pointers every now and then.
I was able to optimize a physical water simulation that was almost unrunnable on browsers to a buttery smooth 60fps version.
You have to learn to use your tools, not try whatever intuitively made sense to you at first (expecting the tool to do all the work) and then whining on Hacker News when it doesn’t work out for you.
We're reliving the early day Google era. It's an objectively simple tool to use, but some people just refuse to put any effort to learn.
I'm thinking the issue is probably that LLMs/harnesses are too easy to use? It crossed the thin line between magic and tooling and blur the mental model. If Claude Code were as hard to use as, say, ComfyUI, perhaps there would be less programmers having absurd expectation of it.
When I listen to some people or to stuff that comes out of the frontier labs, they indeed do sometimes sound like a religious cult talking about deity. I find it disturbing.
My favorite thing lately is that it's very good at resolving dependency security vulnerabilities. GitHub already does a good job at detecting and notifying which packages are vulnerable and at what versions they're fixed, and without AI, it's just a super tedious process to figure out what things all depend on that package, find a version that'll work for all of them (potentially including upgrading the things that depend on it), wire it up, test things, and make any minor changes necessary in our code.
It's not "hard" or novel--it's just tedious grunt work. I'm happy to have AI solving that one.
Echoing the other folks, I have a different experience.
I profile sql performance and LLMs find more opportunities than I could. All it takes is real data, a sql repl and an agent. Just ask the agent to use the repl to EXPLAIN and profile the sql. It works amazingly most of the time.
I did a lot of SQL optimisation with claude code over the last year, what made it work for me was making sure I verify everything with explain analyze. The whole loop was not fully “agentic” because of this but it was still faster and better than when done entirely manually. Claude also suggested plenty of small optimisations I would otherwise not do.
This is the research taste part - often they can be good but human experts are really good at this (also you know your codebase, without the prompt these models have no other background).
Then being able to suggest several things to try and have them go off and build, measure and tweak is hugely useful in my experience.
Also things like making custom visualisations for comparing changes.
People in the comments sound like they're literally asking claude: "hey claude, can you find areas for performance improvements." Nah, it doesn't work like that. You have to be more like: "can you profile [...] and identify hot spots." then looks at results that are suspicious. if you don't ground it to the real world reality of your project then don't be surprised when it makes up its own.
that's the first mistake i made as a new claude user. i only knew what prompts people were sharing on forums. but since they had been written by vibe coders it was all complete non-sense. also think about this: software engineering is filled with highly specialised tooling that can easily improve software quality. the thing is, most developers dont know how to use them, and if they do -- using them is so slow that its almost not worth it.
any tool you can name claude can use. obscure academic proof verifiers, memory corruption scanners, debuggers, profilers, coverage tools, backdoor supply chain pattern scanners... it can run these and have results in seconds. really is like living in the future.
Ask it to create a small sample to reproduce the issue in isolation. If it can't reproduce the performance changes then it drops it.
That might work better. I haven't tried it with SQL but it worked with a few complex UI issues I had. It identified the actual issue after a few false starts.
AI can uncover certain issues and resolve certain problems but it also has its blind spots.
I've been using Claude to create a software architecture diagram. It came up with a lot of useful functionality that we had neglected to show but upon further examination a lot of steps didn't make any sense. It added a box to do validation on some data then it put another box downstream to do validation again on the exact same piece of data (the second box had a different name but upon questioning, Claude conceded that it was the exact same validation step). When doing inference on user input, it put a box to extract specific insights from the prompt and then it put another box to do inference again on the inferred data to categorize it... And this would add latency and unnecessarily strip out useful context when doing the second inference... It really only needed a single inference step...
I re-uploaded the diagram to Claude and confronted it and it conceded to all of my points and it even noticed a theme between them and suggested that the diagram exhibited "a pattern of redundant steps." But it was incapable of suggesting viable solutions besides name changes. All of its solutions made the design more complicated. It was incapable of simplifying the design. I've noticed this with AI code; it can make things more complicated, add more features but struggles to simplify things.
Also, AI is horrible at rating things... It's just too superficial. For example, if you write a huge amount of intentionally over-engineered, tightly coupled code with poor separation of concerns but you do proper linting, define all the types, add a lot of comments, it will give it a higher score than a shorter, more readable and maintainable snippet which exhibits loose coupling and high cohesion because it lacks the superficial aspects.
It's horrible because now business people who understand nothing about coding may ask AI about code quality and it will consistently rank low quality over-engineered projects that are full of bugs, unmaintable and less secure, with a higher score.
Frustratingly, if you point out problems in its judgement, it will concede to your points without reservation, even building on top of your argument... but it will never actually tell you this stuff up-front, unprompted if you don't already know it! It never seems to reveal new insights, at best it can only expand on existing insight which you've already had.
It tells you what you want to know but not what you need to know.
Then you're not grounding it to reality properly. LLMs get bad pretty quickly when you just tell them to pull the answer out of thin air. Pin them to reality with actual performance tests to run to test theories and the results will be much better.
Performance optimization involves simulating real world loads and measurable results. Give them the opportunity to actually have a closed loop if you want more than trivial improvements.
Have you any example materials that show a harness that an agent can be pointed at an app with some sort of telemetry tool, the results it gained and the cost of doing so? Because my experience is the same as the parents - the LLM goes on massive tangents, and the more tangents it goes on the worse the results get.
Tell the coding agent to install
or build its own performance and profiling tools.
A few lines of Python with a time.time() call is enough to start iterating on performance improvements, and the really good models know how to use much more advanced existing tools.
Not sure what materials your hoping for, but I've pointed codex at pprof application endpoints and just told it to identify hotspots and propose lowest hanging fruit/highest ROI items to fix, which I then approved it to open PRs for. Reduced application CPU usage by ~30% on loaded servers.
It's not magic. Fairly obvious oversights that I or someone else on the team could've found and fixed if we had looked, but it was never a business priority so we never did until I curiously spent 5 minutes asking it to do it for me.
Ultimately the harness is me and the experience is like managing an unruly toddler. You have to pay attention to what it does and issue corrections. Skills and prompts and AGENTS.md do some, nested sets of agents do some, but over it all is me keeping track of what it's doing and needs to do.
You have to size the unit of work to its useful attention span, you have to have code architecture that is conducive to units of work, and you have to have tools like ticket managers, etc. that keep the big picture and smaller pictures in mind. Reverting from agile practices to more pre-planning whole project documentation and architecture decisions helps.
But in the end it's you. LLMs have their limits and need humans to direct them.
I had good results when I started AI with building the testing for the actual problem, then it can just run itself over and over without wasting much time on verifying its output.
And at worst you still saved a bunch of time on writing the tooling to test the issue
The trend I've noticed is that AI struggles to think outside the box when making optimisations, which exactly what's needed when you've made all of the practical DB and logic optimisations to the existing code.
Often you need to take a step back and question how the system is working and if there would be better ways to design it so the bottlenecks you're hitting wouldn't exist in the first place. Caching things, adding indexes, tweaking logic – these can help, but you'll quickly hit diminishing returns once you've done all of the obvious stuff.
I've seen people here say how AI is great at optimising code though, but I'm not sure if that's because they're giving it optimisation problems with a lot of low hanging fruit or if they're successfully getting AI to rework their systems to remove bottlenecks. This one area I find AI to still be particularly bad at.
As others have said, these were uninformed guesses which LLMs seem still to be willing to hand out deliberately. If you provide more information (your schemas, the distribution of values in your database tables, the EXPLAIN ANALYZE outputs, DB configuration etc.) it will do a much better job.
I would never undertake a big performance optimization without first measuring that area's total impact. The LLM would then be great to work out the refactoring and you can give it your bench suite (which it probably wrote) to work against.
I guess just another area where the LLM is useful only as long as you remain in charge using your own programming experience as a guide.
Yep, as a performance specialist I've had the same outcomes - it'll find "smoking guns" galore which will sound plausible and be completely wrong.
But that's ok - I still get a lot of value of the tool elsewhere (e.g. writing data analysis scripts, exploring code, breaking down bug artifacts...), but it doesn't stop the tiring BS from other people going "why not just AI it?" (especially from managers).
I usually let them try to "just AI it" and most of them learn quickly that it's not that simple. (The others stay deranged and are making my job miserable.)
That's uninteresting as long as you don't specify the model you used. For example, Mythos was far better at finding security bugs than previous models.
The real datapoint was Firefox not paying any money in Berlin's Pwn2Own competition round this May just gone. Unheard of to have nothing confirmed... they've paid out every event since 2007 (I checked). Does this mean we must move past the low-hanging fruit now? Probably... Certainly indicates some usefulness of these models.
Not that I don't believe its possible to fix a lot of bugs, I also wonder what the actual dynamic was. Were the people in team working much more than usual as well? Given its Google, I wouldn't be surprised if there was an "internal push" to fix more bugs over next X sprints so that they can publish this blog and some manager can show impact and AI adaption to his superior.
Google's entire modus operandi has been "automate everything" for decades. They've been doing this with fuzzers, with project zero and so on. Adding LLMs on top is a very obvious next step. And LLMs improving and finding more bugs also follows. Then improve the harness and the dev tools, to better use the LLMs. And then everything together end-to-end to find-triage-fix-confirm. Your LLMs are as good as the loop they run in, and the loop is as good as the verifier. Seems a reasonable enough dynamic without (or despite, depends how cynical you wanna be) the need for managers to show number goes up on some chart.
I would guess they have been confronted with an initial flood of newly discovered bugs whenever they added a new analysis tool or approach (e.g. automated static analysis, fuzzing, ...). Once working through that initial flurry of newly discovered bugs one would assume that the frequency goes down again.
> In early 2026, we saw a gradual increase in all categories of bug reports, but by March, the shift was apparent: we received more bug reports than we had in the entirety of 2025
sure, moreover - maybe big AI usage significantly influenced the amount of bugs.
So, the picture can be like that:
- 2025: 50bugs found, 45fixed
- 2026: 500bugs found, 450 fixed
This is just my hypothesis, but I suspect that at Chrome‘s engineering team there has been a culture of inaction for the past decade or more. That nothing gets done, no bugs fixed, unless some higher ups at Google sees a business interest in resolving it.
And what has changed now is that the higher ups at Google do indeed see a business interest in fixing bugs and giving the credit to AI to sell us more AI.
1. Our backlog of bugs gets processed quicker because instead of staring at the code for 10 minutes fiuring out what's happening, there's a tool that can reason about it quicker.
2. Code reviews and security reviews happen quicker and produce more findings.
I would think that (m)any team(s) using AI might also be seeing a higher rate of finding and fixing issues.
Even the Linux Kernel (I'd say Windows and Apple too) are seeing the same phenomenon.
The idea that software has gotten so complex that a machine can evaluate code paths better than a human, seems to bristle the fur of many. Some people didn't think we would see the day where that comparative human limitation was laid bare in simpler tasks than they expected. I believe older developers are less likely to be offended, having to deal with this as a matter of course (as the mind declines).
I don't know, this more and more feels like a sentiment projected on to people than anything real these days.. Like even in that linked thread, is personal offense like you lay out here really were you can place Laurent Pinchart's push back? You don't read anything else there at all?
And either way, what, we are going to keep this line going for another 5 years? Aren't you bored?
Exactly this. And there are few bigger targets than Chrome when it comes to finding exploits (OSes and network equipment are probably on par). I'm sure they have devoted large compute resources and human staffing at making sure that they find and fix these issues before anyone else does.
A lot of people here seem to be living in a different universe than me or simply don't know how to work with AI. I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you. They get mad at Excel for the poor investment returns. At this point, this is such a strawman, it isn't worth counter arguing.
I think I'll abandon this discussion and keep using AI quietly while exchanging tips with like-minded people who are interested in using it properly and efficiently.
> A lot of people here seem to be living in a different universe than me
I feel this way on this topic too.
> I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you.
The problem is; how _should_ I use AI? On a previous thread, I had two replies to the same comment, one saying "provide the LLM all the context it needs and let it go ham", and the other saying "Carefully guide and craft it and review everything". Both subthreads had people agreeing.
My experience with LLMs is leaving them unattended gives very poor results within a handful of iterations, and that carefully guiding them can be a value add, but the effort ot do that is very often as much as just writing the damn code myself, particularly if there's a few iterations that go on.
I think the main cause of different views is just the fact that people have very different products and codebases they're working on. Therefore, the requirements and applicability of AI varies a lot, case-by-case. It would be useful if people always explained their exact use cases when making claims one way or another. I understand it's not feasible for many use cases to reveal exact details, but at least some transparency would be great.
I've found AI to be immensely useful in gamedev, mainly UX development, research, and curation. In those areas I let it go quite wild, with mechanical and visual tests for UX, and summaries for research and curation. I've also made skills, custom subagents, hooks, and scripts (all made with AI assistance) to streamline development.
Gamedev is a good example of where AI shines, since the risks of fucking things up are not so big. If I worked on, let's say banking, I'd be much more careful. On my day job, as a data scientist (moving towards product owner) in a large industrial company, I'm much more careful. UX can run on vibes, but anything that touches data processing or business logic needs to be looked at by a human, row by row. Depending on use cases, it may be fine to look at only the tests. The main problem is that there's so much stuff to review, and maybe 98% of it is fine. It's then very boring to try to spot the 2%. I don't have good solutions for that yet, and idk if anyone does.
> It would be useful if people always explained their exact use cases when making claims one way or another. I understand it's not feasible for many use cases to reveal exact details,
It would be useful if people _ever_ explained the use cases. Even Anthropic when they're publishing their materials are hand wavey about this.
> Gamedev is a good example of where AI shines, since the risks of fucking things up are not so big
I couldn't disagree more. I run an engineering team in GameDev, and AI is great at making a gigantic mess. It's great for generating a lot of code, but the minute that doesn't do what you want it to, you now need to go back and learn _everything_ that it's generated. And IME, doing that is as bad as having just written it yourself in the first place.
> he main problem is that there's so much stuff to review, and maybe 98% of it is fine. It's then very boring to try to spot the 2%. I don't have good solutions for that yet, and idk if anyone does.
I tentatively agree here, except I've found about 50% of it is "fine", 40% of it is "this works but isn't really what I want to put into my product" and 10% is "absolutely not".
AI is great at reading, so give it things to read. Claude will try to throw big markdown docs in your repo, but you don't have to do that. Make it keep docs by the code.
I include a line in AGENTS.md that says "We *always* add docstrings to methods, classes, structs, and namespaces - there should be 100% coverage of docstrings."
It will make better choices about what functions to make or remove if you force it to justify why the function exists.
And when you have to go back and understand, it becomes easier.
> It would be useful if people _ever_ explained the use cases. Even Anthropic when they're publishing their materials are hand wavey about this.
Yes, good addition.
> I couldn't disagree more. I run an engineering team in GameDev, and AI is great at making a gigantic mess. It's great for generating a lot of code, but the minute that doesn't do what you want it to, you now need to go back and learn _everything_ that it's generated. And IME, doing that is as bad as having just written it yourself in the first place.
Yeah, I should have been more specific since our use cases are very different. I'm working solo on small games, so it's easier to keep the AI on a leash. I can imagine it can be a nightmare on large code bases with many collaborators.
> I tentatively agree here, except I've found about 50% of it is "fine", 40% of it is "this works but isn't really what I want to put into my product" and 10% is "absolutely not".
Yeah, probably our use cases are very different here too. We mostly develop/maintain dozens of microservices for internal use, mostly greenfield, relatively small in scope, and with 1-3 developers in each. And maybe 98% was a bit too optimistic. The larger and older the codebase, the more problems AI tends to create. But it's very good at helping to understand old codebases (even those written by myself haha).
Oh, and esoteric data science topics can be a minefield too. If there's a topic that's new to me, It's difficult to know what is hallucination and what is not. It's best to let AI suggest alternative methods and then study those yourself.
It’s a very soft skill that changes as the capabilities of AI change.
But one thing that doesn’t change is the need to specify an end goal correctly and precisely. I think the emphasis of knowledge/information-processing work is going to increasingly be placed on verification mechanisms. This is practically equivalent to precisely defining an end goal.
Spend time deeply thinking about what it means for a solution to be correct. What properties will a correct solution have? What of those properties are testable? Write those things down and tell the agent.
As models get better, agents will be able to target more and more difficult end goals. The strategy just becomes more useful. (It’s useful for people as well.)
For example, if you want an agent to write a photo editor, think about what end properties the editor should have. There are reference images for color space and rendering transformations. That’s a good start.
Sometimes the goal will be fuzzier. “I want a feature like CaptureOne where I provide a reference image and it makes my image look like that.” Well, time to think really hard about what that means. Iterate with AI on how to test for that precisely. Come up with some good metrics/heuristics. Maybe it means local contrast should match. Maybe the overall distribution of colors. Maybe something more complicated.
Then you have a target and you can let an implementation agent work against that. If it fails, it’s either because the agent is bad or your target was incorrect or incomplete. As models get better, the limiting factor becomes your ability to correctly define a problem.
Latest models like Opus 5.0 work well in both scenarios. You can just give it a goal and let it make a plan, review the plan and then let it go ham.
Or you can also use it piece-meal and have it do parts of work that you want. It works in both ways and works really well.
I don't think there is any model right now that you can use with 0 oversight, but you can do pretty complex stuff without writing single line of code at this point.
The truth is that using AI well is a skill much like anything else. There's no one single way to use it much like there's no one single way to write a program that serves hackernews.
The reality of the current situation is that we have dozens of families of different model; each of which is differentiated further into various revision numbers and power levels, and depending on which one you use and what the nature of the tasks is, you very much need a different usage style that can only be discerned after multiple (Sometimes very many) sessions of using that individual model for that category of task. It is not one size fits all, and trying to pretend that it is will lead to failure.
> It is not one size fits all, and trying to pretend that it is will lead to failure.
I'm not asking for a one size fits all solution. I'm asking for "vibe coding with Codex on GPT 5.6 with high is the way, but if you want to be more ivnolved, Sonnet 4.6 on Claude is the path. Here's the proof."
Saying that it depends and it's impossible to quantify allows the camp who are claiming it's more productive to say "you're holding it wrong".
What can be concluded is that some get immense value out of Ai and some are incredibly critical of that claim.
I do not know what the secret sauce is.
But I have a responsibility to the team I lead. I need to protect our velocity. Which means that my hiring practices now include specific checks that people are able to work with ai.
In the end, it might be a step thing. Some people are just inherently incapable of working with this technology...
I don't generally use AI to write code I feel I can write efficiently. In these cases, I'll just ask AI for a review. But when I need to write code in areas I'm less confident about, I'll let AI take the reins and the do a thorough review myself. These instances are not too common, though.
I also ask direct questions to AI for brainstorming and advanced programming language usage. In these cases, AI isn't touching the code.
This (fixing bugs, making code better, refactoring), is the best fit for AI.
I think a lot of us were excited that we will finally be able to polish our old software.
Many of us were never given the opportunity and were tasked to produce new features at an ever increasing pace. Depending on where you landed you can be either elated or jaded.
You can today develop a mobile app with 20 screens without ever reviewing any of the code generated.
If you test each feature and have it iterate on your feedback, you can build a decent product in this way.
Does it write too much code? Perhaps. Could I bring those 100k lines through code review in a team where some members nitpick? No.
That does not change the fact that it works, and that you do not need to read the code even today.
Unless progress slows down dramatically and soon, it seems rather likely that most of us will not be reading code in the near future. Their marketing is not wrong.
Reminds me those Bitcoin threads where I would explain a use case it solved for me and everyone was like "nope that's impossible, there's no use case". And before the inevitable replies come in: no, I'm not saying AI is like Bitcoin.
> They get mad at Excel for the poor investment returns
This is quite some mental gymnastics. Is your presumption really that detractors = stupid people who can't differentiate between the challenges in getting Excel to do some specific thing and blindly interpreting an incorrect calculation as an actual accounting assessment of their business?
> I think I'll abandon this discussion and keep using AI
Not joining in the first place to provide such nuggets of wisdom was also always an option
Did you think for a second that maybe other people work with other languages, domains, tools and work approaches than you?
It's so bizarre that noone realizes that there's a bit of a difference between banging together JavaScript into websites, writing firmware for widgets, herding servers or fixing medical devices?
This right here is the truth. Too often people talk in abstract ways, not disclosing what they're actually working on. And then they tell their experiences as universal facts.
The goal of software development is to eventually be done. Yes there are new incentives or new use cases that come up, but the aim should be towards a stable state where you're barely have to work on the software anymore. It's like an house where you spend a good chunk of money to build it, but afterwards you just maintain it.
That's my main issue with AI hypers. There's often no goal in sight, they're just busy for the sake of being busy. They talk about their process, but not their objectives. It's not about progress for them, it's only about being in motion.
How many of those automated fixes were reverted? How many introduced a new bug? What's the false positive rate on the finding agents? The post has counts for everything that went right and nothing for what could go wrong.
Exactly, also it lacks a lot of context as to why they did that.
Possible (probable?) scenario:
- Marketing: "we found and fixed lots of bugs thanks to AI"
- Reality: the KPI is now to fix as many bugs as possible with the help of AI, so they used AI to search old and easy bugs in the backlog, and then fixed it manually
Huh? Maybe a third scenario is that AI helped fix the bugs? Like described in the actual post you are replying to? This level of conspiracy theory is getting a bit ridiculous.
I think you’re right, my base assumption is that the models can code and can fix bugs, and can code more in parallel and faster than humans at a lower cost.
If Google are tackling lower value bugs with AI the number is in a way inflated compared to some utility measure (fixing a smaller number of worse bugs could be preferable) but it’s still things fixed.
At my company there's a lot of discussion about AI and complaints that people run out of tokens within a day, but zero results are shown. No measurable (or measured) gains. Or nothing that people are willing to talk about, in any case.
At my company the performance improvements channels has been exploding, with people claiming giant improvements in latency, throughput, and decreased cost of the services. The cost decreases itself is order of magnitude more (at annualized run rate) more than we pay for tokens. YMMV.
I definitely make use of AI but in my experience I almost always could have done it better myself, the places where I threw AI at the problem I didn't care about the results being good, only good enough.
When we see memory and compute requirements for version x+1 of software decrease instead of increase I will happily say AI is the oracle people proclaim it to be.
In $COMPANY, for the mid-yearly review, the employees were asked whether their AI usage was 1/ efficient, 2/ adoptive (integrated in the way they work) or 3/ transformative.
Saying “never used it” or “I tried and it was useless” was literally not possible.
I'd say that one is not really an issue. In 2012 the Pinkie Pie exploit chain already required chaining 6 bugs to lead to an exploit [1]. Since then we've seen chains requiring more than 10 bugs (!).
If you fix any one of those bugs, the exploit is non-functional anymore. Sorry out of luck.
So if, say, for every ten bugs you fix, you introduce two new ones then it's still a very net win. Unless of course it introduces a bug so bad it becomes a simple exploit not requiring a long chain of exploits.
But in the case of browsers we've only ever been moving to longer and longer chains of exploits required to pwn a browser.
A great many window of opportunities are closing for dark-side hackers / north korean intelligence etc.: there were probably exploit chains still open for exploitation in April that just got closed by Google.
If anything, besides the supply chains attacks in amateur-land, the world didn't stop working: projects (not just browsers but OSes too) are being hardened left and right.
Using AI to find potential bugs is an amazing use case and there really aren't many downsides.
> The post has counts for everything that went right and nothing for what could go wrong.
I'm not saying there aren't a few downsides but the benefits are just too good to ignore.
Or people don't ask questions for which the answer is known to be "None, really."
I get that many don't like what LLMs are doing to the industry, but this is just incorrect reaction to a very specific benefit that's proven beyond doubt (Security hardening).
Accept it imo - LLMs are solving very large problems that have plagued software security.
Sorry it is your reaction that is really weird. These are legitimate questions, and I have definitely seen Claude finding the wrong cause and then implementing completely incorrect/irrelevant fixes, only to find that it didn't work and need to start over. Not saying humans don't do the same thing, but LLMs are far from perfect, and it would be delusional to only talk about successes.
I've done security work before, and I've done it now with frontier. I've seen the difference first hand, and as the OP and so many other articles show, so have the leading experts in the world.
So either they are lying, or you may not yet be seeing and experiencing what they are. If that's wierd, ok.
You did not address my points or those in OP but kept repeating yours (that are not even relevant) in a handwavy way.
And that's fine. You can continue to live in your bubble. Others simply have different experience, and your asserting "AI is perfect" on a forum is not going to matter in terms of everyone's own, first hand experience.
Worrying. Extrapolating a (speculative) future, this means (Google will feel) that soon the chromium base will no longer need the crowd-sourced bug hunting that is open-source. I expect Google to eventually stop working on chromium (in the open) and all current chromium-flavours will become de-facto forks of the last published version of chromium. These forks won't be equally easy to maintain given that the groups running them do not have access to the same level of subsidy as Chrome does with Gemini.
Which I do (actually a fork of firefox - waterfox), but it is still worrying as the viability of chromium-base browsers is what's keeping us having "merely" a chromium-monoculture (with viable ad blockers e.g. and website operators that check compatibility with more than one browser), rather than a chrome-monoculture (where ad-blockers surely would have been killed by now).
AI critique often funnels itself into a narrow bucket: creating code blindly with AI is bad. That's easy to grant.
Adversarial testing, checking developer assumptions, refactor suggestions, small dev tools and even some guided coding all sit on the other side of the spectrum of what you can do with coding and AI. For larger and larger codebases even simple things like tracing dependencies or behavior might be greatly aided.
And the critiques reserved for that narrow bucket on the other end, blindly generating code, are too easily conflated with the rest.
Yes. AI is a tool, it’s supposed to be used a certain way, anything else is a misunderstanding og what AI is. You have to aim it in the direction you want it to go, not expect it to solve all your problems magically
AI isn’t “supposed to be” anything (other than “intelligent”). It just turned out that LLMs have to be used in particular ways to be useful. Nobody planned or designed them to be that way. We all as users are put to do the job of discovering the ways.
LLMs are increasingly not tools anymore but agents. The difference is still one of degree, but it's clear in which direction we are currently moving. It was even clear to some people 10 years ago: https://gwern.net/tool-ai
Can't have been much, as the Chrome project is 20+ years old and LLMs have only recently become a thing; they did not slacken their code review and testing practices with the advent of LLM code generators. And I doubt they do much development on the affected areas at the moment, they mention a 13 year old issue for example.
But it's an open source project, you can go and figure out whether your assertion is correct.
If people are handing off the coding to AI more and more then they'll be less skilled at spotting potential bugs in the first place. AI, course of, could be run to find potential bugs in the AI-coded features before commit. But then you're getting further and further towards a world where code, and code that runs infrastructure, is incomprehensible. We'll have to start sacrificing the first-born to the silicon in the hopes of a good build for code that controls your electrical sub-station.
Look at the git stats for the project. Lines of code submitted haven't changed dramatically. Not everyone is just merging slop into the codebase. There are many ways to use AI, and most established players are not merging vibe coded nonsense.
If you assume that AI can produce code with less bugs, your 100 times increase in fresh bugs means you also increased your speed for new functionality by more than 100 times.
If models can spot 13 years old critical bugs, I think they can also produce fresh code without those bugs. The skillset is the same.
That is a stat you invented out of thin air, based upon nothing but pejorative hope, against all logic or rational consideration of the timeline or the scale of the project, and you declare it the "elephant in the room"?
HN is so weird on the topic of AI, and so many desperately are trying to contrive a reality. Odd stuff.
The elephant in the room is how many people in this discussion are actually alien lifeforms.
The elephant in the room is how many turtles obtained law degrees.
The elephant in the room is whether this is worth discussing because we're all in a simulation.
No, it isn't the elephant in the room. You made up some horseshit that betrays how little you know about the field, which will get upvoted because HN has a hearty contingent of "if I pretend AI is actually useless, maybe that will manifest in reality" zealots. It's bizarre.
Turtles with law degrees don't exist, but LLMs that increase tech debt in code bases do. Maybe recent ones got better, and it's useful to quantify the progress being made.
It's so hard to believe these kind of affirmations coming from a firm that has so many hundreds (thousands ?) of billions invested directly and indirectly in generative AI...
Just admit that Chromium is too complex at this point, we've all seen show HNs doing what could be considered magic in Chromium. We probably need to issue a mortarium in new platform features and focus on a deep clean in bug fixes. I've said that Mozilla is too busy doing redesigns than bug fixes as well.
No, the extra code velocity allowed by AI without proper QA support means there could have been a lot more bugs introduced into products over the last year or so, giving AI more bugs to find. It’s tongue in cheek but not completely uncalled for. It’s been long known that one dishonest way to inflate your bug fix count is to put more bugs into the product to fix. Goodhart’s law is merciless.
I wouldn't say "magically", it's what happens with me and every other engineer I know: we're all detecting bugs and yet we're all writing them too. Not saying that LLMs work the same as humans, but saying it's not a logical fallacy.
The privatization and obfuscation of the security industry is worrying.
Google claims to help open source developers, but donates millions to the Alpha Omega Foundation that did not give access to Mythos to Curl and only found one issue.
The Alpha Omega people appear to be selected on physical appearance and take the money away from open source without doing much.
The entire Chrome article is obviously directed at suits and unsurprisingly hypes AI for better promotion chances.
No worries, writing with AIs can improve the speed bugs enter the code base as well. Would be interesting to know the breakdown.
> the “latent security issue.” Code that is safe and robust in isolation can be transformed into a critical vulnerability by an entirely unrelated, minor logic change elsewhere in the tree.
Similarly, you can accidentally fix a bug without discovering/triaging it, so the pointless repeated bug lifecycle is incomplete
I have a software degree and some work experience but have been doing other things for a decade or so.
I'm curious to try out some "vibe engineering" but it seems a bit daunting to get started, there are so many new tools available on top of the AI itself.
Are there good resources for somebody like me to get started, that'll guide me though how to think about prompting, and using CI and how go have the agent successfully write specs and tests and what-not. I don't mind paying some for a good course, but I find it hard to figure out which one would be worthwhile, or whether there are some youtubers that would be better to follow.
The best option at this point is to just sign up for a paid plan with either ChatGPT or Claude and then ask the model the same thing. My preference would be for ChatGPT and if you've been out of the game for a long time then using the desktop app might be the best choice https://chatgpt.com/codex/
Then try starting with voice mode (if you're comfortable chatting out loud) and just talk your way through it.
They keep breaking inertial scroll on Mac though, which is something that irritates me thousands of times a day.
First they made it so when it came to rest it would then start scrolling again and overshoot by 2 lines before coming to a halt.
They fixed that after a few weeks then it worked until last week and now it’s left in a state where as it’s coming to a halt it jerks every time like dropping frames of the scroll almost.
I know it’s not my machine, config or trackpad because I’m seeing it on multiple computers.
The article is great reflection how LLMs are actually doing Quality assurance on both security and codebase sides, but i still doubt the creative part that means UI side, So far in my experience LLMs at best are unable to determine what the good UI is , may be its not inherently a bug detection rather a creative process
I recently switched from Chrome to Edge after using Chrome for decades. It is so bloated and slow. I can have same number of tabs open in both and Edge is always 30% to 50% more efficient.
spent a several months working on their bug bounty program earlier this year while working full time and lost a lot of motivation after a duplicate combined with lower bounty awards and claude safety guards all at the same time
I can't comment on Chrome bugs, but I can comment on Safari bugs. According to a search of the WebKit Bugzilla, I'm the most prolific filer of web inspector bugs outside of the WebKit team itself.
Recently I noticed an unusually high number of web inspector bugs fixed by one longtime WebKit engineer, and I suspected AI assistance, though the commit messages include no disclosure of this. Nonetheless, the increase in volume was quite dramatic.
The other day, an update to Safari Technology Preview was released, and I attempted to verify whether a couple of my bugs were indeed fixed. However, it was impossible for me to verify, because the latest version of Safari Technology Preview introduced a new web inspector bug that totally broke the features I was testing.
Thus, I remain unimpressed.
Without a doubt, LLMs have demonstrable skills and can produce code much faster than humans. I never thought that producing code fast was wise, though, even before LLMs arrived. For many years I've criticized software development based on management-driven release schedules, where developers are forced to pump out code regardless of quality, regardless of whether it's ready. Your boss may not like it, but code is done when it's done, not when your boss says it has to be done. If software complexity were predictable and reproducible, then indeed management could replace engineers with automation, but that's not how it works in reality.
Back to Chrome, the browser is notorious for continually adding invasive new "features" to the web that nobody wants except advertisers. I wonder how many Chrome bugs were the result of Google push push pushing all of this new crap on users over the years?
Good. Now please fix youtube. Amount of bugs that I'm getting in youtube app on iphone should be embarassing for company like Google. I cannot believe how bad it become in last couple years.
I wish they'd spent more time on improving the UI now. It's largely been the same since Chrome appeared. Their recent vertical tabs are so much inferior to Dia, which is built by a much smaller team.
I’ve heard from people inside Google that they’re prohibited from using anything but their own models. AlbertaTech on YT, ex-Google working on YT said in a video that right before she quit someone had threatened to quit if they couldn’t get access to Claude Code
Kind of, but you can also assert it yourself; the big tech companies have billions of lines of code not publicly available, they can train their in-house code assistants on those instead of or in addition to what's available out in the open.
Earlier today I opened Chrome on my iPhone, went to the Reading List. It started flicking fast and then the phone restarted. That's a first for me on iPhone.
Maybe they could ask the AI to fix their atrocious build times. Only in Chrome do you have 300 line source files that blow up to 20 megabytes after preprocessing as a matter of course. 3 GB per compile job is getting more common - and RAM is expensive right now!
Old bugs at the bottom of the prioritization list deserve to be fixed.
Some were probably just theoretical "if we have over 2B of this item, it'll crash because X/Y/Z and that can be exploited", some are probably in rarely used features.
Once the LLMs are done going through the code, the baseline will be all better and they will probably fix a lot less than they have now. Or they will have LLMs automatically check every single crash bug that has been submitted and take care of those too. I am assuming that they are probably already using this as an input source for their AI.
I've mostly only used Codex for reviews, for Godot/GDScript code, and it helped me catch bugs that would've taken me ages to even notice on my own; games be tricky like that.
Claude was mostly useless or annoying up until the last time I tried it (about 3 months ago)
You have to be vigilant though: Codex often picks out obscure edge cases and suggests adding multiple new functions and flags to avoid issues that would be better off left as fast-failure crashes. Maybe that's because of the 5.6 Sol Max I leave it on.
The biggest bug still has not been fixed here, which is:
Google.
We really need an alternative to this greedy and evil corporations
de-facto controlling a huge portion of the modern www stack. All
decision-making processes are here subjugated to what fits Google's
adEmpire. This is a perpetual system of control amplifying abusive
systems in place. We already see this with the recent age sniffing;
Google just yesterday announced that all android users must hand
over their age to everyone else. Before that google kicked out
or locked out open source alternatives to android (or, at the least,
made their life significantly harder than before). These are not
isolated ways of abuse - this is systematic abuse.
We really need alternatives to Google here. It can not be that one
tyrant company dictates so much of people's open lives (and no,
Firefox is no alternative; Mozilla is basically a nerfed and bribed
entity that has given up on firefox many years ago already; ladybird
may become an alternative at one point in time, but right now is not
and there are questions over how Kling handles the overall project,
but these complaints are significantly below what Google is doing
globally here).
Fixing more bugs in the adChromium code base does not change the underlying problem at hand.
Seems like they have enough AI they could find a way to put ManifestV2 back into the browser and fix things we actually wanted instead of putting more bullshit "AI improvements" we didn't?
give it another year and AI will be filing bugs faster than humans can merge the fixes, and the Chrome team will need a separate AI just to triage the first AI's bug reports
A company valued at $4+ trillion based on the recent AI hype alone is touting the benefits of using said AI, and we're supposed to take this shit for granted.
I'm wondering what the Alphabet employees still commenting on this forum have to say about it? I guess for the right comps they can keep their mouths shut no matter the high level of idiocy involved.
There is hype, but there are also real tangible wins as well. Analyzing code for bugs is something that works very well. We use it for helping us revert changes that lead to a flaking test and it's dramatically faster than previous techniques. It's great at suggesting duplicate bug reports. It's amazing at scaling an API migration throughout a codebase. It's great at prototyping an idea before you decide to do it properly.
There are many things it's not good at (eg design and architecture), but that doesn't invalidate the things it is good at. Everyone is learning how to best use it and there are growing pains. Ignore the hype and try to find the common patterns in what people say.
I don't know about Chromium specifically, but in general Google requires 2 humans to have looked at every change (normally one is the author, but not necessarily for bot-authored changes).
They did roll this policy for Chromium changes a few years ago too.
And it wasn't just 2 humans but 2 Googlers. One Googler author and one Googler reviewer works. But an external author needed 2 Googlers to review changes, and all files needed to be reviewed by one author at least.
So if you touched multiple files owned by different people for a more complex feature, you would naturally get there.
I am actually using AI to learn about Windows OS performance. I have a couple of prompts scheduled to research, test, and implement performance gains in the OS, then post them here [0] and here [1]. I always admired Brendan Greg and Bryan Cantrill at Sun/Illumos/ for their work on OS performance but never had the time to sit down and catch up to their level due to my time in the military.
I got Codex to find and apply a bunch of settings I didn't know about and in the future I hope to use AI to make real changes in performance that venders probably don't have time or resources for.
that being said, what I've learned so far is essentially
WhyIsItAlwaysHN's entire comment "The place to find them would be performance profiles, query plans, telemetry."
I just find it so frustrating that for a user interface that was basically solved in the 90s, Windows GUI still struggles to survive on hardware exceptionally more powerful than what we had 30 years ago
I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change.
In fact, if it wasn't for the fact that it made making the actual changes I identified much easier (move these joins into a CTE etc) it would have been a detriment. Not only did I get sidetracked by a bunch of useless suggestions but I also had to put up with others dumping their raw AI output at me as if it was somehow a meaningful contribution.
The thing that makes it work really well is to make sure it has all the tooling to verify its hypotheses. If you allow it to run the full lifecycle in loops you will be surprised how well it works.
Built exactly this for work a few months ago. Claude has a read-only account to the dev DB and a bunch of python postgres tools, I point it at queries/views and ask it optimize, and it has to:
1) verify identical query results
2) run repeatedly to get average, worst, best, etc duration of runs
Sped up so many legacy things that none of us were ever going to bother with.
the point is more: are people going to flood the zone with infinite bikeshedding that drowns out the real gains
What is bikeshedding supposed to mean here?
It comes from C. Northcote Parkinson's work, where he calls it "The Law of Triviality". The idea is that people clog up decision-making by introducing a swarm of issues that have little relative impact or importance for the project at hand, leading to a massive misallocation of attention. His example is a $10 million construction project for a Nuclear Reactor where as much time is spent debating the materials to be used for a $3000 bike shed as on the actual reactor buildout. The law of triviality says "the time spent on any item of the agenda will be in inverse proportion to the sum [of money] involved."
Source: C. Northcote Parkinson, "Parkinson's Law and Other Studies in Administration" (1957)
Thanks. I know what bikeshedding is. I’m asking what cyandees specifically means. The normal definition of bikeshedding doesn’t seem to make any sense in his sentence.
I’m assuming he’s using it to mean “cruft” or “low value features” but maybe there’s some meaning I’m missing.
Happens without AI at the product level all the time.
Just like humans, AI needs a "reality" to check things against. If we don't provide that in an automated way, somehow, it will always make mistakes and we'd have to point it out. Since, at that point, we're just mediating with what reality gives us and what an AI should do with it.
AI doesn't have enough senses yet. It's trapped in a box.
The AI box experiment has been resolved. We let the AI out of the box if it promises to summarise and desummarise emails.
I’ve had a lot of success with this. Give it a good enough harness, leave it for a few days and see what it finds that actually holds up.
Exactly. In the above case, AI should be able to verify it's own hypothesis by having access to the database's table statistics and planner.
Very few problems really have "all the tooling to verify its hypotheses" though. even if you want to construct such an harness.
Also let me ask you why we need better and better and models if what we have already can produce good output with 'all the tooling to verify its hypotheses'
We don't. You could freeze model development today and we would be able to use our current models to effectively optimize existing code for years to come.
But them it's more like AI assisted performance fuzzing? Just throw stuff against some measuring harness and see what sticks?
Yes, sort of. Not that there is anything wrong with that. Fuzzing yields tons of trash results, the occasional fleck of gold and the even more rare nugget. Then you analyze how it found the flakes and nuggets update your fuzzing harness. Every so often you get to write a paper or give a talk about a breakthrough to share meaningful improvements.
Current AI isn't super effective at making the breakthroughs, but it sure is effective at democratizing the ones you can point it at.
What does that mean? Democratising the breakthroughs you can point LLMs at?
That's how anyone will do it, read code, build hypothesis, test. It may not be very good with ideas compared to human but it will be cheaper.
What tooling makes this go?
The same tooling that you'd use. AI is not magic, it cannot know exactly how a program will run under your exact circumstances and data. Therefore, if it can use a profiler, test suite or application, or be able to read logs, just like you would, it then can make changes, evaluate them, and iterate. Personally, I've created test MCP servers that will perform actions like submit orders, have the AI capture and analyze PerfViews and Concurrency Visualizer, write out tracing data to logs... all the same stuff I was doing by hand for years.
Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD!
I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.
It bothers me that you have to explicitly state this to the agent. Makes me think what else is missing from that file which also needs to be explicitly stated, but I don't know what don't know. "Do a good job"?
This is where the harness shines. It's why products like Cursor have been growing like crazy. If you drop an extremely intelligent person into a problem they've never seen before, the initial context and knowledge-sharing just to get them up to speed is immense. They'll make a lot of mistakes until they're ready to fix your problem. The harness can leapfrog most of that. There is a booming industry in harnesses with no saturation in sight. We're not even close to maximising the utility of current models, let alone the amazing improvements which are happening week to week.
How does the cursor harness help here compared to something more basic like pi coder? I'm talking specifics.
Will, it depends on the AI. Anthropic used to have a lecture-length system prompt for their models to explain this stuff—part of the secret sauce for Claude Code—and famously found that the 5 series models no longer need it.
As usual, if you use anything but the best model available I’m going to state that the better ones do better. If you do use the best model available, then I’ll just mention that Fable still has limits and still needs some guidance.
One thing it does not do is deliberately build tests which test nothing at all, or which restate the code under test. I mention this because certain other models absolutely would.
>Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD!
Tests =/= TDD
> Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD!
Tests onli validate the presence of bugs, not their abscence (Djikstra).
I'll also add that tests look at outputs and don't care how those outputs are derived. E.g. code filtering the entire db in memory will be fine in tests.
Haven't you ever written tests that assert that certain things happen or do not happen along the way? We use mocks all the time to, say, make sure that the code doesn't load the entire db in memory. Then it's there as a regression if somebody down the line wants to do that exact thing for some reason.
"Beware of bugs in the above code; I have only proved it correct, not tried it." - Donald Knuth
Tests prove the things you thought of worked, and it often isn't hard to find the likely edge cases such that you have reasonable confidence everything works. You will be wrong from time to time, but not that often. You can prove code correct, but if the proof is wrong (common when a human is doing it), or the spec is wrong (most people have no clue how to write a comprehensive spec) it can still be wrong.
If the agent can write the tests its supposed to pass and we are worried that agent produces quite a lot of slop (which is why we are doing the tests), then what is the defense against test slop? Test the tests?
Manual testing.
I work on user facing applications, and since the models do not have good taste, testing the UX is essential.
If you spot a bug, usually the model will attempt to reproduce it in a new test case that does cover the actual issue.
Read the tests.
Doing a pass where you just ask the AI to sanity-check the existing tests (against rules like “test against the spec, not the implementation” can also help.
Existing tests.
In this particular case tests won't help you, without the actual data. But overall you're right.
"tests" here could mean have a benchmark the agent can run to determinately evaluate quality.
I don't strictly mean junit unit tests.
In this case, you need benchmarks, in addition to regular tests. You also need concise instructions (or better yet, following your language / ecosystem norms if applicable) to how to run this stuff
I'm convinced this is why there's a bifurcation in old-hand/new-hand results: the old hands include the steps they'd do and the tools they'd use, and the new-hands never learned those tools.
That'll get RL-d in, eventually.
The ability to run queries and get the metadata about the run, e.g. length of run, execution plan, engine, engine params, etc
That’s how I do it in my Laravel apps. I simply ask it to hypothesize, benchmark, apply and verify. Laravel has great tools for that.
Which model are you using? There is monumental difference between models. Even between "frontier models". When people tell these stories, it would be great to also add which model you were using.
As an example Opus 5.0 is in completely different class compared to Cursor Grok 4.5 even if the benchmarks don't show such massive difference. Not even talking about regular stuff like Sonnet or Composer or stuff like that.
Yeah. If you need something to dig deep, you need to try Fable (optionally in /goal mode).
For performance testing, I wrote isolated testbeds that try to impair the system in realistic ways (latency/jitter/bandwidth limit on logical WAN hops when load testing), and Fable is happy to send a bunch of agents at it and iterate until it gets the results it's looking for.
I think that if you are used to Sonnet medium or something, this will surprise you, but models like Fable and Sol on high/xhigh will really dig deep until they meet your goal. (I mostly use this for bug hunting and not perf, but ... I think it can do perf if you set it up right.)
It is very cheap to:
1. Ask Claude Code or coding agent to research the internet, documentation, and Github for examples and learning working with similar problems.
2. Make the ten best examples of solving the problem based on the research.
3. Run each against the database enough times (100, 1000, or 10,000,000 depending) to empirically prove which is optimized. You can set other criteria based on your knowledge.
4. ???
5. Profit
The point is, it is cheap to research combing through 1,000s of examples and learnings and test the best and most relevant.
The mistake there is to point it at code to figure out performance optimizations.
The place to find them would be performance profiles, query plans, telemetry. The guidance for perf still applies, measure before and after change.
The issue is that the code often does not contain the information to do a perf optimization. Eg. you can't tell your cache size, the volumes of data in your DB or the latency of your network through just the text. Should you provide this context, you can get better results.
> The place to find them would be performance profiles, query plans, telemetry.
Tbh, once this information is available (which is the tricky part), in 99% of cases there's really no AI needed to analyze the data, since the 'low hanging fruits' will usually stand out anyway. And once you get into the area where optimization hotspots are no longer obvious, you're already deep in the diminishing returns area and optimizations for one use case or hardware configuration may degrade performance on others. That's my experience anyway.
> there's really no AI needed once this information is available
Most people are not arguing that problems are too tricky for a human to solve once presented with it, but AI can look at 1000 things at the same time across a whole code base and then just come back with the results a human can review.
True, but at least IME, once you get beyond fixing the 'obvious bugs' (how many there are depends on the initial quality of the code base - but I agree that AI is really useful to find those), you'll get into a murky grey area of "maybe false positives" which require a lot of time to validate. As a result you sink a lot of time creating and tweaking code-base-specific rules to try filtering out false positives, and IMHO this is exactly the tipping point where the whole thing becomes pointless because it becomes a bureaucratic monster.
E.g. a good code analysis tool needs to work predictably at button press on any code base.
Still better than nothing of course, e.g. I actually think bug scanning / code analsysis is indeed the one area where LLMs are actually useful, but it suffers from the same 'diminishing returns' problem as traditional approaches, if not worse (e.g. still no silver bullet, but a mostly useful additional tool in the toolbox).
It really depends on the context. On a pure computational code, it can work really well.
I recently optimized some code asking Claude to "make it faster" 2 different programs. For each, it wrote a benchmark, gathered initial data. Emitted some hypothesis and measured data around them with profilers, then did some changes (behind feature flags), checked the output was identical in either side and through profiling that the right code path was taken, and then benchmarked both.
And it did that for multiple successive changes in both codebases. Some changes were purely algorithmic (better complexity), some were related to tradeoffs between memory and computation (caching intermediate results better), some were about creating less intermediate objects to relieve the memory pressure, some where about a better memory layout to improve data locality.
The annoying part is that some of the code that was optimized was generated by Claude in the first place, so it's a bit frustrating that it didn't do those in the first place. But I guess, my iterative approach to making those tools didn't work on big enough data sets at first where it would have been an issue.
You can very easily spot performance issues through code. Allocations are often visible, slower hash maps are often visible, loops are visible, etc.
No you can't.
Yup, I do this regularly now. Whenever I think that something takes longer than it should I just tell it to set up a testing harness to measure each part of the process.
Once I have the baseline I'll just set a goal to improve performance by 10x. Works better than it has any right to. Have to be careful to exclude changes that massively increase complexity for tiny gains after its done, but the big improvements are often sensible choices that I'd also make as an engineer (i.e. more efficient data representation, caching and memoization where it matters, parallel processing, etc.)
A self validating loop goes a long way with getting AI to optimize stuff. We had some builds that used to take a long time (15-20~ minutes), that i just asked Claude to reduce the build time. It brought it down to 2 minutes and under 10 seconds for incremental builds.
Turns out folks had been copy pasting some setup code which took 2 minutes to run, and only needed to be run once per environment. Claude refactored the code to get it to run once, and since we had tests which validated the build artifacts, it was able to roll back if its change had broke the build.
I've consistently found that I see performance issues that the AI misses. It often says "that's not going to be what improves performance, it's noise" and then I get it to do it and it's like a global 30% throughput win lol. I think a lot of performance guidance it'll be trained on is shit - I see devs consistently misunderstand performance too and downplay the impact of anything other than "IO".
Just give them a profiler. They drill down just like a human would and test stuff and validate. It works great.
And without a profiler they do about as bad as humans: spending a lot of time optimizing code that is rarely used (and then often with small n)
Which model/agent/harness tool did you use? I've found what you describe was my exact experience some ~6-8 months ago, but since about a month or so the game has completely changed. Using 5.6 Sol with highest reasoning setting in Codex or Fable in Code, the models come up with a list of possible improvements from static analysis (ranked by complexity/benefit), write and run their own custom profilers and deliver significant performance improvements with barely any input needed from my side. So this is no longer a model issue, it's a user toolchain issue.
Very hard to say anything definitive on this because it's a moving target, but last time I tried models still had a distinct sense of "consistently good, sometimes great at micro, bad at macro". Similar to how, even for relatively pedestrian CRUD, they'll do code that's objectively fine at the function/file/class level but can still make a mess if you don't supervise them at least at a high-level.
You're using it wrong. No really you are.
Give Fabe 5 access to a test database with some data, or even restricted access to your live DB and tell it to optimize then.
I've had stunning success optimizing for performance this way. In a single day, I made the core part of our app 2-3x faster.
Thats fine but then Fable 5 should have requested this information instead of blundering along. So why didn't it? An expert human asked to do the same task would have surely asked for the additional data.
That's what I add at the end of my prompts: "Ask me questions on anything that's not clear related to this task" or similar. Both the CLI and the VSCode plugin have a nice interface designed for this, asking the user questions.
No one said Fable 5 is human.
That's like saying Fable 5 should have told me how to prompt it to solve the Riemann hypothesis like Terence Tao would. The skill of the user still matters.
Sure, but it didn't. And you get way better results if you do that, so do that.
I've had a completely different experience. I usually know the problematic part and then ask the AI to optimize that exact part with benchmarks, telling it to try various different approaches, and it very often gains massive performance increases on various tasks, with me of course steering it and giving it pointers every now and then.
I was able to optimize a physical water simulation that was almost unrunnable on browsers to a buttery smooth 60fps version.
You have to learn to use your tools, not try whatever intuitively made sense to you at first (expecting the tool to do all the work) and then whining on Hacker News when it doesn’t work out for you.
We're reliving the early day Google era. It's an objectively simple tool to use, but some people just refuse to put any effort to learn.
I'm thinking the issue is probably that LLMs/harnesses are too easy to use? It crossed the thin line between magic and tooling and blur the mental model. If Claude Code were as hard to use as, say, ComfyUI, perhaps there would be less programmers having absurd expectation of it.
Also people seeing someone using it very efficiently to do some sliver of work then trying to project that it make all world that much easier
the whole selling point of these tools is they do the work for you
I’m terribly sorry on their behalf. I hope the expression of their experience has not hurt Claude’s feelings (IPO valuation). Won’t happen again.
This part is obviously their fault and they need to use their tools better:
> others dumping their raw AI output at me
Sheesh.
AI can not fail. It can only be failed. AI can not have limitations. Only the user can be limited.
Are you still speaking about a tool or starting a religion?
Seems to me they’re doing the third option: being sarcastic.
When I listen to some people or to stuff that comes out of the frontier labs, they indeed do sometimes sound like a religious cult talking about deity. I find it disturbing.
My favorite thing lately is that it's very good at resolving dependency security vulnerabilities. GitHub already does a good job at detecting and notifying which packages are vulnerable and at what versions they're fixed, and without AI, it's just a super tedious process to figure out what things all depend on that package, find a version that'll work for all of them (potentially including upgrading the things that depend on it), wire it up, test things, and make any minor changes necessary in our code.
It's not "hard" or novel--it's just tedious grunt work. I'm happy to have AI solving that one.
Echoing the other folks, I have a different experience.
I profile sql performance and LLMs find more opportunities than I could. All it takes is real data, a sql repl and an agent. Just ask the agent to use the repl to EXPLAIN and profile the sql. It works amazingly most of the time.
I did a lot of SQL optimisation with claude code over the last year, what made it work for me was making sure I verify everything with explain analyze. The whole loop was not fully “agentic” because of this but it was still faster and better than when done entirely manually. Claude also suggested plenty of small optimisations I would otherwise not do.
This is the research taste part - often they can be good but human experts are really good at this (also you know your codebase, without the prompt these models have no other background).
Then being able to suggest several things to try and have them go off and build, measure and tweak is hugely useful in my experience.
Also things like making custom visualisations for comparing changes.
People in the comments sound like they're literally asking claude: "hey claude, can you find areas for performance improvements." Nah, it doesn't work like that. You have to be more like: "can you profile [...] and identify hot spots." then looks at results that are suspicious. if you don't ground it to the real world reality of your project then don't be surprised when it makes up its own.
that's the first mistake i made as a new claude user. i only knew what prompts people were sharing on forums. but since they had been written by vibe coders it was all complete non-sense. also think about this: software engineering is filled with highly specialised tooling that can easily improve software quality. the thing is, most developers dont know how to use them, and if they do -- using them is so slow that its almost not worth it.
any tool you can name claude can use. obscure academic proof verifiers, memory corruption scanners, debuggers, profilers, coverage tools, backdoor supply chain pattern scanners... it can run these and have results in seconds. really is like living in the future.
CTEs are often slower than using temp tables with indexes.
Ask it to create a small sample to reproduce the issue in isolation. If it can't reproduce the performance changes then it drops it.
That might work better. I haven't tried it with SQL but it worked with a few complex UI issues I had. It identified the actual issue after a few false starts.
>using AI
What model? There is no "AI", there are only specific models.
...and harnesses, routers &c
Which model/mode?
AI can uncover certain issues and resolve certain problems but it also has its blind spots.
I've been using Claude to create a software architecture diagram. It came up with a lot of useful functionality that we had neglected to show but upon further examination a lot of steps didn't make any sense. It added a box to do validation on some data then it put another box downstream to do validation again on the exact same piece of data (the second box had a different name but upon questioning, Claude conceded that it was the exact same validation step). When doing inference on user input, it put a box to extract specific insights from the prompt and then it put another box to do inference again on the inferred data to categorize it... And this would add latency and unnecessarily strip out useful context when doing the second inference... It really only needed a single inference step...
I re-uploaded the diagram to Claude and confronted it and it conceded to all of my points and it even noticed a theme between them and suggested that the diagram exhibited "a pattern of redundant steps." But it was incapable of suggesting viable solutions besides name changes. All of its solutions made the design more complicated. It was incapable of simplifying the design. I've noticed this with AI code; it can make things more complicated, add more features but struggles to simplify things.
Also, AI is horrible at rating things... It's just too superficial. For example, if you write a huge amount of intentionally over-engineered, tightly coupled code with poor separation of concerns but you do proper linting, define all the types, add a lot of comments, it will give it a higher score than a shorter, more readable and maintainable snippet which exhibits loose coupling and high cohesion because it lacks the superficial aspects.
It's horrible because now business people who understand nothing about coding may ask AI about code quality and it will consistently rank low quality over-engineered projects that are full of bugs, unmaintable and less secure, with a higher score.
Frustratingly, if you point out problems in its judgement, it will concede to your points without reservation, even building on top of your argument... but it will never actually tell you this stuff up-front, unprompted if you don't already know it! It never seems to reveal new insights, at best it can only expand on existing insight which you've already had.
It tells you what you want to know but not what you need to know.
Then you're not grounding it to reality properly. LLMs get bad pretty quickly when you just tell them to pull the answer out of thin air. Pin them to reality with actual performance tests to run to test theories and the results will be much better.
Performance optimization involves simulating real world loads and measurable results. Give them the opportunity to actually have a closed loop if you want more than trivial improvements.
Have you any example materials that show a harness that an agent can be pointed at an app with some sort of telemetry tool, the results it gained and the cost of doing so? Because my experience is the same as the parents - the LLM goes on massive tangents, and the more tangents it goes on the worse the results get.
Tell the coding agent to install or build its own performance and profiling tools.
A few lines of Python with a time.time() call is enough to start iterating on performance improvements, and the really good models know how to use much more advanced existing tools.
Not sure what materials your hoping for, but I've pointed codex at pprof application endpoints and just told it to identify hotspots and propose lowest hanging fruit/highest ROI items to fix, which I then approved it to open PRs for. Reduced application CPU usage by ~30% on loaded servers.
It's not magic. Fairly obvious oversights that I or someone else on the team could've found and fixed if we had looked, but it was never a business priority so we never did until I curiously spent 5 minutes asking it to do it for me.
Ideally a blog post on an open source project that shows harness, model, prompt and the before/after measurements plus the PR.
Ultimately the harness is me and the experience is like managing an unruly toddler. You have to pay attention to what it does and issue corrections. Skills and prompts and AGENTS.md do some, nested sets of agents do some, but over it all is me keeping track of what it's doing and needs to do.
You have to size the unit of work to its useful attention span, you have to have code architecture that is conducive to units of work, and you have to have tools like ticket managers, etc. that keep the big picture and smaller pictures in mind. Reverting from agile practices to more pre-planning whole project documentation and architecture decisions helps.
But in the end it's you. LLMs have their limits and need humans to direct them.
Gotcha, so no.
Were you using Fable 5?
Did it ever tell you not to optimize what you should redesign?
Did you give it access to a planner/analyzer?
I had good results when I started AI with building the testing for the actual problem, then it can just run itself over and over without wasting much time on verifying its output.
And at worst you still saved a bunch of time on writing the tooling to test the issue
What's the complain? That AI can't do your job yet?
I've noticed the same thing.
The trend I've noticed is that AI struggles to think outside the box when making optimisations, which exactly what's needed when you've made all of the practical DB and logic optimisations to the existing code.
Often you need to take a step back and question how the system is working and if there would be better ways to design it so the bottlenecks you're hitting wouldn't exist in the first place. Caching things, adding indexes, tweaking logic – these can help, but you'll quickly hit diminishing returns once you've done all of the obvious stuff.
I've seen people here say how AI is great at optimising code though, but I'm not sure if that's because they're giving it optimisation problems with a lot of low hanging fruit or if they're successfully getting AI to rework their systems to remove bottlenecks. This one area I find AI to still be particularly bad at.
As others have said, these were uninformed guesses which LLMs seem still to be willing to hand out deliberately. If you provide more information (your schemas, the distribution of values in your database tables, the EXPLAIN ANALYZE outputs, DB configuration etc.) it will do a much better job.
Why are you sidetracked instead of piping that to another AI?
I would never undertake a big performance optimization without first measuring that area's total impact. The LLM would then be great to work out the refactoring and you can give it your bench suite (which it probably wrote) to work against.
I guess just another area where the LLM is useful only as long as you remain in charge using your own programming experience as a guide.
Yep, as a performance specialist I've had the same outcomes - it'll find "smoking guns" galore which will sound plausible and be completely wrong.
But that's ok - I still get a lot of value of the tool elsewhere (e.g. writing data analysis scripts, exploring code, breaking down bug artifacts...), but it doesn't stop the tiring BS from other people going "why not just AI it?" (especially from managers).
I usually let them try to "just AI it" and most of them learn quickly that it's not that simple. (The others stay deranged and are making my job miserable.)
> I've recently been using AI
That's uninteresting as long as you don't specify the model you used. For example, Mythos was far better at finding security bugs than previous models.
Do you really think they mean they’ve been using mythos
In my mind there are three tiers:
The SOTA: Fable, GPT 5.6 Sol, Opus 5
The "enterprise admin did not turn on the new models": Opus 4.8, GPT 5.5
The "I love hallucinated garbage": Sonnet, Qwen 3.6, GPT 5.4 mini, GPT 5.3 Codex, etc.
Results vary widely
What did your tiers consist of when GPT 5.3 was the latest?
Obviously no. There is also a significant capability difference e.g. between Opus 4.8 and Fable.
The real datapoint was Firefox not paying any money in Berlin's Pwn2Own competition round this May just gone. Unheard of to have nothing confirmed... they've paid out every event since 2007 (I checked). Does this mean we must move past the low-hanging fruit now? Probably... Certainly indicates some usefulness of these models.
Not that I don't believe its possible to fix a lot of bugs, I also wonder what the actual dynamic was. Were the people in team working much more than usual as well? Given its Google, I wouldn't be surprised if there was an "internal push" to fix more bugs over next X sprints so that they can publish this blog and some manager can show impact and AI adaption to his superior.
Google's entire modus operandi has been "automate everything" for decades. They've been doing this with fuzzers, with project zero and so on. Adding LLMs on top is a very obvious next step. And LLMs improving and finding more bugs also follows. Then improve the harness and the dev tools, to better use the LLMs. And then everything together end-to-end to find-triage-fix-confirm. Your LLMs are as good as the loop they run in, and the loop is as good as the verifier. Seems a reasonable enough dynamic without (or despite, depends how cynical you wanna be) the need for managers to show number goes up on some chart.
I would guess they have been confronted with an initial flood of newly discovered bugs whenever they added a new analysis tool or approach (e.g. automated static analysis, fuzzing, ...). Once working through that initial flurry of newly discovered bugs one would assume that the frequency goes down again.
> In early 2026, we saw a gradual increase in all categories of bug reports, but by March, the shift was apparent: we received more bug reports than we had in the entirety of 2025
sure, moreover - maybe big AI usage significantly influenced the amount of bugs. So, the picture can be like that: - 2025: 50bugs found, 45fixed - 2026: 500bugs found, 450 fixed
This is just my hypothesis, but I suspect that at Chrome‘s engineering team there has been a culture of inaction for the past decade or more. That nothing gets done, no bugs fixed, unless some higher ups at Google sees a business interest in resolving it.
And what has changed now is that the higher ups at Google do indeed see a business interest in fixing bugs and giving the credit to AI to sell us more AI.
1. Our backlog of bugs gets processed quicker because instead of staring at the code for 10 minutes fiuring out what's happening, there's a tool that can reason about it quicker.
2. Code reviews and security reviews happen quicker and produce more findings.
I would think that (m)any team(s) using AI might also be seeing a higher rate of finding and fixing issues.
Even the Linux Kernel (I'd say Windows and Apple too) are seeing the same phenomenon.
Linus: "it keeps finding embarrassing bugs"
Linux Kernel: https://lore.kernel.org/all/CAHk-=wi4zC+Ze8e+p3tMv8TtG_80Kzs...
The idea that software has gotten so complex that a machine can evaluate code paths better than a human, seems to bristle the fur of many. Some people didn't think we would see the day where that comparative human limitation was laid bare in simpler tasks than they expected. I believe older developers are less likely to be offended, having to deal with this as a matter of course (as the mind declines).
I don't know, this more and more feels like a sentiment projected on to people than anything real these days.. Like even in that linked thread, is personal offense like you lay out here really were you can place Laurent Pinchart's push back? You don't read anything else there at all?
And either way, what, we are going to keep this line going for another 5 years? Aren't you bored?
More likely just getting ahead of the AI attacks before they hit. The threat risk increase caused by AI has gone off the chart.
Exactly this. And there are few bigger targets than Chrome when it comes to finding exploits (OSes and network equipment are probably on par). I'm sure they have devoted large compute resources and human staffing at making sure that they find and fix these issues before anyone else does.
Nah they just write blog posts when they feel like it /s
A lot of people here seem to be living in a different universe than me or simply don't know how to work with AI. I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you. They get mad at Excel for the poor investment returns. At this point, this is such a strawman, it isn't worth counter arguing.
I think I'll abandon this discussion and keep using AI quietly while exchanging tips with like-minded people who are interested in using it properly and efficiently.
> A lot of people here seem to be living in a different universe than me
I feel this way on this topic too.
> I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you.
The problem is; how _should_ I use AI? On a previous thread, I had two replies to the same comment, one saying "provide the LLM all the context it needs and let it go ham", and the other saying "Carefully guide and craft it and review everything". Both subthreads had people agreeing.
My experience with LLMs is leaving them unattended gives very poor results within a handful of iterations, and that carefully guiding them can be a value add, but the effort ot do that is very often as much as just writing the damn code myself, particularly if there's a few iterations that go on.
I think the main cause of different views is just the fact that people have very different products and codebases they're working on. Therefore, the requirements and applicability of AI varies a lot, case-by-case. It would be useful if people always explained their exact use cases when making claims one way or another. I understand it's not feasible for many use cases to reveal exact details, but at least some transparency would be great.
I've found AI to be immensely useful in gamedev, mainly UX development, research, and curation. In those areas I let it go quite wild, with mechanical and visual tests for UX, and summaries for research and curation. I've also made skills, custom subagents, hooks, and scripts (all made with AI assistance) to streamline development.
Gamedev is a good example of where AI shines, since the risks of fucking things up are not so big. If I worked on, let's say banking, I'd be much more careful. On my day job, as a data scientist (moving towards product owner) in a large industrial company, I'm much more careful. UX can run on vibes, but anything that touches data processing or business logic needs to be looked at by a human, row by row. Depending on use cases, it may be fine to look at only the tests. The main problem is that there's so much stuff to review, and maybe 98% of it is fine. It's then very boring to try to spot the 2%. I don't have good solutions for that yet, and idk if anyone does.
> It would be useful if people always explained their exact use cases when making claims one way or another. I understand it's not feasible for many use cases to reveal exact details,
It would be useful if people _ever_ explained the use cases. Even Anthropic when they're publishing their materials are hand wavey about this.
> Gamedev is a good example of where AI shines, since the risks of fucking things up are not so big
I couldn't disagree more. I run an engineering team in GameDev, and AI is great at making a gigantic mess. It's great for generating a lot of code, but the minute that doesn't do what you want it to, you now need to go back and learn _everything_ that it's generated. And IME, doing that is as bad as having just written it yourself in the first place.
> he main problem is that there's so much stuff to review, and maybe 98% of it is fine. It's then very boring to try to spot the 2%. I don't have good solutions for that yet, and idk if anyone does.
I tentatively agree here, except I've found about 50% of it is "fine", 40% of it is "this works but isn't really what I want to put into my product" and 10% is "absolutely not".
AI is great at reading, so give it things to read. Claude will try to throw big markdown docs in your repo, but you don't have to do that. Make it keep docs by the code.
I include a line in AGENTS.md that says "We *always* add docstrings to methods, classes, structs, and namespaces - there should be 100% coverage of docstrings."
It will make better choices about what functions to make or remove if you force it to justify why the function exists.
And when you have to go back and understand, it becomes easier.
> It would be useful if people _ever_ explained the use cases. Even Anthropic when they're publishing their materials are hand wavey about this.
Yes, good addition.
> I couldn't disagree more. I run an engineering team in GameDev, and AI is great at making a gigantic mess. It's great for generating a lot of code, but the minute that doesn't do what you want it to, you now need to go back and learn _everything_ that it's generated. And IME, doing that is as bad as having just written it yourself in the first place.
Yeah, I should have been more specific since our use cases are very different. I'm working solo on small games, so it's easier to keep the AI on a leash. I can imagine it can be a nightmare on large code bases with many collaborators.
> I tentatively agree here, except I've found about 50% of it is "fine", 40% of it is "this works but isn't really what I want to put into my product" and 10% is "absolutely not".
Yeah, probably our use cases are very different here too. We mostly develop/maintain dozens of microservices for internal use, mostly greenfield, relatively small in scope, and with 1-3 developers in each. And maybe 98% was a bit too optimistic. The larger and older the codebase, the more problems AI tends to create. But it's very good at helping to understand old codebases (even those written by myself haha).
Oh, and esoteric data science topics can be a minefield too. If there's a topic that's new to me, It's difficult to know what is hallucination and what is not. It's best to let AI suggest alternative methods and then study those yourself.
> Gamedev is a good example of where AI shines, since the risks of fucking things up are not so big
Hi, professional game dev here. No it's not. Thank you for reading.
It’s a very soft skill that changes as the capabilities of AI change.
But one thing that doesn’t change is the need to specify an end goal correctly and precisely. I think the emphasis of knowledge/information-processing work is going to increasingly be placed on verification mechanisms. This is practically equivalent to precisely defining an end goal.
Spend time deeply thinking about what it means for a solution to be correct. What properties will a correct solution have? What of those properties are testable? Write those things down and tell the agent.
As models get better, agents will be able to target more and more difficult end goals. The strategy just becomes more useful. (It’s useful for people as well.)
For example, if you want an agent to write a photo editor, think about what end properties the editor should have. There are reference images for color space and rendering transformations. That’s a good start.
Sometimes the goal will be fuzzier. “I want a feature like CaptureOne where I provide a reference image and it makes my image look like that.” Well, time to think really hard about what that means. Iterate with AI on how to test for that precisely. Come up with some good metrics/heuristics. Maybe it means local contrast should match. Maybe the overall distribution of colors. Maybe something more complicated.
Then you have a target and you can let an implementation agent work against that. If it fails, it’s either because the agent is bad or your target was incorrect or incomplete. As models get better, the limiting factor becomes your ability to correctly define a problem.
Latest models like Opus 5.0 work well in both scenarios. You can just give it a goal and let it make a plan, review the plan and then let it go ham.
Or you can also use it piece-meal and have it do parts of work that you want. It works in both ways and works really well.
I don't think there is any model right now that you can use with 0 oversight, but you can do pretty complex stuff without writing single line of code at this point.
The truth is that using AI well is a skill much like anything else. There's no one single way to use it much like there's no one single way to write a program that serves hackernews.
The reality of the current situation is that we have dozens of families of different model; each of which is differentiated further into various revision numbers and power levels, and depending on which one you use and what the nature of the tasks is, you very much need a different usage style that can only be discerned after multiple (Sometimes very many) sessions of using that individual model for that category of task. It is not one size fits all, and trying to pretend that it is will lead to failure.
> It is not one size fits all, and trying to pretend that it is will lead to failure.
I'm not asking for a one size fits all solution. I'm asking for "vibe coding with Codex on GPT 5.6 with high is the way, but if you want to be more ivnolved, Sonnet 4.6 on Claude is the path. Here's the proof."
Saying that it depends and it's impossible to quantify allows the camp who are claiming it's more productive to say "you're holding it wrong".
What can be concluded is that some get immense value out of Ai and some are incredibly critical of that claim.
I do not know what the secret sauce is.
But I have a responsibility to the team I lead. I need to protect our velocity. Which means that my hiring practices now include specific checks that people are able to work with ai.
In the end, it might be a step thing. Some people are just inherently incapable of working with this technology...
I don't generally use AI to write code I feel I can write efficiently. In these cases, I'll just ask AI for a review. But when I need to write code in areas I'm less confident about, I'll let AI take the reins and the do a thorough review myself. These instances are not too common, though.
I also ask direct questions to AI for brainstorming and advanced programming language usage. In these cases, AI isn't touching the code.
There is an argument that there is a lot propaganda in this space - because the issue of AI is now an issue of international security and politics.
I see a lot of the disagreement here has the same form as many of the political arguments from 10 years ago.
This (fixing bugs, making code better, refactoring), is the best fit for AI.
I think a lot of us were excited that we will finally be able to polish our old software.
Many of us were never given the opportunity and were tasked to produce new features at an ever increasing pace. Depending on where you landed you can be either elated or jaded.
I agree with you.
> I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you.
However this is how it’s marketed. C-suite is telling people eventually you won’t need to read code, frontier labs saying programmers won’t exist etc.
The reality is like you said, a tool to accelerate you.
You can today develop a mobile app with 20 screens without ever reviewing any of the code generated.
If you test each feature and have it iterate on your feedback, you can build a decent product in this way.
Does it write too much code? Perhaps. Could I bring those 100k lines through code review in a team where some members nitpick? No.
That does not change the fact that it works, and that you do not need to read the code even today.
Unless progress slows down dramatically and soon, it seems rather likely that most of us will not be reading code in the near future. Their marketing is not wrong.
Just exploit the edge while it lasts lol
Reminds me those Bitcoin threads where I would explain a use case it solved for me and everyone was like "nope that's impossible, there's no use case". And before the inevitable replies come in: no, I'm not saying AI is like Bitcoin.
> They get mad at Excel for the poor investment returns
This is quite some mental gymnastics. Is your presumption really that detractors = stupid people who can't differentiate between the challenges in getting Excel to do some specific thing and blindly interpreting an incorrect calculation as an actual accounting assessment of their business?
> I think I'll abandon this discussion and keep using AI
Not joining in the first place to provide such nuggets of wisdom was also always an option
"I'm so much better than you at this but don't ask me about it. Just trust me and feel bad yourself"
Did you think for a second that maybe other people work with other languages, domains, tools and work approaches than you?
It's so bizarre that noone realizes that there's a bit of a difference between banging together JavaScript into websites, writing firmware for widgets, herding servers or fixing medical devices?
This right here is the truth. Too often people talk in abstract ways, not disclosing what they're actually working on. And then they tell their experiences as universal facts.
The reason they don’t disclose what they’re working on is because it’s always some CRUD app
The goal of software development is to eventually be done. Yes there are new incentives or new use cases that come up, but the aim should be towards a stable state where you're barely have to work on the software anymore. It's like an house where you spend a good chunk of money to build it, but afterwards you just maintain it.
That's my main issue with AI hypers. There's often no goal in sight, they're just busy for the sake of being busy. They talk about their process, but not their objectives. It's not about progress for them, it's only about being in motion.
Did they fix the bug where Chrome tries to track you and your behavior wherever you go and whatever you do in the world?
How many of those automated fixes were reverted? How many introduced a new bug? What's the false positive rate on the finding agents? The post has counts for everything that went right and nothing for what could go wrong.
Exactly, also it lacks a lot of context as to why they did that.
Possible (probable?) scenario:
- Marketing: "we found and fixed lots of bugs thanks to AI"
- Reality: the KPI is now to fix as many bugs as possible with the help of AI, so they used AI to search old and easy bugs in the backlog, and then fixed it manually
Huh? Maybe a third scenario is that AI helped fix the bugs? Like described in the actual post you are replying to? This level of conspiracy theory is getting a bit ridiculous.
I think you’re right, my base assumption is that the models can code and can fix bugs, and can code more in parallel and faster than humans at a lower cost.
If Google are tackling lower value bugs with AI the number is in a way inflated compared to some utility measure (fixing a smaller number of worse bugs could be preferable) but it’s still things fixed.
They also don’t really provide an explanation for the big uptick in bugs found M146+ in the first place.
Is that better testing? Or more bugs were being introduced in the first place?
> The post has counts for everything that went right and nothing for what could go wrong.
That's AI for you.
At Amazon we have many forums to share our AI wins, but none to share AI failures or disappoinments.
No wonder execs make bad decisions regarding AI, they only hear completely one-sided stories.
At my company there's a lot of discussion about AI and complaints that people run out of tokens within a day, but zero results are shown. No measurable (or measured) gains. Or nothing that people are willing to talk about, in any case.
At my company the performance improvements channels has been exploding, with people claiming giant improvements in latency, throughput, and decreased cost of the services. The cost decreases itself is order of magnitude more (at annualized run rate) more than we pay for tokens. YMMV.
I hope to see more of this across the industry.
I definitely make use of AI but in my experience I almost always could have done it better myself, the places where I threw AI at the problem I didn't care about the results being good, only good enough.
When we see memory and compute requirements for version x+1 of software decrease instead of increase I will happily say AI is the oracle people proclaim it to be.
In $COMPANY, for the mid-yearly review, the employees were asked whether their AI usage was 1/ efficient, 2/ adoptive (integrated in the way they work) or 3/ transformative.
Saying “never used it” or “I tried and it was useless” was literally not possible.
"It can't be that stupid—you must be prompting it wrong." - Ed Zitron
> How many introduced a new bug?
I'd say that one is not really an issue. In 2012 the Pinkie Pie exploit chain already required chaining 6 bugs to lead to an exploit [1]. Since then we've seen chains requiring more than 10 bugs (!).
If you fix any one of those bugs, the exploit is non-functional anymore. Sorry out of luck.
So if, say, for every ten bugs you fix, you introduce two new ones then it's still a very net win. Unless of course it introduces a bug so bad it becomes a simple exploit not requiring a long chain of exploits.
But in the case of browsers we've only ever been moving to longer and longer chains of exploits required to pwn a browser.
A great many window of opportunities are closing for dark-side hackers / north korean intelligence etc.: there were probably exploit chains still open for exploitation in April that just got closed by Google.
If anything, besides the supply chains attacks in amateur-land, the world didn't stop working: projects (not just browsers but OSes too) are being hardened left and right.
Using AI to find potential bugs is an amazing use case and there really aren't many downsides.
> The post has counts for everything that went right and nothing for what could go wrong.
I'm not saying there aren't a few downsides but the benefits are just too good to ignore.
[1] https://blog.chromium.org/2012/05/tale-of-two-pwnies-part-1....
And how many were introduced by AI?
It's the opposite: humans. The most critical one (sandbox escape) has been sitting there for 13 years.
The question is how many new bugs were introduced through those AI fixes.
Is it less than a human would have? Is it more? Is it the same type of bugs?
I could be convinced either way and it's an interesting thing to ponder in my opinion.
I assumed it was really just AI that found them, which allowed Google to know to fix them.
You're not supposed to ask sane questions here
Or people don't ask questions for which the answer is known to be "None, really."
I get that many don't like what LLMs are doing to the industry, but this is just incorrect reaction to a very specific benefit that's proven beyond doubt (Security hardening).
Accept it imo - LLMs are solving very large problems that have plagued software security.
Sorry it is your reaction that is really weird. These are legitimate questions, and I have definitely seen Claude finding the wrong cause and then implementing completely incorrect/irrelevant fixes, only to find that it didn't work and need to start over. Not saying humans don't do the same thing, but LLMs are far from perfect, and it would be delusional to only talk about successes.
I've done security work before, and I've done it now with frontier. I've seen the difference first hand, and as the OP and so many other articles show, so have the leading experts in the world.
So either they are lying, or you may not yet be seeing and experiencing what they are. If that's wierd, ok.
You did not address my points or those in OP but kept repeating yours (that are not even relevant) in a handwavy way.
And that's fine. You can continue to live in your bubble. Others simply have different experience, and your asserting "AI is perfect" on a forum is not going to matter in terms of everyone's own, first hand experience.
This is a tacit admission it was Mythos, not Gemini, despite alluding to experiments with Gemini earlier on:
> Added support for model interoperability to leverage the unique strengths of both open-weights and proprietary models
Worrying. Extrapolating a (speculative) future, this means (Google will feel) that soon the chromium base will no longer need the crowd-sourced bug hunting that is open-source. I expect Google to eventually stop working on chromium (in the open) and all current chromium-flavours will become de-facto forks of the last published version of chromium. These forks won't be equally easy to maintain given that the groups running them do not have access to the same level of subsidy as Chrome does with Gemini.
Google already controls Chrome/Chromium direction so much that you should use Firefox if you value an open web.
Which I do (actually a fork of firefox - waterfox), but it is still worrying as the viability of chromium-base browsers is what's keeping us having "merely" a chromium-monoculture (with viable ad blockers e.g. and website operators that check compatibility with more than one browser), rather than a chrome-monoculture (where ad-blockers surely would have been killed by now).
AI critique often funnels itself into a narrow bucket: creating code blindly with AI is bad. That's easy to grant.
Adversarial testing, checking developer assumptions, refactor suggestions, small dev tools and even some guided coding all sit on the other side of the spectrum of what you can do with coding and AI. For larger and larger codebases even simple things like tracing dependencies or behavior might be greatly aided.
And the critiques reserved for that narrow bucket on the other end, blindly generating code, are too easily conflated with the rest.
Yes. AI is a tool, it’s supposed to be used a certain way, anything else is a misunderstanding og what AI is. You have to aim it in the direction you want it to go, not expect it to solve all your problems magically
AI isn’t “supposed to be” anything (other than “intelligent”). It just turned out that LLMs have to be used in particular ways to be useful. Nobody planned or designed them to be that way. We all as users are put to do the job of discovering the ways.
LLMs are increasingly not tools anymore but agents. The difference is still one of degree, but it's clear in which direction we are currently moving. It was even clear to some people 10 years ago: https://gwern.net/tool-ai
AI has skills no individual can have. That part is true.
But AI is not smarter than you, AI is as smart as the user who uses it. It often makes wrong decisions unless user corrects it.
Elephant in the room : how many of these bugs were written by LLMs in the first place ?
Because creating 100x more bugs and fixing 100x more isn't something to be proud of.
Can't have been much, as the Chrome project is 20+ years old and LLMs have only recently become a thing; they did not slacken their code review and testing practices with the advent of LLM code generators. And I doubt they do much development on the affected areas at the moment, they mention a 13 year old issue for example.
But it's an open source project, you can go and figure out whether your assertion is correct.
The single 13yo issue is anecdata.
They obviously have the full git blame statistics but chose not to include them in the blog post, which is a bit concerning.
If people are handing off the coding to AI more and more then they'll be less skilled at spotting potential bugs in the first place. AI, course of, could be run to find potential bugs in the AI-coded features before commit. But then you're getting further and further towards a world where code, and code that runs infrastructure, is incomprehensible. We'll have to start sacrificing the first-born to the silicon in the hopes of a good build for code that controls your electrical sub-station.
Look at the git stats for the project. Lines of code submitted haven't changed dramatically. Not everyone is just merging slop into the codebase. There are many ways to use AI, and most established players are not merging vibe coded nonsense.
If you assume that AI can produce code with less bugs, your 100 times increase in fresh bugs means you also increased your speed for new functionality by more than 100 times.
If models can spot 13 years old critical bugs, I think they can also produce fresh code without those bugs. The skillset is the same.
That is a stat you invented out of thin air, based upon nothing but pejorative hope, against all logic or rational consideration of the timeline or the scale of the project, and you declare it the "elephant in the room"?
HN is so weird on the topic of AI, and so many desperately are trying to contrive a reality. Odd stuff.
this is really not reddit
It's not an invention it's a question. If the number is <20% that's great news for the Chrome team.
The elephant in the room is how many people in this discussion are actually alien lifeforms.
The elephant in the room is how many turtles obtained law degrees.
The elephant in the room is whether this is worth discussing because we're all in a simulation.
No, it isn't the elephant in the room. You made up some horseshit that betrays how little you know about the field, which will get upvoted because HN has a hearty contingent of "if I pretend AI is actually useless, maybe that will manifest in reality" zealots. It's bizarre.
Turtles with law degrees don't exist, but LLMs that increase tech debt in code bases do. Maybe recent ones got better, and it's useful to quantify the progress being made.
This situation seemed familiar... https://i.ibb.co/RGvQgfmX/i-fixed-more-bugs.png
Title is inaccurate. Fixed more “security bugs” not “bugs”. The article is specifically about security vulnerabilities.
with excellent test cases, AI can do wonders, human's task in AI world is to just write as many tests cases as we can.
I can believe it. So many small CSS quirks that have bothered me for years have suddenly been fixed, meaning I can remove my work-arounds.
Like what?
It's so hard to believe these kind of affirmations coming from a firm that has so many hundreds (thousands ?) of billions invested directly and indirectly in generative AI...
Just admit that Chromium is too complex at this point, we've all seen show HNs doing what could be considered magic in Chromium. We probably need to issue a mortarium in new platform features and focus on a deep clean in bug fixes. I've said that Mozilla is too busy doing redesigns than bug fixes as well.
This is only a flex if AI also didn’t cause an increase in bugs that needed to be fixed.
So, your conjecture is that the LLM's skill to detect bugs somehow magically disappears the moment they start writing new code?
No, the extra code velocity allowed by AI without proper QA support means there could have been a lot more bugs introduced into products over the last year or so, giving AI more bugs to find. It’s tongue in cheek but not completely uncalled for. It’s been long known that one dishonest way to inflate your bug fix count is to put more bugs into the product to fix. Goodhart’s law is merciless.
I wouldn't say "magically", it's what happens with me and every other engineer I know: we're all detecting bugs and yet we're all writing them too. Not saying that LLMs work the same as humans, but saying it's not a logical fallacy.
Actually yeah, that would align with my experience with it.
You're absolutely right! Let me fix that.
The privatization and obfuscation of the security industry is worrying.
Google claims to help open source developers, but donates millions to the Alpha Omega Foundation that did not give access to Mythos to Curl and only found one issue.
The Alpha Omega people appear to be selected on physical appearance and take the money away from open source without doing much.
The entire Chrome article is obviously directed at suits and unsurprisingly hypes AI for better promotion chances.
Convenient that the LLM vendor is the one saying their own product works, isn’t it?
Company advertises their products. In other news, water is wet
That's why you should be critical of their press releases.
Nice marketing Google, now tell us how many of the bugs that were fixed were actually introduced by AI usage to begin with?
No worries, writing with AIs can improve the speed bugs enter the code base as well. Would be interesting to know the breakdown.
> the “latent security issue.” Code that is safe and robust in isolation can be transformed into a critical vulnerability by an entirely unrelated, minor logic change elsewhere in the tree.
Similarly, you can accidentally fix a bug without discovering/triaging it, so the pointless repeated bug lifecycle is incomplete
> Google fixed more Chrome bugs in June than over the past two years, thanks to AI
And introduced how many new bugs? thanks to AI ...
At this time, a generic AI snark is tiring. AI is not perfect, but it's an amazingly useful tool, we should accept it and move on.
AI hype is also tiring.
I have a software degree and some work experience but have been doing other things for a decade or so. I'm curious to try out some "vibe engineering" but it seems a bit daunting to get started, there are so many new tools available on top of the AI itself. Are there good resources for somebody like me to get started, that'll guide me though how to think about prompting, and using CI and how go have the agent successfully write specs and tests and what-not. I don't mind paying some for a good course, but I find it hard to figure out which one would be worthwhile, or whether there are some youtubers that would be better to follow.
Any tips?
The best option at this point is to just sign up for a paid plan with either ChatGPT or Claude and then ask the model the same thing. My preference would be for ChatGPT and if you've been out of the game for a long time then using the desktop app might be the best choice https://chatgpt.com/codex/
Then try starting with voice mode (if you're comfortable chatting out loud) and just talk your way through it.
They keep breaking inertial scroll on Mac though, which is something that irritates me thousands of times a day.
First they made it so when it came to rest it would then start scrolling again and overshoot by 2 lines before coming to a halt.
They fixed that after a few weeks then it worked until last week and now it’s left in a state where as it’s coming to a halt it jerks every time like dropping frames of the scroll almost.
I know it’s not my machine, config or trackpad because I’m seeing it on multiple computers.
The article is great reflection how LLMs are actually doing Quality assurance on both security and codebase sides, but i still doubt the creative part that means UI side, So far in my experience LLMs at best are unable to determine what the good UI is , may be its not inherently a bug detection rather a creative process
I assume this was thanks to Project Glasswing + Mythos Preview?
Sad they didn't say `Thanks to Gemini`
Concerning even
I recently switched from Chrome to Edge after using Chrome for decades. It is so bloated and slow. I can have same number of tabs open in both and Edge is always 30% to 50% more efficient.
The next step is to build software without bugs
spent a several months working on their bug bounty program earlier this year while working full time and lost a lot of motivation after a duplicate combined with lower bounty awards and claude safety guards all at the same time
Interesting to see how many new bugs were created from all these changes.
lol given the state of Gemini at the moment, they probably used Claude.
Still can't stand Chrome.
won't forgive them either for making it harder to filter out ads
Lies, obviously. AI is worthless.
And how many vulnerabilities did they introduce ;)
Did they fix spell check? Because it still doesn't work and all the solutions are basically "wipe and start over".
Did they fix that bug with ublock not working properly on Chrome?
I can't comment on Chrome bugs, but I can comment on Safari bugs. According to a search of the WebKit Bugzilla, I'm the most prolific filer of web inspector bugs outside of the WebKit team itself.
Recently I noticed an unusually high number of web inspector bugs fixed by one longtime WebKit engineer, and I suspected AI assistance, though the commit messages include no disclosure of this. Nonetheless, the increase in volume was quite dramatic.
The other day, an update to Safari Technology Preview was released, and I attempted to verify whether a couple of my bugs were indeed fixed. However, it was impossible for me to verify, because the latest version of Safari Technology Preview introduced a new web inspector bug that totally broke the features I was testing.
Thus, I remain unimpressed.
Without a doubt, LLMs have demonstrable skills and can produce code much faster than humans. I never thought that producing code fast was wise, though, even before LLMs arrived. For many years I've criticized software development based on management-driven release schedules, where developers are forced to pump out code regardless of quality, regardless of whether it's ready. Your boss may not like it, but code is done when it's done, not when your boss says it has to be done. If software complexity were predictable and reproducible, then indeed management could replace engineers with automation, but that's not how it works in reality.
Back to Chrome, the browser is notorious for continually adding invasive new "features" to the web that nobody wants except advertisers. I wonder how many Chrome bugs were the result of Google push push pushing all of this new crap on users over the years?
I’d rephrase this to something like: “People fixed more Chrome bugs (…), thanks to people who leveraged AI tools”
Good. Now please fix youtube. Amount of bugs that I'm getting in youtube app on iphone should be embarassing for company like Google. I cannot believe how bad it become in last couple years.
a lot of these bugs wouldn't have ever been found without AI
I wish they'd spent more time on improving the UI now. It's largely been the same since Chrome appeared. Their recent vertical tabs are so much inferior to Dia, which is built by a much smaller team.
Maybe fix all the decade-long old bugs first.
AI meaning Claude or Codex, because Gemini is a laughingstock for coding. Also Gemini was mentioned only once in the article.
I’ve heard from people inside Google that they’re prohibited from using anything but their own models. AlbertaTech on YT, ex-Google working on YT said in a video that right before she quit someone had threatened to quit if they couldn’t get access to Claude Code
The Gemini (and Claude / Codex) that we have, is much different to their internal models / harnesses.....
The latest and greatest internally at Google is more or less just Antigravity 2.0 with the latest Gemini models.
Trust me bro?
Kind of, but you can also assert it yourself; the big tech companies have billions of lines of code not publicly available, they can train their in-house code assistants on those instead of or in addition to what's available out in the open.
Earlier today I opened Chrome on my iPhone, went to the Reading List. It started flicking fast and then the phone restarted. That's a first for me on iPhone.
More important thing:
Do you import new more bugs?
Maybe they could ask the AI to fix their atrocious build times. Only in Chrome do you have 300 line source files that blow up to 20 megabytes after preprocessing as a matter of course. 3 GB per compile job is getting more common - and RAM is expensive right now!
This makes zero sense with the amount of engineers they have, most of those bugs even would be fixed for free by highly competent people.
But how many new ones did it introduce?
Real question is: how much those "fixed" bugs actually bothered the users to begin with?
But since Google cares mostly about its investors, the numbers and a mention of Gemini in such a blog post are more important.
Old bugs at the bottom of the prioritization list deserve to be fixed.
Some were probably just theoretical "if we have over 2B of this item, it'll crash because X/Y/Z and that can be exploited", some are probably in rarely used features.
Once the LLMs are done going through the code, the baseline will be all better and they will probably fix a lot less than they have now. Or they will have LLMs automatically check every single crash bug that has been submitted and take care of those too. I am assuming that they are probably already using this as an input source for their AI.
The whole point is that these are security bugs that didn't necessarily bother people so far but could be exploited
Me too! It's actually tiring! In a good way
I've mostly only used Codex for reviews, for Godot/GDScript code, and it helped me catch bugs that would've taken me ages to even notice on my own; games be tricky like that.
Claude was mostly useless or annoying up until the last time I tried it (about 3 months ago)
You have to be vigilant though: Codex often picks out obscure edge cases and suggests adding multiple new functions and flags to avoid issues that would be better off left as fast-failure crashes. Maybe that's because of the 5.6 Sol Max I leave it on.
The biggest bug still has not been fixed here, which is:
Google.
We really need an alternative to this greedy and evil corporations de-facto controlling a huge portion of the modern www stack. All decision-making processes are here subjugated to what fits Google's adEmpire. This is a perpetual system of control amplifying abusive systems in place. We already see this with the recent age sniffing; Google just yesterday announced that all android users must hand over their age to everyone else. Before that google kicked out or locked out open source alternatives to android (or, at the least, made their life significantly harder than before). These are not isolated ways of abuse - this is systematic abuse.
We really need alternatives to Google here. It can not be that one tyrant company dictates so much of people's open lives (and no, Firefox is no alternative; Mozilla is basically a nerfed and bribed entity that has given up on firefox many years ago already; ladybird may become an alternative at one point in time, but right now is not and there are questions over how Kling handles the overall project, but these complaints are significantly below what Google is doing globally here).
Fixing more bugs in the adChromium code base does not change the underlying problem at hand.
> We really need an alternative to this greedy and evil corporations de-facto controlling a huge portion of the modern www stack.
We do / have? Multiple even. Of course, the vast majority of consumers doesn't care so they don't get as much traction.
Seems like they have enough AI they could find a way to put ManifestV2 back into the browser and fix things we actually wanted instead of putting more bullshit "AI improvements" we didn't?
If only there was alternative browsers...
give it another year and AI will be filing bugs faster than humans can merge the fixes, and the Chrome team will need a separate AI just to triage the first AI's bug reports
Big AI player promotes AI through blog to make AI look better woo
A company valued at $4+ trillion based on the recent AI hype alone is touting the benefits of using said AI, and we're supposed to take this shit for granted.
I'm wondering what the Alphabet employees still commenting on this forum have to say about it? I guess for the right comps they can keep their mouths shut no matter the high level of idiocy involved.
There is hype, but there are also real tangible wins as well. Analyzing code for bugs is something that works very well. We use it for helping us revert changes that lead to a flaking test and it's dramatically faster than previous techniques. It's great at suggesting duplicate bug reports. It's amazing at scaling an API migration throughout a codebase. It's great at prototyping an idea before you decide to do it properly.
There are many things it's not good at (eg design and architecture), but that doesn't invalidate the things it is good at. Everyone is learning how to best use it and there are growing pains. Ignore the hype and try to find the common patterns in what people say.
Did anyone check the code changes manually? What if it is just slop code?
I don't know about Chromium specifically, but in general Google requires 2 humans to have looked at every change (normally one is the author, but not necessarily for bot-authored changes).
They did roll this policy for Chromium changes a few years ago too.
And it wasn't just 2 humans but 2 Googlers. One Googler author and one Googler reviewer works. But an external author needed 2 Googlers to review changes, and all files needed to be reviewed by one author at least.
So if you touched multiple files owned by different people for a more complex feature, you would naturally get there.
Google has not really embraced vibe coded slop. Humans are involved before any code merges and the quality bar remains high.
Did you go and look at their open source repositories and open code review tools instead of fearmongering on HN? The answer can be found.
So will the Chromium bug 5569 finally get fixed?
I'm not sure why you're asking here instead of opening the ticket and having a look at the current status.
I am actually using AI to learn about Windows OS performance. I have a couple of prompts scheduled to research, test, and implement performance gains in the OS, then post them here [0] and here [1]. I always admired Brendan Greg and Bryan Cantrill at Sun/Illumos/ for their work on OS performance but never had the time to sit down and catch up to their level due to my time in the military.
I got Codex to find and apply a bunch of settings I didn't know about and in the future I hope to use AI to make real changes in performance that venders probably don't have time or resources for.
that being said, what I've learned so far is essentially WhyIsItAlwaysHN's entire comment "The place to find them would be performance profiles, query plans, telemetry."
I just find it so frustrating that for a user interface that was basically solved in the 90s, Windows GUI still struggles to survive on hardware exceptionally more powerful than what we had 30 years ago
0 - https://www.lacksan.com/updates/ 1 - https://github.com/Lacksan-Dev/HP-ZBook-Performance