Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.
Why have we not seen an improvements in products?
While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.
Is everyone just running full speed in circles or something?
A good chunk of what my company has been doing with AI falls into either burning down our known tech-debt and "easy wins" that no one ever had the bandwidth to approach... And improving / automating our processes. The former is having a direct and meaningful impact on the quality and availability of our services.
Our QA, formerly a fairly frequent blocker of all our releases, are doing more in-depth reviews and catching issues earlier in our release process. They have become unblocked to the point they are actively chasing down work that starts to slip.
We have cleaned up and tuned both our security alerts and operations logs and improved our tenant isolation in our service in a way that makes customer and formal audits SIGNIFICANTLY easier.
We're setting ourselves up for faster human development of the hard-things. Our development environment and infrastructure are faster, cleaner, more auditable processes, and cheaper overall to operate.
These fixes mostly don't show up in our product change logs, and definitely don't fall into "new features". It would largely be invisible to the outside world, but our costs are going down (though to be fair, not offsetting the spend on AI to date), internal productivity has improved, operational incidents are down, and customer satisfaction is up.
PS5 emulation has gone from barely working to running Dark Souls at 10+ FPS with virtually no graphical glitches… in 6 weeks. If you’re not familiar with normal emulator development time, this is… quite extraordinary. There have been insane progress on decompilations and many other things in the emulator space.
Everyone’s trying to figure out how to convert this speed to product features at scale, but enterprises are like container ships. Lots of might but slow to turn. The littler companies can actually take advantage of this and produce higher quality products at much faster speed. I think you’re expecting too much in the short term and too little in the long term. AI-native companies are gonna eat everyone’s lunch, once they figure out how to actually do it reliably.
Also, in 12 months, we went from seeing game dissassembly and decompilations projects be slow and annoying, to suddenly having enough mapower where people are taking the liberty to CHOOSE which decompilations to support. DK64's release was 80% anti-AI marketing, not because they were strictly anti-AI, but because they were telling the OTHER decompilation to fuck off and learn some goddamned standards. Completely unthinkable before.
There was a video by a layperson 2 years ago where they wanted to know the mechanics of some elusive pokemon pinball spawns where there was a whole bunch of missinformation about it online. The people behind that project had the correct addresses but they were completely unlabeled and the layman had to essentially figure out and do some of the work themselves to sort it out and figure it out. Nowadays that would never happen. An LLM will just do it for you within the day.
Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
For that other stuff you mentioned like the right click menu. Those huge corporate projects suffer more from layers of institutional dysfunction and will be very very slow to show any improvement. Their dysfunction can't be solved with just faster coding.
Using AI to build more features is easier than using AI to improve existing projects. People will gradually figure out how to do latter too, it'll just take longer.
That's just volume though. I also know, and have no trouble believing, that github is going down partly due to the weight of all the vibecoded pushes. But that does not necessarily translate to people's needs and wants being covered, for all we know iOS just has a plague of unused POCs.
Do you have actual productive examples? As in, products with a real userbase that couldn't exist or be scaled pre-AI? Genuinely asking, I might have missed some large hits. The closest I can remember was bun rewrite kerfuffle, which seemed more a marketing action than anything.
If we sorted through all those new apps and ignored all the crap, I'm pretty sure we would find an overall increase in actually useful apps. I just shipped a new app myself, and I think it's useful, and I wouldn't have finished mine without AI assistance.
> products with a real userbase that couldn't exist or be scaled pre-AI?
Coding hasn't been the bottleneck for product creation for a while. So I don't think there are going to be many examples that fit that exact criteria. Any app idea was 'possible' before, it just took more developer hours to do it. So now teams are doing more things that were weren't worth the effort before, but are a lot more feasible with AI assistance.
more code faster is competing against some of the largest tax hikes of the last ~70 years, economic uncertainty on what the taxes tomorrow will look like, combined with an energy crisis.
Are you claiming productivity should be growing much more slowly then it is over the past few months or years? Can you show me any analysis that supports that claim?
Because I have seen plenty of analysis published in the past 30 years that puzzle over the productivity growth stagnation in the face of an exponential growth in capital expenditure in computer technology.
So by all means share with me some of the groundbreaking results showing that we can finally see more productivity growth than would be otherwise expected by the conjecture.
Once the previous goalpost gets demolished on HN, there's always someone new to put a new one. We aren't quite there yet but eventually the goalpost moves to something that is just impossible to even measure. That'll be the end game surely
> I'm pretty sure we would find an overall increase in actually useful apps
I think you're probably right but at what point does it become diminishing returns? I have observed the app market to be overly saturated for years and rarely download something other than a mobile-banking update.
> If we sorted through all those new apps and ignored all the crap, I'm pretty sure we would find an overall increase in actually useful apps. I just shipped a new app myself, and I think it's useful, and I wouldn't have finished mine without AI assistance.
App store revenue actually decreased! That suggests that the apps are largely not useful, unless you make the implausible assumption that nearly none of them are charging money.
What you're saying wouldn't be hard to prove, you're acting as if we aren't aware of usage statistics. We can even tell if MAU (monthly active users) are increasing or not for particular apps. So we are aware of them and what do they actually say?
In the gaming space, retro console PC ports have absolutely exploded in the last year. People are modding old games with 4k textures, ray tracing, DLSS 5, widescreen support, uncapped framerates, dual-screen (e.g. Ayn Thor), etc. PS5 emulation has gone from almost nothing to AAA titles in-game in like 2 months, and apparently now runs on xbox (lol).
The other day someone posted a reverse engineered GPU driver for their Mac[0].
Anyway, what makes you think those pushes aren't individuals meeting their own needs? I expect it will take a while for this to really sink in, but the future is people asking the computer to build exactly the app they want.
whats the large hits that have happened in the past couple years?
I think ai has raised the floor quite heavily on what it takes to be a large hit.
all those vibecoded pushes are people building bits and bobs and variations on each other, instead of buying it or using a common service. If it could be done by ai, it will be written off as unimportant.
Has anyone said that AI is fixing user research, actual idea generation, marketing and audience reach? Or are you assuming that better code should be making all of that irrelevant?
It either delivers better outcomes or fails to do so for not being as general and useful as touted.
The original sales pitch from the major player in this technology hype cycle is "cure cancer, fix global warming, take over the economy". Not "more LOC".
Not OP, but I ported a legacy .NET WebForms application to Blazor. The actual code migration was completed over roughly a 48-hour period, followed by fairly extensive testing.
We were fortunate to already have a strong end-to-end test suite written in Python, so we could run the new application against the same tests and verify that the existing functionality was preserved. QA found around 20 bugs, which we fixed pretty quickly before launching.
After the migration, we also moved the application's authentication from Shibboleth to Entra OAuth and deployed it to our OpenShift infrastructure. We couldn't do that with the old WebForms application because our cluster doesn't have Windows worker nodes, so the legacy app had been stuck running on VMs. Getting it onto OpenShift gave us another operational and cost-saving benefit beyond simply modernizing the codebase.
I did this back in January, when the models finally became capable enough for this kind of work. I believe I used GPT-5.2 through Codex. Successfully completing this project is what finally got me fully on the AI bandwagon. I had used AI before, but mostly for smaller tasks like writing code, refactoring individual methods, or making isolated changes.
The application is now modernized, more stable, faster, and more functional than it was before.
I think the project worked as well as it did for two important reasons. First, I had deep domain expertise in both the application and its surrounding systems because I was the original developer. Second, our QA team's test suite was comprehensive enough to validate the functionality that actually mattered. AI dramatically accelerated the work, but there still needed to be someone who understood what the application was supposed to do and a reliable way to verify that the new implementation behaved correctly.
None of this means we couldn't have done the migration without AI. We absolutely could have. The difference is that AI made it possible to do it at practically zero cost in terms of engineering time and money compared with the alternatives.
We had wanted to move away from WebForms for years, but with the size of our team and the constant stream of new feature requests, there was never a realistic opportunity to stop development for months and focus on a rewrite. I estimate that doing the migration myself without AI would have taken at least six months to do properly. The other option would have been hiring a contractor, which I would estimate at $80,000 or more over six to nine months.
Based on my token usage, those roughly 48 hours of working with Codex to port the application cost about $150. Those were January prices, so I don't know what the equivalent cost would be today.
For me, that was the project that changed AI from something useful for assisting with individual coding tasks into something I saw as capable of fundamentally changing the economics of software engineering work.
I have this theory that the economics of AI development would make it so it's much more viable to make in-house, custom, apps rather than paying subscriptions for apps. I have even seen this play out on a personal level. A friend of mine who has no dev experience vibe-coded something custom that evite does. So it's possible measuring the number of apps miscounts ones that are not public or shared. I'm not negating anything you said, just adding something about metrics on apps.
Last time I looked the other app stores didn’t show anywhere near that much of a bump (that was a several months ago though).
And a few months ago I went through a random sample of new apps on the App Store and the vast majority were just wrappers for AI APIs. So it was more of a new gold rush situation than a productivity bump.
> Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
That's exactly the opposite of what I want to see. As an app store user, I want to see fewer, higher quality apps, not more shovelware or slop.
>Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
People mean where are the good new apps AI made possible. Of course it has a 84% year-over-year slop app submission.
Probably for the same reason that SV companies hiring thousands of developers struggled to improve their product much past the original product, that was built by a handful of people.
Scale in headcount was a tactic to get investment, then you had to find stuff for everyone to work on. Suddenly people have the time to engineer so hard that we get runtime JSON defined CSS rendering engines to produce the same buttons we've had since 1995, instead of just writing a stylesheet and html.
Code output velocity from AI threatens to be useful, except it's also prone to over-engineering and burning tokens on the unnecessary. It's learned from the best after all. My suspicion is there is a lot getting done, but it's just not that impactful to flagship products.
As others have noted, there is a lot of new work going into passion projects that would have never happened otherwise, and that is cool. But I wouldn't hold my breath for SV tech to become super pragmatic and effective.
With pay-per-token, there’s also an incentive for over-engineered but functionally harmless architectures. A json based css engine is probably something that can be test-cased really well in the training set.
those thousands of developers aren't working on the FB feed or whatever...
they're working on the ad tech business and other business-related systems involved including internal tools.
and all the other platform engineering shit under the hood that makes much of the web scaleable... and lots of it is open source and contributed to by various engineers from these companies.
end of the day, these are businesses. they're not charities or whatever casual shit.
nobody is stopping you from building a competing product that's lean or whatever.
why don't you do something like that? i'm sure you're a genius.
Similar to what others have said, we have had big changes in our systems, thanks to LLMs. Some examples of what we have done:
- Rewrote a data extraction and PDF bounding box algorithm - LLM provided the tooling to visualize the output of algorithm and find the right rules for our needs.
- Migrated an old SciBERT model that was bundled into a 6GB docker container that needed GPU, with to an ONNX based inference container about 900MB running on CPU.
- Setup a K3S cluster to replace our Nomad cluster.
- Optimized an algorithm that used SQLite with better indexes and optimized querying with some 60-80% performance gains.
We have reduce resource usage in specific areas of the application drastically. These were all possible before, but LLMs provided the tooling to iterate and deliver it in time and cost that seemed prohibitive before.
Now are the end users going to see the benefit and is the product magically better? Well no. The chicken answer is I am not involved in that side to know. But a more realistic answer is, user experience and product fit is not something LLMs can solve. That's still upto the humans to figure out and I think that's where this "nothing has improved" feeling comes from.
The follow on question is if it's making us all so much more productive, where is the increased revenue? As far as I can tell, it's mostly the AI labs seeing that, not everyone using them (modulo small founders building new things and doing okay, I think)
If every software company saw a roughly equal improvement in productivity, they'd still be splitting the same customer base amongst each other - not clear whether revenue would actually go up for anyone.
Perhaps some new markets could be entered that weren't feasible before?
Personally I believe that this boost in productivity will not necessarily lead to greater revenue.
All the companies have the same access to AI, and AI is making them all better at doing what they were doing before (writing software). So some companies that use AI really well may be able to take market share from other companies that are slow. But I think this might just lead to a more intense competition for customers.
Kinda like what happened to music after digital recording became the thing - we have more music than before and the music is better, but being a musician became a much more intense competition to find an audience.
Well I don't really know what will happen in the future. My first thought is that what will happen depends on the company and industry. I can think of some non tech companies where AI might not matter for a while, like a car wash or a lumber mill. But long term likely yes for tech companies and companies where tech makes a difference.
Isn’t cloudflare a pretty clear answer to these? They are launching more products than ever, some of them clearly vibe coded, and their revenue is exploding.
Do you have a source for that? Are they raising more money or receiving more money for AI-oriented products or are they actually making more money on consumer/B2B end products?
Agreed, but give it time. The tipping point of good models only just arrived with Fable 5, GLM 5.3, Grok 4.6, Muse Spark 1.3, and the like. Those are the only models that do a better job than me, and I'm happy to hang up my IDE—and that was barely a month ago.
I've been fixing everything that has been sitting there—not blocking us, but slowing us down—all the things we never had the bandwidth for. We're now squeezing more out of development, CI, and production.
I'm not sure why, but I'm concerned this may be the heyday, and we might not get this again at this price or speed. So I figured we should clear the backlog while we can still afford it and still have the ability to do so.
But now things have changed, and the bigger, grander ideas are starting to brew.
> Why have we not seen an improvements in products? [..] Is everyone just running full speed in circles or something?
The simplest explanation is that they don't give a flying flamingo about what you or I consider "improvements to products".
This report is an example.
There are several changes that modify CI behaviour, where the article gives no corresponding quality measurement.
They replaced type aware custom lint rules with AST-only static analysis. They don't say anything about what those new rules detect, didn't do old-vs-new rule comparison. They switched the TypeScript check from tsc to tsgo. Again, they are very proud of the performance improvement, but don't seem to care about diagnostic equivalence. The list goes on. They don't even report pass/fail agreement between the old and the new CI. They have 4x more tests, but no idea whether this big test suite works any better than the smaller old one, or even whether it works at all.
The same reason it took many years for corperate america to get a real productivity increase from computers and the internet, all the old ways of doing things had to be redone. I think the largest companies are least equipped to take advantage of AI productivity gains. Agile no longer makes sense, Org charts as no longer make sense, etc...
I've been bootstrapping for two years now, just me and AI tooling.
Code moves much faster, product taste doesn't. Once you have your marching orders you can make features 10x faster. Shots off target still miss though. It doesn't matter if your code gets generated 10x faster when it doesn't resonate with users.
In my experience everyone is just rebuilding the same wheel over and over.
A lot of people may be more empowered to create things now with less up front effort but it doesn’t lead to having better ideas or more actual system architects.
Why do you assume "we not seen an improvements in products"? Who is "we"? Have you done an exhaustive (or even half-assed) analysis or is this just your "vibe"?
>Why do you assume "we not seen an improvements in products"? Who is "we"?
Have you? As in, can you name products you use daily that have been transformed night and day?
I am not talking about the existence of minor improvement, mind you. I am denying the existence of anything that is minimally consistent with the level of discourse that surrounds development itself (throwing away practices, checks and balances, hitting constant roadblocks due to the new speed, etc).
A true 100x requires no analysis. If we could suddenly build and improve houses at 100x I would know because I would be typing this from my 5th living room.
> When did previous releases stop meeting your criteria?
Ages ago. It was really noticeable when they prioritized features over quality.
> Such as?
HN is full of them. I don’t have great have specific example at the top of my head, but lots of hard reverse engineering/low level projects credit AI and that they would never have happened without it these days.
AI doesn’t make a responsive right-click menu a higher business priority. But we are seeing a ton of small custom projects that are as easy to dismiss as they are to abandon.
Surely it makes the backlog clear faster to the point where you reach the non-priority stuff? But I'll bite, what business priorities are being created at 100x?
The real question is why are people too lazy to spin up their own phorge. It's a form of learned helplessness supported by a supposed business case to prefer buying services rather than bootstrapping your organization with proven technology.
I've definitely noticing buggier and buggier software, that's for sure. Even our internal tooling and CI pipelines and all that kinda stuff has taken nosedive thanks to the deluge of garbage
the important thing to microsoft is still finding new ways to make money, not to make your clicks faster. they might even be using their new velocity to make clicks slower with an ad in between.
there's alao much less incentive to build things to share. people can just make their own widget app and not need yours
Well you mention Android, iPhone, Nintendo, and Microsoft. Those are the biggest companies, and many of them don't even use the best coding tools. So they are not going to be the ones improving.
On the other hand, look at the YC companies. I think the last batch or two is growing revenue faster than any batches ever before. Those are the companies that are being sped up a huge amount by AI.
It’s hard to imagine a company more all in on ai than Google. The pressure to use it is intense and they are throwing ai at basically every problem. If a proposal doesn’t have ai in the title somewhere it has no chance of funding.
I don’t really think it has made Google products better, but it is absolutely the expectation.
well people are spending tokens like crazy for sure, Anthropic and all other labs and everything related to AI seem to be making tons of money.
On a serious note, it might take a while to realize the actual benefits or losses. Its clearly not a good signal when people whose job is to manage other people start writing their own pet AI projects. This just indicates that AI has created this big job insecurity among everyone. At the end of the day being an engineer, eventually the job is relatively safer when there is so much code being written out there.
AI makes a lot of drudgery type coding tasks a lot easier (think code migrations, etc.)
But actually coming up with, testing, and rolling out loved new user features. AI isn't that good at that and we can't really "prompt" that out of it as easily as we can prompt a py2 to py3 conversion of old code
I mean, looking at the release notes for things like say the Linux kernel it is clear that the improvements are there at a massive rate of change. But it's small stuff, it's performance improvements, it's quality of life. If you do things right people will think you've done nothing at all. What did you expect? Businesses to come up with whole new business lines, or is it just that backlogs of work is getting burned down that was going to be gotten to eventually? From what I can tell from the PM forums, the next bottleneck is the Product Org coming up with ideas worth implementing as fast as the developers can deliver them.
In my case it's not the CI that's the bottleneck. It's the human testing side. Does it work, sure. But does it actually do the thing we want (and more importantly) does it do it in a way our customers will understand and actually like?
I feel like if anything LLMs have reduced coding/software to "throw as much as possible at the wall and see what sticks". It feels quite shortsighted and wasteful, especially considering that humanity needs to get better about how it produces and consumes energy. It's sort of bleak.
Which is a bit terrifying and removes the ‘engineering’ component of a critical part of the job, in exchange for slot machine / pray to ai gods…which comes with disconnect from the code, something I assume will adversely compound over time.
to me the last part if the most important part. product management hasn't been automated at all, and is more important than ever. you can't just keep adding features to make a great product
Think of it as a layered problem. If the bottom layer (CI) cannot keep up with the output of agents, then solving problems at a higher layer - like user experience checks - will be exponentially slower and less reliable. Kind of like how optimizing tight inner loops makes your whole program faster.
QA is not automatable. QA is about finding the things that you didn't think of and so didn't write a test for it. That many people think qa is not important shows in the bad software we have.
Clearly you have to replace obsolete human testers with agentic AI testers, duh.
At some point, with all this velocity, human users become the bottleneck, unable to keep up with and learn all the changes and new features. Luckily, there's a simple solution: just replace the human users with agentic AI users.
> The forceful executive, Henry Ford II, and the leader of the automobile workers union, Walter Reuther, both saw many examples of advanced machinery operating at the plant. The words they exchanged brilliantly encapsulated the paradox of automation:
> Henry Ford II: Walter, how are you going to get those robots to pay your union dues?
> Walter Reuther: Henry, how are you going to get them to buy your cars?
I think you're being sarcastic, but there is actual truth behind what you're saying.
Because every developer is now a slop cannon by default, by default users will experience churn and whiplash, and things will break all over the place. As you point out, this is bad. It's also impossible to fix without deploying agents on the QA side. Like it or hate it, agentic testing is inevitable to protect users from the churn and noise caused by the slop cannon. I don't think that replaces test engineers at all - if anything it makes the job more fun. If you've ever had to keep playwright tests in sync with the target manually, and kept the CI environment up to speed with toolchain changes, you know what I mean.
Whether the "slop cannon by default" situation could have been avoided in the first place, is another question... But we're here now and there's no going back. Might as well deal with it as best as we can.
> Moving our workloads off GitHub Actions to third-party runners with faster CPUs, higher-performance storage, and better cache infrastructure gave us faster machines to run the same pipeline on
Yeah, was not surprised to read this. Actions is convenient if you already use GitHub, but it can also be pretty slow. Given reliability is also a major issue with GitHub these days I expect to see more orgs moving to different pipelines
I think its actually because GitHub isn't solely running on Azure. They are currently using a mix of on prem, AWS, and Azure. The Azure migration has been a challenge with Azure running into capacity issues due to AI load.
We switched our Actions workload to blacksmith.sh (not affiliated) and have been pretty happy with how fast and inexpensive they are. I wouldn't be surprised to see this trend continue.
I’m conflicted about Linear’s progression. I dislike some of the features but on the whole they’ve managed to keep the software pleasant to use, it doesn’t feel to me that it is drifting towards Jira territory, rather, it feels like it is losing the carefully considered product design because now code is cheap to generate. I’m not worried about it turning into Jira but it has lost its soul. Still a great product.
I haven't used Linear in any serious capacity, but I feel like there's plenty of market in "Jira that doesn't feel like it hates its users" and "Jira but we care about performance".
Not saying that they're shooting for either of those segments, but someone should. Plenty of enterprise orgs that need (or are convinced they need) Jira's featureset.
All software becomes popular because it's not the old behemoth, but it's new, fast, and simple. All software then grows and grows until it becomes the new behemoth, and the circle of life begins anew.
My biggest takeaway (as a one man band 20 leagues below this spending time optimizing Gitlab free plan and self hosted runners): they did not worry about any of this until 100M ARR and more 1B valuation[0].
I lead a bazel conversion for a pretty complex piece of software written in 5+ programming languages and shipping native binaries to all 3 major OSes a few years ago, and it took multiple years to get it done.
For a less complex project (1 programming language, still shipping to all 3 major OSes), with my knowledge and agents I got the bazel conversion done in 2 weeks.
The setup cost for bazel just went down by a lot, and I don't think the industry as a whole is aware of that yet.
The entire industry, including its outputs that LLMs are trained on, hasn’t reconsidered what’s easy vs. hard or fast vs. slow. LLMs consistently recommend against code changes because they will take “a weekend”. No, Claude. You will do the work and it will take 20 minutes.
Can't comment on Bazel specifically, but having worked with both nx and turbo, the bottleneck was usually network and disk IOPS rarely compute.
Even fully cached outputs needs to fetched and read from a remote server[1]. A step n-1 outout fetched from remote cache server need to written to disk and then again read by step n[3] - all disk I/O and network bound operations.
10s may be achievable/realistic goal in the Java/C++ world where Bazel normally seen. In TS eco-system most people would be over the moon to get into ballpark of 1-2m for a decently large monorepo.
We should define Build more clearly here, if you mean running just transpile/compile steps or the full series of steps that includes tests (as the linear post here is talking about). It is hard to see even a small sub-set of a large suite of test that require a virtual DOM or a real browser can run in 10s or less.
[1] Typical for say managed CI setup .
[3] Common run-of-the-mill frontend + backend stacks in different languages etc.
If you don't need to rebuild anything, bazel can fetch only the final artifact (not the intermediates) from the remote cache.
Also, if you have persistent CI workers with a persistent bazel instance, you save on some network roundtrips, but that's obviously harder to set up and make bulletproof.
Typically you need the intermediates to compute if you need the next one , so you cannot skip to final until you have the intermediaries.
The final asset/artifact is rarely small either. even best optimized artifacts can be few hundred MB docker image or more commonly multiple image layers running GBs in size .
each step is a network pull then recompute cache if stale and keep going till end .
Bazel seems to have a lot of tradeoffs, from setup time of the sandbox for each task, to ergonomics that lead folks to maintain parallel 'normal' tooling.
Plus, 'with a warm cache' is doing heavy lifting, what's the real cache hit rate for a week of development? Investing in improving the cold build and frequent actions is still important with bazel or any incremental builder.
I'm not sure it's useful to talk about bazel broadly, it's actual performance and behavior comes down to the rules you use. You can configure bazel like turbo/nx and cache tsc/vitest/eslint on each package.json module, and get course cached units that are evicted on every change, or you can use gazelle and target per-file actions which are only invalidated when their dependencies change. But that trades off batching unless you use workers.
Depends on which targets and how granular the caching is.
- if you touch package.json that might invalidate everything
- touch core and you'll invalidate everything
- touch one file in api and you may run all api tests (see gazelle)
I ran an experiment where I migrated a package to bazel then replayed a weeks worth of changes and it saved 20%. That's nothing to scoff at, but not the headline numbers you see after a full hot build.
With the widespread increase in speed/commit cadence/content added to repos, I wonder if there's an angle here for companies with a lot of CI/CD needs to simply start self-hosting their CI/CD machines on premises.
In my experience, self hosting your CI/CD runners had the biggest impact in cost savings throughout, while also allowing for more powerful machines, which directly means quicker CI/CD runs
My take: it seems like systems should become smaller, more isolated, and contract-oriented.
I have been a long time proponent of monoliths, but it seems like agents would be happier with smaller, more isolated services. The more isolated, the better. Contracts between the service components only. Then it can iterate internally as long as it satisfies the contract. If it needs to, it can version the contract and keep iterating.
None of them are bad. It's like saying a bike vs a scooter vs a car is bad. Use them correctly and for what they're intended and they work fine. Use them improperly for the wrong things and suddenly people think the tools suck, when it's the humans who misused them that suck.
I think to solve this problem properly, we have to stop treating "build" and "test" as separate buckets of work, to be designed and scaled separately. It's all CI. As soon as you try seriously scaling out tests, you will run into build bottlenecks. To truly scale CI you need a scheduler that understands your build, test environment, and all the glue in between, well enough to schedule it in a way that actually speeds things up. That is very difficult and not something that even the best build tools can do - yes, even Bazel. Bazel can run tests but it's not nearly as good at it than at building.
Agentic Coding has been a huge strain on CI,
I have been using Bazel to improve our build times and ultimately building customized runners to improve our CI
Anyways great blog post from linear team a lot to learn from it
Mostly around, at least the majority of 'stuff' I've worked on (both before and after the rise of coding agents) myself or others took enough time to make sure that anything done locally, if you run the tests locally, you're at least 90% of the way there as far as what CI/CD does.
I suppose the flipside being, most of those projects had less churn (i.e. one person was working on a service at a time, and we had good contracts between services.) Also, Our local boxes were way better than our CI boxes, so there was incentive to run locally versus waiting 2-10x the time for CI to run...
I suspect a substantial part of this is an avalanche of useless testing.
If you even review PRs still: when was the last time you didn't just skip over tests? And if you ever looked at tests in an LLM-heavy PR, how many of those tests tested something useful, and not just built-ins and trivial behaviours?
There's at least some awareness in the industry of how LLMs generate a lot of boilerplate in business logic. It feels like we're much less aware of how much of it is in tests.
Our CI has thousands of tests and is done in 2 minutes. Everytime it grows above 3 I add more parallelization and keep it ultra fast. Been working great for our team
I wonder if the GitHub actions outages we keep seeing is due to themselves making self-hosted runners paid, hence bringing broke/cheap users back from hosted runners to their garbage infra.Meanwhile I'm running my Codeberg Actions on the free Oracle ARM machine 2 cores 12GB ram (previously 4 cores 24GB) and way more reliable. Keep winning bozos.
Edit: After second thought, I guess "alternative runner" providers still have to pay the self-hosted tax. So M$ actually saves and makes money by not scaling their infra and driving people to alternative providers they can tax freely. Actual geniuses.
They abandoned (or at least postponed) the idea of charging for self-hosted runners usage.
Actions management plane is the most problematic part of the platform (in regards to scale), so they are basically losing money with self-hosted runners.
Scaling of VMs shouldn't be a problem, since that is practically stateless, whereas there are lots of operations performed for every job/workflow.
We are working on something here, pushgate.dev. It forces the agent to run tests; we are also working on adding support for static analysis of bundles using Jev. Onboarding is a bit of a mess right now, but send me a message cole@testifysec.com if you are interested.
I did that by self-hosting Github Runners, using Podman to build images (it doesn't squirt your build context through a socket every time!), and using Docker images as content-addressable cache.
> Despite our test suites almost quadrupling since the start of the year
I think the SDLC needs to be re-evaluated, tests, specifically unit tests, are just cosmetics. "wow its so easy to get high coverage now!" as opposed to why we're doing it in the first place
I don't find agents to be using tests any differently than a junior or mid level developer. I don't find humans to be using tests any differently than a junior or mid level developer either.
Basically, the tests are never guiding the features, they aren't highlighting regressions, the test is simply modified for the updated application. Unit tests especially.
I have found end to end tests to be useful, and also unblocked. They can tell when the actual user experience has changed, and go back to the feature's implementation to stop altering the user experience in unexpected ways.
additionally, I find this across every industry that was worried about AI. basically, whatever best practice was neglected due to lack of investment into your org is now being done without additional investment into your org. that's a good thing. now we need to look at the purpose of the best practice, and if that purpose itself was already solved in the process.
I just feel apps like linear are increasingly getting in the way of full send agentic development where sub agent orchestration is done through agent to agent messaging, work trees, on demand git restructuring and epoch specific coordination plains, often .md files. The smaller the human component of total product development gets, the more this may be the case.
> where sub agent orchestration is done through agent to agent messaging
How do you expect to see the history/record of what the agents did and why? Is it enough to see it in PRs? Do you expect tickets that have the design and history? How are you thinking of agents being able to historically resolve reasoning/why/decisions made in earlier passes?
Genuine open question here. My assumption is that a GH or Linear or Jira is still useful as a decision store. It may as well be a custom app over Postgres, but it seems like something is needed to store this and for observability. A GH/Linear/Jira is nice if only because of standard APIs and integration points (whatever you build would likely end up duplicating a subset of those).
Linear is a key part of my prompting technique. I write specifications in Linear, and the history I generate in it becomes a critical source of context for my agents. Prompting my agents has become “look at XYZ-124 in Linear and ask me any questions you have.”
My recent winnowwallet.com still has build times of over 10 minutes. I'm convinced the agent really wants build times around 5m to move at a quick pace. Also to not drive me insane. It took a ton of work to get it down from 45 minutes because my tests launch a full version of the app, and walk it through major usecases while recording video and screenshots. I then use AI to qa this. It also generates its webpage this way. All from CI/CD
I cannot believe we are in 2026 and CI/CD hasn't evolved enough to even consider hot updates. Burn your CI/CD pipelines to the ground and start over again without any of the slop. Don't let anyone who calls themselves DevOps Engineers design it again, only people who are System Administrators.
(Disclosure: I'm a cofounder of RWX) For anybody wanting to solve similar problems and considering Bazel, take a look RWX. It's built around the same concepts of content-based caching and graph-based task execution, but it's far more runtime agnostic and easier to adopt. https://rwx.com
Maybe I'm alone in this, but as someone who is in tech, I don't know what CI is, and I don't think it's unreasonable to expect it to have an expansion within the article the first time you use it...
AI is in the cultural zeitgeist, but you gotta expand most other things at least once.
So if you don't know what CI is, what does "I am someone in tech", mean? Marques Brownlee probably doesn't know CI is, but he's also in tech. Because if you by mean "I am in tech" that "I am a software engineer", and you don't know what CI is, then boy; I'd be worried for you.
Also the case of 'learning that someone did updates directly on AWS console instead of terraform and losing 0.5 or more days cleaning up the resulting mess'
I am a Site Reliability Engineer, I have been for close to a decade, and this is not as common an abbreviation as you think it is. Also, my entire job isn't in question because I didn't memorize shorthand for a term that I don't use in my day-to-day.
Really appreciate getting downvoted to hell for saying "We can't throw readers a simple bone?"
All I'm asking for is CI(Continuous Integration) in the first paragraph.
I didn't say I don't know what Continuous Integration was. I said I didn't know what "CI" stood for.
Cannot stress enough, this is like, basic communication/writing that they teach in elementary school.
But I'm so happy you're able to get off on your feelings of superiority that you always automatically know what those two letters stand for. I hope that translates to meaningful happiness and fulfillment in your life.
Within the engineering tech sphere, CI/CD have also been terms that have been standard for at least a decade now.
You would probably be rejected from most interviews at the first stage if you didn't vaguely know what they mean at this point.
Linears entire product is tailored towards software engineers/engineers in general or people who work alongside engineers, so its not surprise their posts have a bit of assumed knowledge.
(Continuous Integration, and Continuous Deployment, are fine if someone tells me what the letters literally stand for, but sorry for not being so tech-brained that I haven't memorized every possible TLA that's ever been used in tech).
Most days I feel like I must be the dumbest person on HN. I don't understand what 80%+ of submissions are about. But if it sounds interesting, I'll dig into it a bit and learn a few things along the way.
I recommend making a habit of googling or asking AI about terms you run into. As someone who is in tech, you will hear tech jargon your whole career that will not be explained if it's considered standard terminology for the audience being spoken to. It is better to learn to educate yourself than expect others to go out of their way.
It used to be a standard expectation, but that was before google or AI could have answered your question in less time and typing that your complaint on HN required. Especially when the author is writing a blog post for an audience that knows what CI is.
Here's my constant question:
Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.
Why have we not seen an improvements in products?
While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.
Is everyone just running full speed in circles or something?
A good chunk of what my company has been doing with AI falls into either burning down our known tech-debt and "easy wins" that no one ever had the bandwidth to approach... And improving / automating our processes. The former is having a direct and meaningful impact on the quality and availability of our services.
Our QA, formerly a fairly frequent blocker of all our releases, are doing more in-depth reviews and catching issues earlier in our release process. They have become unblocked to the point they are actively chasing down work that starts to slip.
We have cleaned up and tuned both our security alerts and operations logs and improved our tenant isolation in our service in a way that makes customer and formal audits SIGNIFICANTLY easier.
We're setting ourselves up for faster human development of the hard-things. Our development environment and infrastructure are faster, cleaner, more auditable processes, and cheaper overall to operate.
These fixes mostly don't show up in our product change logs, and definitely don't fall into "new features". It would largely be invisible to the outside world, but our costs are going down (though to be fair, not offsetting the spend on AI to date), internal productivity has improved, operational incidents are down, and customer satisfaction is up.
How is dev satisfaction?
A luxury of the past, for >90% of devs?
Which devs are you asking about, the humans or the AI agents?
doesn't really matter now, does it? Someone's gonna do the job, whatever it looks like.
PS5 emulation has gone from barely working to running Dark Souls at 10+ FPS with virtually no graphical glitches… in 6 weeks. If you’re not familiar with normal emulator development time, this is… quite extraordinary. There have been insane progress on decompilations and many other things in the emulator space.
Everyone’s trying to figure out how to convert this speed to product features at scale, but enterprises are like container ships. Lots of might but slow to turn. The littler companies can actually take advantage of this and produce higher quality products at much faster speed. I think you’re expecting too much in the short term and too little in the long term. AI-native companies are gonna eat everyone’s lunch, once they figure out how to actually do it reliably.
Also, in 12 months, we went from seeing game dissassembly and decompilations projects be slow and annoying, to suddenly having enough mapower where people are taking the liberty to CHOOSE which decompilations to support. DK64's release was 80% anti-AI marketing, not because they were strictly anti-AI, but because they were telling the OTHER decompilation to fuck off and learn some goddamned standards. Completely unthinkable before.
There was a video by a layperson 2 years ago where they wanted to know the mechanics of some elusive pokemon pinball spawns where there was a whole bunch of missinformation about it online. The people behind that project had the correct addresses but they were completely unlabeled and the layman had to essentially figure out and do some of the work themselves to sort it out and figure it out. Nowadays that would never happen. An LLM will just do it for you within the day.
To further this point, I’ve picked up the torch and got Switch2 controllers working in Dolphin and Cemu
Wind Waker in 4K w/official Bluetooth GameCube controller
(Nintendo went out of their way to invent a new protocol so that it didn’t “just work” like OG switch controllers)
Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
For the question where are the alternative OSes? Here is one that I've seen. There's probably more - https://www.reddit.com/r/ClaudeAI/comments/1wfpydl/i_asked_c...
For that other stuff you mentioned like the right click menu. Those huge corporate projects suffer more from layers of institutional dysfunction and will be very very slow to show any improvement. Their dysfunction can't be solved with just faster coding.
Using AI to build more features is easier than using AI to improve existing projects. People will gradually figure out how to do latter too, it'll just take longer.
That's just volume though. I also know, and have no trouble believing, that github is going down partly due to the weight of all the vibecoded pushes. But that does not necessarily translate to people's needs and wants being covered, for all we know iOS just has a plague of unused POCs.
Do you have actual productive examples? As in, products with a real userbase that couldn't exist or be scaled pre-AI? Genuinely asking, I might have missed some large hits. The closest I can remember was bun rewrite kerfuffle, which seemed more a marketing action than anything.
> That's just volume though
If we sorted through all those new apps and ignored all the crap, I'm pretty sure we would find an overall increase in actually useful apps. I just shipped a new app myself, and I think it's useful, and I wouldn't have finished mine without AI assistance.
> products with a real userbase that couldn't exist or be scaled pre-AI?
Coding hasn't been the bottleneck for product creation for a while. So I don't think there are going to be many examples that fit that exact criteria. Any app idea was 'possible' before, it just took more developer hours to do it. So now teams are doing more things that were weren't worth the effort before, but are a lot more feasible with AI assistance.
Has economic growth accelerated? TFP, labor productivity?
Where is the net benefit people keep talking about when they claim productivity gains are obvious because "more code faster, can't you see?"
more code faster is competing against some of the largest tax hikes of the last ~70 years, economic uncertainty on what the taxes tomorrow will look like, combined with an energy crisis.
Are you claiming productivity should be growing much more slowly then it is over the past few months or years? Can you show me any analysis that supports that claim?
Because I have seen plenty of analysis published in the past 30 years that puzzle over the productivity growth stagnation in the face of an exponential growth in capital expenditure in computer technology.
So by all means share with me some of the groundbreaking results showing that we can finally see more productivity growth than would be otherwise expected by the conjecture.
No the whole world sucks and you're the smartest guy who ever lived. What are you trying to find here?
Once the previous goalpost gets demolished on HN, there's always someone new to put a new one. We aren't quite there yet but eventually the goalpost moves to something that is just impossible to even measure. That'll be the end game surely
Was the goal of any investment ever not 'increase productivity'?
> I'm pretty sure we would find an overall increase in actually useful apps
I think you're probably right but at what point does it become diminishing returns? I have observed the app market to be overly saturated for years and rarely download something other than a mobile-banking update.
> If we sorted through all those new apps and ignored all the crap, I'm pretty sure we would find an overall increase in actually useful apps. I just shipped a new app myself, and I think it's useful, and I wouldn't have finished mine without AI assistance.
App store revenue actually decreased! That suggests that the apps are largely not useful, unless you make the implausible assumption that nearly none of them are charging money.
https://www.ft.com/content/e5533f32-e4c2-4ce1-9d2e-a831e9654...
What you're saying wouldn't be hard to prove, you're acting as if we aren't aware of usage statistics. We can even tell if MAU (monthly active users) are increasing or not for particular apps. So we are aware of them and what do they actually say?
In the gaming space, retro console PC ports have absolutely exploded in the last year. People are modding old games with 4k textures, ray tracing, DLSS 5, widescreen support, uncapped framerates, dual-screen (e.g. Ayn Thor), etc. PS5 emulation has gone from almost nothing to AAA titles in-game in like 2 months, and apparently now runs on xbox (lol).
The other day someone posted a reverse engineered GPU driver for their Mac[0].
Anyway, what makes you think those pushes aren't individuals meeting their own needs? I expect it will take a while for this to really sink in, but the future is people asking the computer to build exactly the app they want.
[0] https://news.ycombinator.com/item?id=49717638
People don't know what they want.
whats the large hits that have happened in the past couple years?
I think ai has raised the floor quite heavily on what it takes to be a large hit.
all those vibecoded pushes are people building bits and bobs and variations on each other, instead of buying it or using a common service. If it could be done by ai, it will be written off as unimportant.
the userbase of the future is 1, maybe 10
Has anyone said that AI is fixing user research, actual idea generation, marketing and audience reach? Or are you assuming that better code should be making all of that irrelevant?
It either delivers better outcomes or fails to do so for not being as general and useful as touted.
The original sales pitch from the major player in this technology hype cycle is "cure cancer, fix global warming, take over the economy". Not "more LOC".
OK but you talked about people going fast. 84% YOY increase in volume shows people are going a lot faster.
You can move the goalposts to "yeah but it's not any good" but 90% of everything is crap anyway, and that was true long before AI.
Not OP, but I ported a legacy .NET WebForms application to Blazor. The actual code migration was completed over roughly a 48-hour period, followed by fairly extensive testing.
We were fortunate to already have a strong end-to-end test suite written in Python, so we could run the new application against the same tests and verify that the existing functionality was preserved. QA found around 20 bugs, which we fixed pretty quickly before launching.
After the migration, we also moved the application's authentication from Shibboleth to Entra OAuth and deployed it to our OpenShift infrastructure. We couldn't do that with the old WebForms application because our cluster doesn't have Windows worker nodes, so the legacy app had been stuck running on VMs. Getting it onto OpenShift gave us another operational and cost-saving benefit beyond simply modernizing the codebase.
I did this back in January, when the models finally became capable enough for this kind of work. I believe I used GPT-5.2 through Codex. Successfully completing this project is what finally got me fully on the AI bandwagon. I had used AI before, but mostly for smaller tasks like writing code, refactoring individual methods, or making isolated changes.
The application is now modernized, more stable, faster, and more functional than it was before.
I think the project worked as well as it did for two important reasons. First, I had deep domain expertise in both the application and its surrounding systems because I was the original developer. Second, our QA team's test suite was comprehensive enough to validate the functionality that actually mattered. AI dramatically accelerated the work, but there still needed to be someone who understood what the application was supposed to do and a reliable way to verify that the new implementation behaved correctly.
None of this means we couldn't have done the migration without AI. We absolutely could have. The difference is that AI made it possible to do it at practically zero cost in terms of engineering time and money compared with the alternatives.
We had wanted to move away from WebForms for years, but with the size of our team and the constant stream of new feature requests, there was never a realistic opportunity to stop development for months and focus on a rewrite. I estimate that doing the migration myself without AI would have taken at least six months to do properly. The other option would have been hiring a contractor, which I would estimate at $80,000 or more over six to nine months.
Based on my token usage, those roughly 48 hours of working with Codex to port the application cost about $150. Those were January prices, so I don't know what the equivalent cost would be today.
For me, that was the project that changed AI from something useful for assisting with individual coding tasks into something I saw as capable of fundamentally changing the economics of software engineering work.
I have this theory that the economics of AI development would make it so it's much more viable to make in-house, custom, apps rather than paying subscriptions for apps. I have even seen this play out on a personal level. A friend of mine who has no dev experience vibe-coded something custom that evite does. So it's possible measuring the number of apps miscounts ones that are not public or shared. I'm not negating anything you said, just adding something about metrics on apps.
> For the question where are the alternative OSes? Here is one that I've seen. There's probably more
And that's just the people operating in public instead of private :P
Last time I looked the other app stores didn’t show anywhere near that much of a bump (that was a several months ago though).
And a few months ago I went through a random sample of new apps on the App Store and the vast majority were just wrappers for AI APIs. So it was more of a new gold rush situation than a productivity bump.
The OS one is funny to me, it's been hard to keep osdev.org online due to the insane amount AI bot traffic.
Thank you for your site dude! I had so much fun and learned so much building my own toy OS over a decade ago.
Any notable breakthroughs in the top 100 apps on the app store that could attribute their success to AI?
> Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
That's exactly the opposite of what I want to see. As an app store user, I want to see fewer, higher quality apps, not more shovelware or slop.
> higher quality apps, not more shovelware or slop.
Your prejudice is showing.
>Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
People mean where are the good new apps AI made possible. Of course it has a 84% year-over-year slop app submission.
Any real "killer apps" though?
Probably for the same reason that SV companies hiring thousands of developers struggled to improve their product much past the original product, that was built by a handful of people.
Scale in headcount was a tactic to get investment, then you had to find stuff for everyone to work on. Suddenly people have the time to engineer so hard that we get runtime JSON defined CSS rendering engines to produce the same buttons we've had since 1995, instead of just writing a stylesheet and html.
Code output velocity from AI threatens to be useful, except it's also prone to over-engineering and burning tokens on the unnecessary. It's learned from the best after all. My suspicion is there is a lot getting done, but it's just not that impactful to flagship products.
As others have noted, there is a lot of new work going into passion projects that would have never happened otherwise, and that is cool. But I wouldn't hold my breath for SV tech to become super pragmatic and effective.
With pay-per-token, there’s also an incentive for over-engineered but functionally harmless architectures. A json based css engine is probably something that can be test-cased really well in the training set.
those thousands of developers aren't working on the FB feed or whatever...
they're working on the ad tech business and other business-related systems involved including internal tools.
and all the other platform engineering shit under the hood that makes much of the web scaleable... and lots of it is open source and contributed to by various engineers from these companies.
end of the day, these are businesses. they're not charities or whatever casual shit.
nobody is stopping you from building a competing product that's lean or whatever.
why don't you do something like that? i'm sure you're a genius.
Similar to what others have said, we have had big changes in our systems, thanks to LLMs. Some examples of what we have done:
We have reduce resource usage in specific areas of the application drastically. These were all possible before, but LLMs provided the tooling to iterate and deliver it in time and cost that seemed prohibitive before.Now are the end users going to see the benefit and is the product magically better? Well no. The chicken answer is I am not involved in that side to know. But a more realistic answer is, user experience and product fit is not something LLMs can solve. That's still upto the humans to figure out and I think that's where this "nothing has improved" feeling comes from.
The follow on question is if it's making us all so much more productive, where is the increased revenue? As far as I can tell, it's mostly the AI labs seeing that, not everyone using them (modulo small founders building new things and doing okay, I think)
If every software company saw a roughly equal improvement in productivity, they'd still be splitting the same customer base amongst each other - not clear whether revenue would actually go up for anyone.
Perhaps some new markets could be entered that weren't feasible before?
But there have been trillions invested in infra for AI. Surely those investors are going to need to see a return at some point?
Personally I believe that this boost in productivity will not necessarily lead to greater revenue.
All the companies have the same access to AI, and AI is making them all better at doing what they were doing before (writing software). So some companies that use AI really well may be able to take market share from other companies that are slow. But I think this might just lead to a more intense competition for customers.
Kinda like what happened to music after digital recording became the thing - we have more music than before and the music is better, but being a musician became a much more intense competition to find an audience.
probably a naive question but then would we expect companies not using it to lose market share to competitors that are using it?
Well I don't really know what will happen in the future. My first thought is that what will happen depends on the company and industry. I can think of some non tech companies where AI might not matter for a while, like a car wash or a lumber mill. But long term likely yes for tech companies and companies where tech makes a difference.
Isn’t cloudflare a pretty clear answer to these? They are launching more products than ever, some of them clearly vibe coded, and their revenue is exploding.
I think you could make that argument, yeah.
https://au.finance.yahoo.com/quote/NET/financials/
Interesting to look at, a decent example for sure.
It may not have to do with increased revenue, but it does have everything to do with reduced cost, especially developer's cost.
I bet every company is finding up how to level up their employees via AI, so that they can use less of them in the future.
So even without increased revenue, AI has its (mis)uses.
The increased revenue is in startups, like the recent couple YC batches.
Do you have a source for that? Are they raising more money or receiving more money for AI-oriented products or are they actually making more money on consumer/B2B end products?
Agreed, but give it time. The tipping point of good models only just arrived with Fable 5, GLM 5.3, Grok 4.6, Muse Spark 1.3, and the like. Those are the only models that do a better job than me, and I'm happy to hang up my IDE—and that was barely a month ago.
I've been fixing everything that has been sitting there—not blocking us, but slowing us down—all the things we never had the bandwidth for. We're now squeezing more out of development, CI, and production.
I'm not sure why, but I'm concerned this may be the heyday, and we might not get this again at this price or speed. So I figured we should clear the backlog while we can still afford it and still have the ability to do so.
But now things have changed, and the bigger, grander ideas are starting to brew.
> Why have we not seen an improvements in products? [..] Is everyone just running full speed in circles or something?
The simplest explanation is that they don't give a flying flamingo about what you or I consider "improvements to products".
This report is an example.
There are several changes that modify CI behaviour, where the article gives no corresponding quality measurement.
They replaced type aware custom lint rules with AST-only static analysis. They don't say anything about what those new rules detect, didn't do old-vs-new rule comparison. They switched the TypeScript check from tsc to tsgo. Again, they are very proud of the performance improvement, but don't seem to care about diagnostic equivalence. The list goes on. They don't even report pass/fail agreement between the old and the new CI. They have 4x more tests, but no idea whether this big test suite works any better than the smaller old one, or even whether it works at all.
Are they really adding 2,000 tests a week to their codebase?
AI is a rocket motor.
Doesn't matter where we're going, as long as we get there FAST!
It's a sickness.
The same reason it took many years for corperate america to get a real productivity increase from computers and the internet, all the old ways of doing things had to be redone. I think the largest companies are least equipped to take advantage of AI productivity gains. Agile no longer makes sense, Org charts as no longer make sense, etc...
I've been bootstrapping for two years now, just me and AI tooling.
Code moves much faster, product taste doesn't. Once you have your marching orders you can make features 10x faster. Shots off target still miss though. It doesn't matter if your code gets generated 10x faster when it doesn't resonate with users.
In my experience everyone is just rebuilding the same wheel over and over.
A lot of people may be more empowered to create things now with less up front effort but it doesn’t lead to having better ideas or more actual system architects.
Why do you assume "we not seen an improvements in products"? Who is "we"? Have you done an exhaustive (or even half-assed) analysis or is this just your "vibe"?
>Why do you assume "we not seen an improvements in products"? Who is "we"?
Have you? As in, can you name products you use daily that have been transformed night and day?
I am not talking about the existence of minor improvement, mind you. I am denying the existence of anything that is minimally consistent with the level of discourse that surrounds development itself (throwing away practices, checks and balances, hitting constant roadblocks due to the new speed, etc).
A true 100x requires no analysis. If we could suddenly build and improve houses at 100x I would know because I would be typing this from my 5th living room.
The new iOS has actually increased the quality for the first time in years. There are open source projects now that tackle problems nobody has before.
It’s not night and day (as you’d expect from the hype), but I see some differences.
> The new iOS has actually increased the quality for the first time in years.
In what ways? When did previous releases stop meeting your criteria?
> There are open source projects now that tackle problems nobody has before.
Such as?
> When did previous releases stop meeting your criteria?
Ages ago. It was really noticeable when they prioritized features over quality.
> Such as?
HN is full of them. I don’t have great have specific example at the top of my head, but lots of hard reverse engineering/low level projects credit AI and that they would never have happened without it these days.
AI doesn’t make a responsive right-click menu a higher business priority. But we are seeing a ton of small custom projects that are as easy to dismiss as they are to abandon.
Surely it makes the backlog clear faster to the point where you reach the non-priority stuff? But I'll bite, what business priorities are being created at 100x?
All of the work work at my company has been internal facing. People are rightly leery of exposing things made by an LLM to the public.
The real question is why are people too lazy to spin up their own phorge. It's a form of learned helplessness supported by a supposed business case to prefer buying services rather than bootstrapping your organization with proven technology.
I've definitely noticing buggier and buggier software, that's for sure. Even our internal tooling and CI pipelines and all that kinda stuff has taken nosedive thanks to the deluge of garbage
incentives are not aligned to improving products.
the important thing to microsoft is still finding new ways to make money, not to make your clicks faster. they might even be using their new velocity to make clicks slower with an ad in between.
there's alao much less incentive to build things to share. people can just make their own widget app and not need yours
Well you mention Android, iPhone, Nintendo, and Microsoft. Those are the biggest companies, and many of them don't even use the best coding tools. So they are not going to be the ones improving.
On the other hand, look at the YC companies. I think the last batch or two is growing revenue faster than any batches ever before. Those are the companies that are being sped up a huge amount by AI.
It’s hard to imagine a company more all in on ai than Google. The pressure to use it is intense and they are throwing ai at basically every problem. If a proposal doesn’t have ai in the title somewhere it has no chance of funding.
I don’t really think it has made Google products better, but it is absolutely the expectation.
well people are spending tokens like crazy for sure, Anthropic and all other labs and everything related to AI seem to be making tons of money.
On a serious note, it might take a while to realize the actual benefits or losses. Its clearly not a good signal when people whose job is to manage other people start writing their own pet AI projects. This just indicates that AI has created this big job insecurity among everyone. At the end of the day being an engineer, eventually the job is relatively safer when there is so much code being written out there.
I've been noticing much faster UX changes in the Youtube for Android App.
I don't know if it's AI or not, and I haven't liked all the changes, but there have been more of them than I remember before.
The problem, as ever, is figuring what's worth doing and what's not. LLMs are not as useful there.
> Switch 2 remains unhacked
What is Switch 2 security doing here? (independently of it having so few features added per update)
AI makes a lot of drudgery type coding tasks a lot easier (think code migrations, etc.)
But actually coming up with, testing, and rolling out loved new user features. AI isn't that good at that and we can't really "prompt" that out of it as easily as we can prompt a py2 to py3 conversion of old code
We 100% have. I've seen significant improvements in many of the services I use.
Drop a couple names, I'd like to know who is leveraging AI well
Not doubting you, care to share what those improvements were?
I mean, looking at the release notes for things like say the Linux kernel it is clear that the improvements are there at a massive rate of change. But it's small stuff, it's performance improvements, it's quality of life. If you do things right people will think you've done nothing at all. What did you expect? Businesses to come up with whole new business lines, or is it just that backlogs of work is getting burned down that was going to be gotten to eventually? From what I can tell from the PM forums, the next bottleneck is the Product Org coming up with ideas worth implementing as fast as the developers can deliver them.
The improvement isn't happening because company are also actively firing people in name of layoff and aren't actively hiring.
In my case it's not the CI that's the bottleneck. It's the human testing side. Does it work, sure. But does it actually do the thing we want (and more importantly) does it do it in a way our customers will understand and actually like?
I feel no one talks about this, and yet it is glaringly obvious. Are we even solving the right problem? No one cares. Push code. Number go up.
I feel like if anything LLMs have reduced coding/software to "throw as much as possible at the wall and see what sticks". It feels quite shortsighted and wasteful, especially considering that humanity needs to get better about how it produces and consumes energy. It's sort of bleak.
Which is a bit terrifying and removes the ‘engineering’ component of a critical part of the job, in exchange for slot machine / pray to ai gods…which comes with disconnect from the code, something I assume will adversely compound over time.
hasn't improved software either. it's made the overall quality worse if anything.
to me the last part if the most important part. product management hasn't been automated at all, and is more important than ever. you can't just keep adding features to make a great product
Think of it as a layered problem. If the bottom layer (CI) cannot keep up with the output of agents, then solving problems at a higher layer - like user experience checks - will be exponentially slower and less reliable. Kind of like how optimizing tight inner loops makes your whole program faster.
It's true because it usually makes no sense to QA test a build where the test suite has not yet passed.
Uh, no. If stage X is the bottleneck, it's immaterial how much you speed up stage Y.
You're assuming QA reviews won't be fully automated, and triggered from a CI pipeline.
QA is not automatable. QA is about finding the things that you didn't think of and so didn't write a test for it. That many people think qa is not important shows in the bad software we have.
We can assume lots of things, but at some point the rubber must hit the road.
What exactly are you using to back up this claim?
> QA reviews [...] fully automated
Clearly you have to replace obsolete human testers with agentic AI testers, duh.
At some point, with all this velocity, human users become the bottleneck, unable to keep up with and learn all the changes and new features. Luckily, there's a simple solution: just replace the human users with agentic AI users.
Maybe we can also replace the customers with agentic consumers.
> The forceful executive, Henry Ford II, and the leader of the automobile workers union, Walter Reuther, both saw many examples of advanced machinery operating at the plant. The words they exchanged brilliantly encapsulated the paradox of automation:
> Henry Ford II: Walter, how are you going to get those robots to pay your union dues?
> Walter Reuther: Henry, how are you going to get them to buy your cars?
https://quoteinvestigator.com/2011/11/16/robots-buy-cars/
I think you're being sarcastic, but there is actual truth behind what you're saying.
Because every developer is now a slop cannon by default, by default users will experience churn and whiplash, and things will break all over the place. As you point out, this is bad. It's also impossible to fix without deploying agents on the QA side. Like it or hate it, agentic testing is inevitable to protect users from the churn and noise caused by the slop cannon. I don't think that replaces test engineers at all - if anything it makes the job more fun. If you've ever had to keep playwright tests in sync with the target manually, and kept the CI environment up to speed with toolchain changes, you know what I mean.
Whether the "slop cannon by default" situation could have been avoided in the first place, is another question... But we're here now and there's no going back. Might as well deal with it as best as we can.
TLDR: it's not all bad :)
> Moving our workloads off GitHub Actions to third-party runners with faster CPUs, higher-performance storage, and better cache infrastructure gave us faster machines to run the same pipeline on
Yeah, was not surprised to read this. Actions is convenient if you already use GitHub, but it can also be pretty slow. Given reliability is also a major issue with GitHub these days I expect to see more orgs moving to different pipelines
I wonder how much faster GitHub actions could be if they weren’t running on Azure. Because Azure is either slow or very expensive.
It's not Azure as much as self-hosted runners are basement bin servers on clearly massively oversubscribed machines.
I think its actually because GitHub isn't solely running on Azure. They are currently using a mix of on prem, AWS, and Azure. The Azure migration has been a challenge with Azure running into capacity issues due to AI load.
We switched our Actions workload to blacksmith.sh (not affiliated) and have been pretty happy with how fast and inexpensive they are. I wouldn't be surprised to see this trend continue.
Good thing Linear has been finished for a long time doesn't need more features, so AI coding can go slow. Oh damn, it's busy becoming the next Jira :(
I’m conflicted about Linear’s progression. I dislike some of the features but on the whole they’ve managed to keep the software pleasant to use, it doesn’t feel to me that it is drifting towards Jira territory, rather, it feels like it is losing the carefully considered product design because now code is cheap to generate. I’m not worried about it turning into Jira but it has lost its soul. Still a great product.
I haven't used Linear in any serious capacity, but I feel like there's plenty of market in "Jira that doesn't feel like it hates its users" and "Jira but we care about performance".
Not saying that they're shooting for either of those segments, but someone should. Plenty of enterprise orgs that need (or are convinced they need) Jira's featureset.
All software becomes popular because it's not the old behemoth, but it's new, fast, and simple. All software then grows and grows until it becomes the new behemoth, and the circle of life begins anew.
My biggest takeaway (as a one man band 20 leagues below this spending time optimizing Gitlab free plan and self hosted runners): they did not worry about any of this until 100M ARR and more 1B valuation[0].
[0] https://linear.app/now/sharing-growth-with-the-people-buildi...
If you can pay the setup cost, Bazel will get you build times ~10s with a warm cache for even massive projects.
I lead a bazel conversion for a pretty complex piece of software written in 5+ programming languages and shipping native binaries to all 3 major OSes a few years ago, and it took multiple years to get it done.
For a less complex project (1 programming language, still shipping to all 3 major OSes), with my knowledge and agents I got the bazel conversion done in 2 weeks.
The setup cost for bazel just went down by a lot, and I don't think the industry as a whole is aware of that yet.
Were any of those using JS/TS? I've seen Bazel perform wonders with many compiled languages, but I'm not impressed with the JS ecosystem.
Nothing substantial, no. I've never personally experienced builds to be so slow to begin investigating bazel as an option for JS/TS.
Tsgo, oxlint, caching dependencies etc. what linear outlined in their blog post would be more impactful for the average TS project I've worked on.
The entire industry, including its outputs that LLMs are trained on, hasn’t reconsidered what’s easy vs. hard or fast vs. slow. LLMs consistently recommend against code changes because they will take “a weekend”. No, Claude. You will do the work and it will take 20 minutes.
Can't comment on Bazel specifically, but having worked with both nx and turbo, the bottleneck was usually network and disk IOPS rarely compute.
Even fully cached outputs needs to fetched and read from a remote server[1]. A step n-1 outout fetched from remote cache server need to written to disk and then again read by step n[3] - all disk I/O and network bound operations.
10s may be achievable/realistic goal in the Java/C++ world where Bazel normally seen. In TS eco-system most people would be over the moon to get into ballpark of 1-2m for a decently large monorepo.
We should define Build more clearly here, if you mean running just transpile/compile steps or the full series of steps that includes tests (as the linear post here is talking about). It is hard to see even a small sub-set of a large suite of test that require a virtual DOM or a real browser can run in 10s or less.
[1] Typical for say managed CI setup .
[3] Common run-of-the-mill frontend + backend stacks in different languages etc.
If you don't need to rebuild anything, bazel can fetch only the final artifact (not the intermediates) from the remote cache.
Also, if you have persistent CI workers with a persistent bazel instance, you save on some network roundtrips, but that's obviously harder to set up and make bulletproof.
Typically you need the intermediates to compute if you need the next one , so you cannot skip to final until you have the intermediaries.
The final asset/artifact is rarely small either. even best optimized artifacts can be few hundred MB docker image or more commonly multiple image layers running GBs in size .
each step is a network pull then recompute cache if stale and keep going till end .
Bazel can even skip the final artefact!
well for test runs you kinda need the binary to run, but you're correct, if the job is just "does this build" no download necessary.
Bazel seems to have a lot of tradeoffs, from setup time of the sandbox for each task, to ergonomics that lead folks to maintain parallel 'normal' tooling.
Plus, 'with a warm cache' is doing heavy lifting, what's the real cache hit rate for a week of development? Investing in improving the cold build and frequent actions is still important with bazel or any incremental builder.
I'm not sure it's useful to talk about bazel broadly, it's actual performance and behavior comes down to the rules you use. You can configure bazel like turbo/nx and cache tsc/vitest/eslint on each package.json module, and get course cached units that are evicted on every change, or you can use gazelle and target per-file actions which are only invalidated when their dependencies change. But that trades off batching unless you use workers.
Most PRs only touch a handful of targets so cache hits are extremely high in practice.
Depends on which targets and how granular the caching is. - if you touch package.json that might invalidate everything - touch core and you'll invalidate everything - touch one file in api and you may run all api tests (see gazelle)
I ran an experiment where I migrated a package to bazel then replayed a weeks worth of changes and it saved 20%. That's nothing to scoff at, but not the headline numbers you see after a full hot build.
With the widespread increase in speed/commit cadence/content added to repos, I wonder if there's an angle here for companies with a lot of CI/CD needs to simply start self-hosting their CI/CD machines on premises.
In my experience, self hosting your CI/CD runners had the biggest impact in cost savings throughout, while also allowing for more powerful machines, which directly means quicker CI/CD runs
My take: it seems like systems should become smaller, more isolated, and contract-oriented.
I have been a long time proponent of monoliths, but it seems like agents would be happier with smaller, more isolated services. The more isolated, the better. Contracts between the service components only. Then it can iterate internally as long as it satisfies the contract. If it needs to, it can version the contract and keep iterating.
Microservices are still bad. You want either a modular monolith or FaaS with most of the work in a domain library.
None of them are bad. It's like saying a bike vs a scooter vs a car is bad. Use them correctly and for what they're intended and they work fine. Use them improperly for the wrong things and suddenly people think the tools suck, when it's the humans who misused them that suck.
I think to solve this problem properly, we have to stop treating "build" and "test" as separate buckets of work, to be designed and scaled separately. It's all CI. As soon as you try seriously scaling out tests, you will run into build bottlenecks. To truly scale CI you need a scheduler that understands your build, test environment, and all the glue in between, well enough to schedule it in a way that actually speeds things up. That is very difficult and not something that even the best build tools can do - yes, even Bazel. Bazel can run tests but it's not nearly as good at it than at building.
Agentic Coding has been a huge strain on CI, I have been using Bazel to improve our build times and ultimately building customized runners to improve our CI
Anyways great blog post from linear team a lot to learn from it
This still surprises me.
Mostly around, at least the majority of 'stuff' I've worked on (both before and after the rise of coding agents) myself or others took enough time to make sure that anything done locally, if you run the tests locally, you're at least 90% of the way there as far as what CI/CD does.
I suppose the flipside being, most of those projects had less churn (i.e. one person was working on a service at a time, and we had good contracts between services.) Also, Our local boxes were way better than our CI boxes, so there was incentive to run locally versus waiting 2-10x the time for CI to run...
I suspect a substantial part of this is an avalanche of useless testing.
If you even review PRs still: when was the last time you didn't just skip over tests? And if you ever looked at tests in an LLM-heavy PR, how many of those tests tested something useful, and not just built-ins and trivial behaviours?
There's at least some awareness in the industry of how LLMs generate a lot of boilerplate in business logic. It feels like we're much less aware of how much of it is in tests.
I call this moment "the Great CI Bottleneck of 2026". Here are my thoughts on it: https://dagger.io/blog/the-great-ci-bottleneck-of-2026/
Our CI has thousands of tests and is done in 2 minutes. Everytime it grows above 3 I add more parallelization and keep it ultra fast. Been working great for our team
I wonder if the GitHub actions outages we keep seeing is due to themselves making self-hosted runners paid, hence bringing broke/cheap users back from hosted runners to their garbage infra.Meanwhile I'm running my Codeberg Actions on the free Oracle ARM machine 2 cores 12GB ram (previously 4 cores 24GB) and way more reliable. Keep winning bozos.
Edit: After second thought, I guess "alternative runner" providers still have to pay the self-hosted tax. So M$ actually saves and makes money by not scaling their infra and driving people to alternative providers they can tax freely. Actual geniuses.
They abandoned (or at least postponed) the idea of charging for self-hosted runners usage.
Actions management plane is the most problematic part of the platform (in regards to scale), so they are basically losing money with self-hosted runners.
Scaling of VMs shouldn't be a problem, since that is practically stateless, whereas there are lots of operations performed for every job/workflow.
vitest load balancing on test duration times instead of by file would be a nice performance win
We actually tried this and (for us) it doesn't lead to enough of a gain that's worth the extra complexity.
We are working on something here, pushgate.dev. It forces the agent to run tests; we are also working on adding support for static analysis of bundles using Jev. Onboarding is a bit of a mess right now, but send me a message cole@testifysec.com if you are interested.
Speed up CI and the next bottleneck just moves to deploy and rollback, which do not scale the same way.
You are right, but one problem at a time :)
Heh. I fixed my CI/CD flows last year to be no more than 60 seconds: https://pics.ealex.net/share/2IpZSt2cxrKWzeUokNNmk4eRI1jjzF2...
And this includes _full_ linting and unit tests.
I did that by self-hosting Github Runners, using Podman to build images (it doesn't squirt your build context through a socket every time!), and using Docker images as content-addressable cache.
I will do a write-up about it this week...
> Despite our test suites almost quadrupling since the start of the year
I think the SDLC needs to be re-evaluated, tests, specifically unit tests, are just cosmetics. "wow its so easy to get high coverage now!" as opposed to why we're doing it in the first place
I don't find agents to be using tests any differently than a junior or mid level developer. I don't find humans to be using tests any differently than a junior or mid level developer either.
Basically, the tests are never guiding the features, they aren't highlighting regressions, the test is simply modified for the updated application. Unit tests especially.
I have found end to end tests to be useful, and also unblocked. They can tell when the actual user experience has changed, and go back to the feature's implementation to stop altering the user experience in unexpected ways.
additionally, I find this across every industry that was worried about AI. basically, whatever best practice was neglected due to lack of investment into your org is now being done without additional investment into your org. that's a good thing. now we need to look at the purpose of the best practice, and if that purpose itself was already solved in the process.
I just feel apps like linear are increasingly getting in the way of full send agentic development where sub agent orchestration is done through agent to agent messaging, work trees, on demand git restructuring and epoch specific coordination plains, often .md files. The smaller the human component of total product development gets, the more this may be the case.
Question here:
How do you expect to see the history/record of what the agents did and why? Is it enough to see it in PRs? Do you expect tickets that have the design and history? How are you thinking of agents being able to historically resolve reasoning/why/decisions made in earlier passes?Genuine open question here. My assumption is that a GH or Linear or Jira is still useful as a decision store. It may as well be a custom app over Postgres, but it seems like something is needed to store this and for observability. A GH/Linear/Jira is nice if only because of standard APIs and integration points (whatever you build would likely end up duplicating a subset of those).
Linear is a key part of my prompting technique. I write specifications in Linear, and the history I generate in it becomes a critical source of context for my agents. Prompting my agents has become “look at XYZ-124 in Linear and ask me any questions you have.”
My recent winnowwallet.com still has build times of over 10 minutes. I'm convinced the agent really wants build times around 5m to move at a quick pace. Also to not drive me insane. It took a ton of work to get it down from 45 minutes because my tests launch a full version of the app, and walk it through major usecases while recording video and screenshots. I then use AI to qa this. It also generates its webpage this way. All from CI/CD
I cannot believe we are in 2026 and CI/CD hasn't evolved enough to even consider hot updates. Burn your CI/CD pipelines to the ground and start over again without any of the slop. Don't let anyone who calls themselves DevOps Engineers design it again, only people who are System Administrators.
(Disclosure: I'm a cofounder of RWX) For anybody wanting to solve similar problems and considering Bazel, take a look RWX. It's built around the same concepts of content-based caching and graph-based task execution, but it's far more runtime agnostic and easier to adopt. https://rwx.com
Maybe I'm alone in this, but as someone who is in tech, I don't know what CI is, and I don't think it's unreasonable to expect it to have an expansion within the article the first time you use it...
AI is in the cultural zeitgeist, but you gotta expand most other things at least once.
So if you don't know what CI is, what does "I am someone in tech", mean? Marques Brownlee probably doesn't know CI is, but he's also in tech. Because if you by mean "I am in tech" that "I am a software engineer", and you don't know what CI is, then boy; I'd be worried for you.
CI is really only significant on the programming side of things. Networking, systems administration, hardware, etc often don’t have that workflow.
sysadmins CI is hoping the machine still boots after an update. Unless you are lucky you got two identical machines to have a test env.
Also the case of 'learning that someone did updates directly on AWS console instead of terraform and losing 0.5 or more days cleaning up the resulting mess'
I see you have not discovered infrastructure-as-code (IaC). We have CI for our infra.
I’m familiar, the issue is IaC only works on a subset of system administration jobs.
Maintaining systems in a tunnel boring machine is a very different than an internet startup.
I am a Site Reliability Engineer, I have been for close to a decade, and this is not as common an abbreviation as you think it is. Also, my entire job isn't in question because I didn't memorize shorthand for a term that I don't use in my day-to-day.
Really appreciate getting downvoted to hell for saying "We can't throw readers a simple bone?"
All I'm asking for is CI(Continuous Integration) in the first paragraph.
What do you think about this? https://claude.ai/share/54eef6b8-b702-4f3e-968c-b19184e951f4
I didn't say I don't know what Continuous Integration was. I said I didn't know what "CI" stood for.
Cannot stress enough, this is like, basic communication/writing that they teach in elementary school.
But I'm so happy you're able to get off on your feelings of superiority that you always automatically know what those two letters stand for. I hope that translates to meaningful happiness and fulfillment in your life.
If expecting more from others is what gives someone a sense of superiority, then what is that you're doing with your expectations?
Next time, Google it or ask ChatGPT. That's what elementary kids do when they don't know something.
"AI is in the cultural zeitgeist"
Within the engineering tech sphere, CI/CD have also been terms that have been standard for at least a decade now.
You would probably be rejected from most interviews at the first stage if you didn't vaguely know what they mean at this point.
Linears entire product is tailored towards software engineers/engineers in general or people who work alongside engineers, so its not surprise their posts have a bit of assumed knowledge.
I know what a CD (Compact Disk) is.
(Continuous Integration, and Continuous Deployment, are fine if someone tells me what the letters literally stand for, but sorry for not being so tech-brained that I haven't memorized every possible TLA that's ever been used in tech).
> haven’t memorized every possible TLA
TLA or TLA+?
CI -> Continuous Integration
Its best thought of as the testing systems that are run as part of pull request review / merge to main / build processes.
Thank you
Man, what a weird thing to complain about.
Most days I feel like I must be the dumbest person on HN. I don't understand what 80%+ of submissions are about. But if it sounds interesting, I'll dig into it a bit and learn a few things along the way.
I recommend making a habit of googling or asking AI about terms you run into. As someone who is in tech, you will hear tech jargon your whole career that will not be explained if it's considered standard terminology for the audience being spoken to. It is better to learn to educate yourself than expect others to go out of their way.
It's wild to me that people think asking for
"Continuous Integration (CI)"
in the first line of a paragraph of a long article is "going out of your way". It used to be the standard expectation for using shorthand.
It used to be a standard expectation, but that was before google or AI could have answered your question in less time and typing that your complaint on HN required. Especially when the author is writing a blog post for an audience that knows what CI is.
Do you want to be right, or do you want to learn?