People want to dunk on GitHub, but honestly going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system anyone has ever built here. Even if it was over the span of ~8 months, many things will fall over and then as you patch those the next domino will fall.
I feel bad for the team, but will be great growth opportunities for all.
> going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system anyone has ever built here
This is likely a pricing failure. Systems breaking due to over-usage is like going to the store and finding that they're sold out. You're much less likely to sell out if you charge more. Meanwhile, GitHub not only didn't raise prices, they cut them at the beginning of the year: https://github.com/resources/insights/2026-pricing-changes-f...
Bear in mind as well that the Pro tier includes 3,000 minutes/month and the Enterprise tier ($21/seat/month, and no published minimum seat floors) includes 50,000 minutes/month. $21/50,000 minutes/month for a solopreneur is practically giving away the farm.
GitHub needs to drastically increase pricing, at least for Actions; their current pricing is practically a WTO dumping violation.
It makes sense but in a post AI world - what is of value? Not the software itself, but data and network effects. MS needs to move from a world where their proprietary software makes money, to one where they enable a captive network and have data; Github provides both.
So maybe them providing service at literally all costs to themselves is a smart strategy long term.
I don't see the connection. Pricing may effect their long term prospects, but I don't see any evidence these issues are being caused by a shortage of metered / allocated resources. They should have access to as much Azure compute as necessary.
> They should have access to as much Azure compute as necessary.
"Should" doesn't imply "does". GitHub is competing with all of the other Microsoft products that need Azure compute, in addition to Microsoft customers. If a customer is willing to buy the compute at a higher margin than the internal chargeback rate, it's definitely not a given that GitHub is getting that compute over the customer. It's hunger games for compute everywhere right now.
Sure, that is possible, but there's still no evidence that compute scarcity is the issue here. It seems a stretch to assume it must be the pricing model when there are a multitude of challenges presented by this kind of ramp.
They wanted direct access to the repositories for other purposes than serving the customer. Such as training data for copilot, and who knows what else.
Problem is while we have a visceral feel for what the "value" of GitHub is to human users for human usage, I'm not sure we have a feel for what the value of the agentic equivalent is.
IMO it's probably very high on paper if you think through what kind of work is unlocked at agent speeds, provided you have a very capable work-execution fabric.
But since we're anchored to human pricing, whatever they propose will feel expensive.
2 billion per day is 23,148 per second on average. Perhaps peak bursts are >100K per second. That's a seriously insane number to keep up with at GitHub's scale. I've worked on systems that write >100K things per second and when it has to be globally distributed and relatively consistent within seconds, that's a very tough challenge.
I was very excited about this comment until I got to this part:
> That's a seriously insane number to keep up with at GitHub's scale.
What exactly is it about GitHub's workload that makes us feel like the specific 23k things happening in that slice of one second are tantamount to insanity? In my experience, much of GitHub activity is changing labels on issues and posting plaintext comments.
Technically speaking, this rate of things happening per second is well within the territory of what one large server can handle. I know it sounds wild to imply that we could run all of the GitHub public cloud off a single machine, but how crazy is it in reality when we start to consider the actual I/O bandwidth that is available in a leading edge server platform? All of VisaNet circa 2009 could run off a single mainframe. That's global. It is plausible they still operate this way.
Virtually 100% of the aspects of GitHub that would quickly saturate I/O on the transactional core can be offloaded to horizontal systems like blob storage and CDNs.
I have difficulty finding what exactly is talked about as 2b per day, but it is not all operations. GitHub saw 473.3 billion events per month lately (https://github.blog/engineering/architecture-optimization/bu...), or ~183K events per second on average. So the 2b number is something much more expensive.
> What exactly is it about GitHub's workload that makes us feel like the specific 23k things happening in that slice of one second are tantamount to insanity?
>
> Technically speaking, this rate of things happening per second is well within the territory *of what one large server can handle*
...for real?
Just as a mental exercise, consider what is happening when a GH action kicks off and everything involved in that.
The amount of write I/O we're talking about here is absolutely massive just on the GH side of it.
A Netflix or Hulu serving videos isn't the same thing because that is a read-heavy workload that works great in CDNs. GitHub has massive amounts of write volume.
It is huge scale, but on the other hand, almost entire GitHub although huge, is made of isolated islands of data (repositories). This is relatively easy to scale. Extreme example would be dedicated server for each repo and its Actions with their own isolated state.
There has to be some economic incentive in place if their scaling issues impact the entire platform. If their system can't handle it then they need a toll system for these agents until the scaling is in place.
They probably had a redistribution of free-tier and paid-tier usage in the last year-ish.
Just backseat gaming but a lot of those issues could probably be solved if they'd (temporarly) reduce the free tier. I'd say simply dropping Actions from it would reduce the load by a lot because then most vibe coders wouldn't bother integrating a CI/CD and either just use local pre-commit scripts or stop using those entirely. Not using a CI also drops the requirement for using PRs on solo-dev projects so they might also reduce load there.
I agree. We should also consider cutting residential electricity and plumbing, since that extra water could be used to cool down all the extra power allocated to data centers.
As a fan of analogies, this one is horrible. The magnitude of impact is so different. The impact of the GitHub change would be a mild and fleeting annoyance for those impacted. They just have to prompt a refactor, simple with LLM. It effects the biggest leeches in the GitHub ecosystem, so it’s targeted. The public utility impacts everyone and has no simple alternative or adjustment, people are stuck to suffer through it.
It's not an analogy. I use github actions. Many open source projects use github actions since it's _there_. In exchange, github gains conversions since users of their free tier grow accustomed with the service and recommend it for corporate use as well.
This system worked pre-LLM and turned github into the de facto _hub_ for software development. It was so good that microsoft, a company previously known for its "developers, developers, developers" stance decided to spend a cool bill on it.
The proposed suggestion is that we all must make sacrifices for the machine god: Developers shouldn't write code, only correct the machine when it's wrong, mathematicians must read the slop and derive new science from it and regular people must refrain from talking dirty to it so then
The proposed solution was you pay more for what you consume, therefore consumption goes down making the system more stable. This makes sense until you extend it to public utilities and change it to a ration/limit. First, GitHub users care about GitHub stability. Nobody else does. Second, alternatives exist, nobody is forcing GitHub on anyone else. Third, no limitation is proposed just a cost adjustment. All reasons the GitHub arguement is sound.
Majority of public utility consumers don’t give a crap about GitHub or AI. They’d likely tell you to kick rocks if you proposed any type of ration or limitation for your assertion that “we all must make sacrifices for the machine god”. Data center development is getting plenty of community pushback for exactly this reason.
Your train of thought is probably more average amongst GitHub user population but I fear is wildly off average when compared to a general population/the public.
No I don't, it sounded like an analogy - comparison of different things - you refuted that and doubled down on your stance. Why would I infer sarcasm?
> Increasing the cost from zero to nonzero is a limitation, not a cost adjustment.
That is nonsensical. "Increasing the cost ... is not a cost adjustment". It's not a limitation if you're willing to pay the new cost. Thus, not a direct limitation. It's optional based on how much you value the service. Doing the same to water/electric would just be increasing the rates and letting supply/demand economics take over, you are saying people need to be forced to use less without mentioning any cost adjustment... not clear if you know what a cost adjustment means.
It's not an analogy as in, it's what's actually being proposed by some OP that's less invested in this than you are:
"we should remove the previously freely granted ability to use github actions because vibecoders are fucking shit up for other users" is the same thing as "we should appease the [edit: followers of the] machine god", which is the same thing as "we should give up water and power to the machine" and "we should build our websites to be more machine-friendly"
Now please apply the above to your second argument.
Water and power is optional, based on how much you value the service.
That is why you should have inferred sarcasm.
As for the step from zero to nonzero, it is relevant because the service stops being available to those without the means to make payment online, which I believe is the majority of the world by population (sorry, visa and mastercard are not particularly popular outside the western sphere of influence)
They made the judgement call to make private accounts free before the onslaught of AI. Now every coding agent is using GitHub as their file system backend and pushing it to limits no one has heard of.
If Microsoft had brains they would leverage it to become the de facto file system for all ai agents before cursor or anthropic figure it out.
The demand in question is upper leadership's demands to do more with less and do it faster and who cares if it works out worse for a lot of people at any point?
Ultimately every outcome is the result of management decisions. The question you have shallowly dismissed is what decisions GitHub leadership has made to allow these frequent outages.
I don't recall this being an issue for Facebook, YouTube, Google, Instagram... All of which have had much better up-time than GitHub, without the excuses. I don't even like those services or companies, but I have to give the engineering teams props for being professional.
I like GitHub (maybe liked?), but their up-time isn't acceptable and I don't get the "poor little Microsoft/8B acquisition" story either.
GitHub has had garbage uptime since long before the advent of LLM’s. It may be easy to point at the scale and say “no one can do it”, which may be true, but it’s also possible that uptime has never really been a priority at GitHub
Presumably, they still want the free tier to help acquire new customers?
You could say they should separate the human users from the slopsters in any tier. This could be done with some quota/rate limiting model.
But, if they are either on the fence or believing that agent-based development is the way forward, then that new reality is the one they would need to support. Cutting it off will send those customers elsewhere.
And, if you've managed to handle the new workload, it is basically in the noise to also handle artisanal human developers in the background on the same solution...
When your business model is to provide valuable thing for free to corner out competition out of business, don’t be surprised that your service gets to be used asymmetrically.
> going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system
Okay, and?
GitHub chose this. They explicitly market it as a pro-AI platform. They literally invited this giant self-DDoS! GitHub can trivially stop the deluge any time they want, simply by introducing sensible rate limits.
This isn't some poor early startup getting Slashdotted. What we are seeing is upper management intentionally sacrificing existing paying customers, just to avoid having to be the one to suggest slowing down the AI hypetrain.
GitHub knows that it has a mostly-captive audience. Screwing over your day is a sacrifice they are willing to make to keep Line Going Up.
Allowing that growth was a policy decision, and a bad one at that. They could have acted differently and they deserve all the complaints for not doing so.
When it became apparent that this growth was on the trajectory, instead of doing something to limit it and thus preserving the reliability of the platform, GitHub (or maybe Microsoft) decided to embrace that growth and allow their own platform to fail. They didn’t need to do that. And they shouldn’t have done that. They deserve all the criticisms coming at them for doing this.
You say “they” as there is a single coherent mind in this case. In the large orgs there are a lot of little independent minds doing their little things, barely producing coherent result through shared incentives.
In this case most simple explanation is inaction. Yeah, it was coming, but nobody was exactly in position or cared enough to do something about it.
Yeah, only yesterday I let claude build and deploy me a end to end project that will flicker my lifx light-bulb when commits to any open source project lands. In the morning I saw it stopped around 11 pm PST yesterday. I was wondering why and I was wondering, how easy it is do a complete working project like this in minutes and what is stopping people to do 100s of such project per day. Github wasn't built for this scenario.
Free Github Actions weren't built for this scenario.
Make Github Actions a paid-tier exclusive and this problem goes away.
GitHub are too scared to do that after the backlash around charging people for their own runners, but this would be a very different thing, simply stop giving free actions to people who have never paid them a penny.
> GitHub are too scared to do that after the backlash around charging people for their own runners, but this would be a very different thing, simply stop giving free actions to people who have never paid them a penny.
I had no idea they charged you to run your own runner. I am immediately of the opinion that is wrong.
Conversely, I would have no issues if GHA become a premium product. Or maybe "your actions only run when we have spare compute". I've personally given Github money (back when private repos cost $$) and wouldn't be concerned about doing it again.
They don't charge you for your own runner but they wanted to. The downtime hasn't really impacted me and not really a reason for me to leave github, but making me pay for using my own infrastructure would quickly encourage me to leave github.
> Conversely, I would have no issues if GHA become a premium product.
This is a real problem for acquiring customers. CI/CD / Actions in general has a low initial perceived value and a high perceived value after use while being primarily a commodity product (serverless compute). To acquire customers they need to offer a free tier, to make money they need to charge their heavy users more, but because it is a commodity their heavy users will self-host runners or jump ship.
Action minutes is a really good proxy for value gain from cloud git services, where repo count, user count, file count/size, etc isn't.
I think a reasonable amount is fine for them to serve as the orchestrator, queue, and control plane for self hosted runners, has to be reasonable though. Think monthly utility service cost to maintain service vs usage.
It isn't just that they wanted to charge for using your own action runners. It's that the price was almost as much as using GH's hosted action runnners. Which didn't even include the cost of hosting your own runners.
I think removing the free tier altogether would be a bad move, especially given that some important open source projects rely on it. But they could possibly lower some of the limits, especially for new accounts, with a way for open source projects to apply for more, and maybe have an automatic scale up for projects that are more popular, by some metric. And maybe limit the free tier to public repos, and maybe even only repos that use an OSI approved license or something.
Likely easier said then done, but why not separate infra out from shared vs dedicated hosts? Something like, actions will still run but at some degraded capacity, where actions and PRs are still at least functional.
My company has heavy handily moved from self-hosted version control system to github and now we’re suffering more outages. Though the self hosted system had less features, and we needed to maintain our own jenkins infrastructure for CI/CD, at least we really only suffered from very major cloud-provider level outages
worse than this is that if you fork a repo that has actions, the actions will run on your account as well - which is completely unnecessary. I have all sorts of stale forks that are just churning away with actions that I dont care about at all because I am not getting charged.
The solution, of course, is not to simply start charging for all actions, because then everyone would wake up to all sorts of mystery charges. They'd need to notify that forked actions would be turned off and then opt-in for paid actions.
Is this really the case? Because GitHub availability has been going downhill since way before models hit that critical mass of quality end of last year.
Why does everything work perfectly for a week after an outage is fixed? Usage is still 30x and even higher because everyone catches up from the downtime.
Sounds more like a bloated, vibe coded stack that has to be unclogged every 2 weeks.
GitHub is responsible for load-testing their systems to find their breaking point, they are responsible for setting appropriate quotas and rate limits, they are responsible for forecasting growth, they are responsible for sourcing capacity to match that growth, and they are responsible for pricing access to that capacity.
If this is some sort of AI-boom related load problem, they are also partly responsible for fanning those flames. I'm skeptical of the AI-boom narrative because other forges are not falling over so frequently; it seems unlikely that all competitors are massively over-provisioned or are not getting hammered by the same claimed boom.
Something doesn't add up and I am not inclined to give GitHub management the benefit of the doubt.
I have not understood, and still do not understand, why everyone locks themselves into github's CI/CD pipeline. It's not hard to set up checks that run on a generic docker setup. I have a home server that I use for all my builds and checks, with a github actions config to run manually if I can't VPN into my home network.
Besides being more reliable, it's just faster to run on a machine you control.
If you're a small/FOSS project, it's not worth the time overhead to set up and maintain and troubleshoot.
If you're a large project, companies generally get more value paying for SaaS and letting devs do dev, rather than troubleshooting Jenkins for the billionth time.
The real value self-host brings is somewhere in the middle, where you need paid-level features with 0 fiat (and can pay in time instead).
It's probably the biggest, bestest, hardestest proof of 'worse is better' anyone could think of. The system is so dumb and inconsistent and simplistic it's actually a selling point: 'everyone in my team knows yaml' is an argument I kept hearing, but this conveniently ignores the fact that a) yaml is a serialization format, not a business logic description language b) the values in the yamls are a cursed mix of shell, whatever js variant it is c) actions themselves are mixes of scripts that accept completely schemaless values, so good luck figuring things out (at least in the pre-LLM era, anyway).
It's like everyone collectively agreed that assembly is bad, yes, but building CI/CD in assembly is actually good.
There’s a great deal to be said for having (inter alia) matrices, job dependencies, and cross-platform runners already working and available. Docker won’t immediately provide those things.
I disabled CI on my part and need to make a separate "ci run" comment to start the CI. I found it increasingly annoying to have CI run "on every commit / push" because I only really need it to be green on the prod-release, not on every commit.
Or rather rate limit CI or something. Free repos that don't use large scale LLM activity to trigger a humongous amount of workload onto the github servers don't have to be caught in the crossfire.
For a small team I setup Forgejo with self hosted CI on Framework desktop. Connect via Tailscale. Runs amazingly well, feels good to have independence from this
I was just debating if I was going to side step actions altogether for a new build and deployment flow.
I'm at a point where I'm going to set two permanent branches on the remote, one pertaining to each environment. The targets will poll for the latest commit at their configured branch and perform a rebuild internally if a change is detected. Assuming git core ops are still alive, it is possible to keep this running. It's also easy to migrate to a different vendor.
How long have you been writing software? This is pretty much how we did it pre-EC2 except it was SVN + S/CP for the tip of the changes. There is a ton of work around this to build on!
Perhaps they need to make actions free and actions plus. Actions free does everything the current product does, but has deferred updates in the UI. And possibly have actions free/plus push updates to your own agentic infrastructure instead. Just configure a URL and all your actions updates go there
My company moved from Azure DevOps to GitHub because of the AI features. ADO was reliable and easy to use, and GitHub has been neither for us. Now Microsoft is adding AI to ADO too, so the main reason for the move is going away :) ADO is much better in my opinion
How long until they increase their plans' pricing and reduce the free Actions minutes to drive bad users away? I would rather have 500 minutes/month of reliable GHA than this.
Given that Github seems to persistently have these scaling challenges, IMO either one of these two things must be true:
1) either they are less competent at running large scale infra vs other large tech companies
2) or git itself adds unique challenges, that make this more difficult to scale
I think 2) is much more likely than 1).
So why stick with git? Is it feasible to make a new VCS that scales better if you start from scratch?
Obviously getting adoption of something else would be a challenge. But Github has enough market-share that they could plausibly pull this off. Particularly using costs or reliability to incentivize migration.
(Not saying that this would necessarily be good for the community in general but I think good for Github.)
Not really hard to understand code is a arbitrary commodity and anyone can grab a cloud model and push code and hook into ci, then have no idea theyre triggering rebuilds everytime they rename their todo apps title.
Basically yes, that's modern day cloud native large scale scalable software. Many smaller software components working together, and sometimes you get a perfect storm of issues or a single point of failure (like DNS) that fails.
I have a noninformed thought on this... I think a lot of these issues are actually because they're not on modern Cloud yet.
I think GitHub was rolling everything themselves when Microsoft bought them. Ever since, MS has been working to strategically shift to Azure but that's a difficult and careful process that takes time.
In the meantime, the AI boom hit and usage went through the roof, crippling aging non-scaled infrastructure that hasn't had a chance to be modernized yet.
A lot of issues might very well be "we broke something while migrating", but I think a lot are also simply capacity issues on legacy.
For someone the size of GitHub, the cloud is not infinite or infinitely scalable, and running their own datacenters is fine and completely normal and not at all "aging" or "crippled". Like the meme says, the cloud is just someone else's computer running in someone else's datacenter.
The traffic increases they cite as the reason are absolutely the reason. Those numbers are insane.
Anything people are building with agentic coding could be called a "house of cards."
It should not surprise people when things start collapsing when there have been little falsehoods and random bits of senseless codes written by agents all over the place.
GitLab CI, BuildKite, Semaphore, Concourse (development started up again, last release was two weeks ago)...
GitHub Actions is only OK for relatively simple CI needs, or if your project is open-source and you enjoy having free CI. No major professional/enterprise CI need should depend on it.
I moved my team from GitHub Actions to RWX and I can't even express how much better it has been. It’s literally 10 times faster, qualitatively different.
So I see an interesting side effect of all of this in your comment. The groupthink seems to conclude that the root of GH's issues are:
- A dramatic increase in its traffic due to vibecoding
- Azure just generally not being very reliable
Now, if you're not a paying customer, maybe you just ignore this and chalk it up to "I'm getting what I paid for". And if you're a paying customer, you migrate away. What happens to GH then? Well, you (and other paying customers) are probably only a small part of the first of the identified causes, and not at all a contributor to the second. So it's not like you moving away will make GH more reliable, but it does lessen MS's motivations for fixing things: GH as a business unit works by having a free tier and a premium tier and that's fine as long as the profit from the premium tier exceeds the cost of the free tier.
THat's long winded way of saying: I wonder if you/other paying customers leaving is the real start of a GH death spiral. Less paying customers == less MS investment == less paying customers == less MS investment, etc.
For us it's not just about reliability, but risk exposure. The last couple of years really changed what our security model looks like. And it's not in public clouds, on SaaS products. LLMs are killing SaaS for sure, not by replacing them but making them a concentrated risk.
Using CI as a gate does not scale. We really think you need to push that testing to the local system and use something like pushgate.dev to validate that the agents actually ran the tests and generated provenance.
You should probably disclose that you own pushgate.dev in your comment
Edit: just to be clear, i'm not saying this to be snarky, I'm saying it because the comment reads like he is a user of pushgate.dev, as opposed to the owner
People want to dunk on GitHub, but honestly going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system anyone has ever built here. Even if it was over the span of ~8 months, many things will fall over and then as you patch those the next domino will fall.
I feel bad for the team, but will be great growth opportunities for all.
> going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system anyone has ever built here
This is likely a pricing failure. Systems breaking due to over-usage is like going to the store and finding that they're sold out. You're much less likely to sell out if you charge more. Meanwhile, GitHub not only didn't raise prices, they cut them at the beginning of the year: https://github.com/resources/insights/2026-pricing-changes-f...
And this is beside them cutting the Pro tier from $7/seat/month to $4/seat/month back in 2020 and not touching the price since: https://docs.github.com/en/get-started/learning-about-github...
Bear in mind as well that the Pro tier includes 3,000 minutes/month and the Enterprise tier ($21/seat/month, and no published minimum seat floors) includes 50,000 minutes/month. $21/50,000 minutes/month for a solopreneur is practically giving away the farm.
GitHub needs to drastically increase pricing, at least for Actions; their current pricing is practically a WTO dumping violation.
It makes sense but in a post AI world - what is of value? Not the software itself, but data and network effects. MS needs to move from a world where their proprietary software makes money, to one where they enable a captive network and have data; Github provides both.
So maybe them providing service at literally all costs to themselves is a smart strategy long term.
Pretty sure compute, storage, and transit are all of value, pre and post AI. At least, pretty sure those things still cost GitHub money.
I don't see the connection. Pricing may effect their long term prospects, but I don't see any evidence these issues are being caused by a shortage of metered / allocated resources. They should have access to as much Azure compute as necessary.
> They should have access to as much Azure compute as necessary.
"Should" doesn't imply "does". GitHub is competing with all of the other Microsoft products that need Azure compute, in addition to Microsoft customers. If a customer is willing to buy the compute at a higher margin than the internal chargeback rate, it's definitely not a given that GitHub is getting that compute over the customer. It's hunger games for compute everywhere right now.
Sure, that is possible, but there's still no evidence that compute scarcity is the issue here. It seems a stretch to assume it must be the pricing model when there are a multitude of challenges presented by this kind of ramp.
https://archive.is/q0cPI
"Microsoft turns to Amazon for help with GitHub's AI-driven capacity issues"
> This is likely a pricing failure.
They wanted direct access to the repositories for other purposes than serving the customer. Such as training data for copilot, and who knows what else.
Pricing low helped ensure access to that data.
> This is likely a pricing failure.
Problem is while we have a visceral feel for what the "value" of GitHub is to human users for human usage, I'm not sure we have a feel for what the value of the agentic equivalent is.
IMO it's probably very high on paper if you think through what kind of work is unlocked at agent speeds, provided you have a very capable work-execution fabric.
But since we're anchored to human pricing, whatever they propose will feel expensive.
solopreneurs were fine - they were self-limiting.
slopreneurs are the issue
2 billion per day is 23,148 per second on average. Perhaps peak bursts are >100K per second. That's a seriously insane number to keep up with at GitHub's scale. I've worked on systems that write >100K things per second and when it has to be globally distributed and relatively consistent within seconds, that's a very tough challenge.
I was very excited about this comment until I got to this part:
> That's a seriously insane number to keep up with at GitHub's scale.
What exactly is it about GitHub's workload that makes us feel like the specific 23k things happening in that slice of one second are tantamount to insanity? In my experience, much of GitHub activity is changing labels on issues and posting plaintext comments.
Technically speaking, this rate of things happening per second is well within the territory of what one large server can handle. I know it sounds wild to imply that we could run all of the GitHub public cloud off a single machine, but how crazy is it in reality when we start to consider the actual I/O bandwidth that is available in a leading edge server platform? All of VisaNet circa 2009 could run off a single mainframe. That's global. It is plausible they still operate this way.
Virtually 100% of the aspects of GitHub that would quickly saturate I/O on the transactional core can be offloaded to horizontal systems like blob storage and CDNs.
I have difficulty finding what exactly is talked about as 2b per day, but it is not all operations. GitHub saw 473.3 billion events per month lately (https://github.blog/engineering/architecture-optimization/bu...), or ~183K events per second on average. So the 2b number is something much more expensive.
Just as a mental exercise, consider what is happening when a GH action kicks off and everything involved in that.
The amount of write I/O we're talking about here is absolutely massive just on the GH side of it.
A Netflix or Hulu serving videos isn't the same thing because that is a read-heavy workload that works great in CDNs. GitHub has massive amounts of write volume.
It is huge scale, but on the other hand, almost entire GitHub although huge, is made of isolated islands of data (repositories). This is relatively easy to scale. Extreme example would be dedicated server for each repo and its Actions with their own isolated state.
I’d call it impossible if they weren’t already (mostly) doing it
Impossible is a very strong word. There are core systems at all the major cloud providers with the same constraints handling more requests per second.
There has to be some economic incentive in place if their scaling issues impact the entire platform. If their system can't handle it then they need a toll system for these agents until the scaling is in place.
I'm sure they have KPIs, and I wouldn't be surprised if one of their KPIs is % growth in agentic/AI-driven actions.
Be careful what you wish for, because we might get a toll system for humans instead.
> because we might get a toll system for humans instead.
We all have businesses to run, this downtime costs us money already.
They probably had a redistribution of free-tier and paid-tier usage in the last year-ish.
Just backseat gaming but a lot of those issues could probably be solved if they'd (temporarly) reduce the free tier. I'd say simply dropping Actions from it would reduce the load by a lot because then most vibe coders wouldn't bother integrating a CI/CD and either just use local pre-commit scripts or stop using those entirely. Not using a CI also drops the requirement for using PRs on solo-dev projects so they might also reduce load there.
I agree. We should also consider cutting residential electricity and plumbing, since that extra water could be used to cool down all the extra power allocated to data centers.
As a fan of analogies, this one is horrible. The magnitude of impact is so different. The impact of the GitHub change would be a mild and fleeting annoyance for those impacted. They just have to prompt a refactor, simple with LLM. It effects the biggest leeches in the GitHub ecosystem, so it’s targeted. The public utility impacts everyone and has no simple alternative or adjustment, people are stuck to suffer through it.
It's not an analogy. I use github actions. Many open source projects use github actions since it's _there_. In exchange, github gains conversions since users of their free tier grow accustomed with the service and recommend it for corporate use as well.
This system worked pre-LLM and turned github into the de facto _hub_ for software development. It was so good that microsoft, a company previously known for its "developers, developers, developers" stance decided to spend a cool bill on it.
The proposed suggestion is that we all must make sacrifices for the machine god: Developers shouldn't write code, only correct the machine when it's wrong, mathematicians must read the slop and derive new science from it and regular people must refrain from talking dirty to it so then
why the fuck not give it our water and power too?
The proposed solution was you pay more for what you consume, therefore consumption goes down making the system more stable. This makes sense until you extend it to public utilities and change it to a ration/limit. First, GitHub users care about GitHub stability. Nobody else does. Second, alternatives exist, nobody is forcing GitHub on anyone else. Third, no limitation is proposed just a cost adjustment. All reasons the GitHub arguement is sound.
Majority of public utility consumers don’t give a crap about GitHub or AI. They’d likely tell you to kick rocks if you proposed any type of ration or limitation for your assertion that “we all must make sacrifices for the machine god”. Data center development is getting plenty of community pushback for exactly this reason.
Your train of thought is probably more average amongst GitHub user population but I fear is wildly off average when compared to a general population/the public.
You do understand that I was sarcastic in my "suggestion", right?
Increasing the cost from zero to nonzero is a limitation, not a cost adjustment.
No I don't, it sounded like an analogy - comparison of different things - you refuted that and doubled down on your stance. Why would I infer sarcasm?
> Increasing the cost from zero to nonzero is a limitation, not a cost adjustment.
That is nonsensical. "Increasing the cost ... is not a cost adjustment". It's not a limitation if you're willing to pay the new cost. Thus, not a direct limitation. It's optional based on how much you value the service. Doing the same to water/electric would just be increasing the rates and letting supply/demand economics take over, you are saying people need to be forced to use less without mentioning any cost adjustment... not clear if you know what a cost adjustment means.
Dude,
It's not an analogy as in, it's what's actually being proposed by some OP that's less invested in this than you are:
"we should remove the previously freely granted ability to use github actions because vibecoders are fucking shit up for other users" is the same thing as "we should appease the [edit: followers of the] machine god", which is the same thing as "we should give up water and power to the machine" and "we should build our websites to be more machine-friendly"
Now please apply the above to your second argument.
Water and power is optional, based on how much you value the service.
That is why you should have inferred sarcasm.
As for the step from zero to nonzero, it is relevant because the service stops being available to those without the means to make payment online, which I believe is the majority of the world by population (sorry, visa and mastercard are not particularly popular outside the western sphere of influence)
They made the judgement call to make private accounts free before the onslaught of AI. Now every coding agent is using GitHub as their file system backend and pushing it to limits no one has heard of.
If Microsoft had brains they would leverage it to become the de facto file system for all ai agents before cursor or anthropic figure it out.
The problem with that argument is that GitHub was unreliable before vibe coding
I do agree they were unreliable before, I even moved my current org to GitLab because of it.
That being said, they are definitely even less reliable now due to demand.
Do we know it’s demand? It at least seems plausible operations has been forced to adopt AI at the expense of stability.
These two questions are actually the same.
The demand in question is upper leadership's demands to do more with less and do it faster and who cares if it works out worse for a lot of people at any point?
Ultimately every outcome is the result of management decisions. The question you have shallowly dismissed is what decisions GitHub leadership has made to allow these frequent outages.
But shouldn't super human AI be able to fix all the bugs for free and make the service infinitely scalable? That's what I was told at least
I don't recall this being an issue for Facebook, YouTube, Google, Instagram... All of which have had much better up-time than GitHub, without the excuses. I don't even like those services or companies, but I have to give the engineering teams props for being professional.
I like GitHub (maybe liked?), but their up-time isn't acceptable and I don't get the "poor little Microsoft/8B acquisition" story either.
GitHub has had garbage uptime since long before the advent of LLM’s. It may be easy to point at the scale and say “no one can do it”, which may be true, but it’s also possible that uptime has never really been a priority at GitHub
Why not separate systems between paying customers and free tier vibecode slopsters?
Presumably, they still want the free tier to help acquire new customers?
You could say they should separate the human users from the slopsters in any tier. This could be done with some quota/rate limiting model.
But, if they are either on the fence or believing that agent-based development is the way forward, then that new reality is the one they would need to support. Cutting it off will send those customers elsewhere.
And, if you've managed to handle the new workload, it is basically in the noise to also handle artisanal human developers in the background on the same solution...
They have that. I might be wrong, but I don't think they're having these issues with their enterprise offering.
Enterprise is the highest tier of paying tho. Our company is working on the Team plan is pretty much affected by each and every of those outages.
Enterprise is $21/seat/month, could consider upgrading? Not that it should be needed; but it’s not that expensive.
It's not that expensive, yes - but it's more than 5x what we are currently paying for it.
Yes they are.
Source: enterprise cloud customer, it's broken all the time.
There is GHES onprem which indeed doesn't get affected by these outages, but it's shit.
When your business model is to provide valuable thing for free to corner out competition out of business, don’t be surprised that your service gets to be used asymmetrically.
Yes but MS are driving that growth so they should really have done some impact analysis.
AI is being brute forced everywhere. The impact analysis for everyone is, shit will break but we can use AI to fix it.
> going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system
Okay, and?
GitHub chose this. They explicitly market it as a pro-AI platform. They literally invited this giant self-DDoS! GitHub can trivially stop the deluge any time they want, simply by introducing sensible rate limits.
This isn't some poor early startup getting Slashdotted. What we are seeing is upper management intentionally sacrificing existing paying customers, just to avoid having to be the one to suggest slowing down the AI hypetrain.
GitHub knows that it has a mostly-captive audience. Screwing over your day is a sacrifice they are willing to make to keep Line Going Up.
no need to feel bad for the company encouraging slop and breaking the oss ecosystem
They deserve to be dunked on so long as they're the ones that keep begging for people to abuse their platform this way
Allowing that growth was a policy decision, and a bad one at that. They could have acted differently and they deserve all the complaints for not doing so.
When it became apparent that this growth was on the trajectory, instead of doing something to limit it and thus preserving the reliability of the platform, GitHub (or maybe Microsoft) decided to embrace that growth and allow their own platform to fail. They didn’t need to do that. And they shouldn’t have done that. They deserve all the criticisms coming at them for doing this.
You say “they” as there is a single coherent mind in this case. In the large orgs there are a lot of little independent minds doing their little things, barely producing coherent result through shared incentives.
In this case most simple explanation is inaction. Yeah, it was coming, but nobody was exactly in position or cared enough to do something about it.
Yeah, only yesterday I let claude build and deploy me a end to end project that will flicker my lifx light-bulb when commits to any open source project lands. In the morning I saw it stopped around 11 pm PST yesterday. I was wondering why and I was wondering, how easy it is do a complete working project like this in minutes and what is stopping people to do 100s of such project per day. Github wasn't built for this scenario.
Free Github Actions weren't built for this scenario.
Make Github Actions a paid-tier exclusive and this problem goes away.
GitHub are too scared to do that after the backlash around charging people for their own runners, but this would be a very different thing, simply stop giving free actions to people who have never paid them a penny.
> GitHub are too scared to do that after the backlash around charging people for their own runners, but this would be a very different thing, simply stop giving free actions to people who have never paid them a penny.
I had no idea they charged you to run your own runner. I am immediately of the opinion that is wrong.
Conversely, I would have no issues if GHA become a premium product. Or maybe "your actions only run when we have spare compute". I've personally given Github money (back when private repos cost $$) and wouldn't be concerned about doing it again.
That is to say, I think you've nailed it.
They don't charge you for your own runner but they wanted to. The downtime hasn't really impacted me and not really a reason for me to leave github, but making me pay for using my own infrastructure would quickly encourage me to leave github.
> Conversely, I would have no issues if GHA become a premium product.
This is a real problem for acquiring customers. CI/CD / Actions in general has a low initial perceived value and a high perceived value after use while being primarily a commodity product (serverless compute). To acquire customers they need to offer a free tier, to make money they need to charge their heavy users more, but because it is a commodity their heavy users will self-host runners or jump ship.
Action minutes is a really good proxy for value gain from cloud git services, where repo count, user count, file count/size, etc isn't.
Its a fun pricing puzzle.
I think a reasonable amount is fine for them to serve as the orchestrator, queue, and control plane for self hosted runners, has to be reasonable though. Think monthly utility service cost to maintain service vs usage.
It isn't just that they wanted to charge for using your own action runners. It's that the price was almost as much as using GH's hosted action runnners. Which didn't even include the cost of hosting your own runners.
I think removing the free tier altogether would be a bad move, especially given that some important open source projects rely on it. But they could possibly lower some of the limits, especially for new accounts, with a way for open source projects to apply for more, and maybe have an automatic scale up for projects that are more popular, by some metric. And maybe limit the free tier to public repos, and maybe even only repos that use an OSI approved license or something.
Likely easier said then done, but why not separate infra out from shared vs dedicated hosts? Something like, actions will still run but at some degraded capacity, where actions and PRs are still at least functional.
My company has heavy handily moved from self-hosted version control system to github and now we’re suffering more outages. Though the self hosted system had less features, and we needed to maintain our own jenkins infrastructure for CI/CD, at least we really only suffered from very major cloud-provider level outages
> Likely easier said than done
You said it.
And it is also significantly more difficult to retrifit that design onto a running system than to design it like that from the beginning.
worse than this is that if you fork a repo that has actions, the actions will run on your account as well - which is completely unnecessary. I have all sorts of stale forks that are just churning away with actions that I dont care about at all because I am not getting charged.
The solution, of course, is not to simply start charging for all actions, because then everyone would wake up to all sorts of mystery charges. They'd need to notify that forked actions would be turned off and then opt-in for paid actions.
They already have a free tier cap, I believe around 100 action minutes per week or something like that.
(I know because I recently hit it)
Is this really the case? Because GitHub availability has been going downhill since way before models hit that critical mass of quality end of last year.
Why they didn't prompt something like "please make whole system more reliable, don't make mistakes"?
Exactly! Let's keep it light and casual!
Wow I missed the forest for the trees!
This is the...<insert more tokens for this recursively repetitive joke>
These jokes are getting tiresome. They’re not down because they’re using vibecoded software, they’re down because their usage is up 30x year-on-year.
Why does everything work perfectly for a week after an outage is fixed? Usage is still 30x and even higher because everyone catches up from the downtime.
Sounds more like a bloated, vibe coded stack that has to be unclogged every 2 weeks.
Both things are possible. We don't know that vibecoded software doesn't contribute to the problem.
You also don't know that it is either
Yes, but you can't say with confidence "It is not A it is B." It could be A or B, or both, or neither.
From what I understand, it's mostly because Azure is a huge dumpster fire
GitHub is responsible for load-testing their systems to find their breaking point, they are responsible for setting appropriate quotas and rate limits, they are responsible for forecasting growth, they are responsible for sourcing capacity to match that growth, and they are responsible for pricing access to that capacity.
If this is some sort of AI-boom related load problem, they are also partly responsible for fanning those flames. I'm skeptical of the AI-boom narrative because other forges are not falling over so frequently; it seems unlikely that all competitors are massively over-provisioned or are not getting hammered by the same claimed boom.
Something doesn't add up and I am not inclined to give GitHub management the benefit of the doubt.
This sort of supports your point, but aiui many forges have either scaled back free tiers (gitlab) or outright banned sloppy content (codeberg).
I have not understood, and still do not understand, why everyone locks themselves into github's CI/CD pipeline. It's not hard to set up checks that run on a generic docker setup. I have a home server that I use for all my builds and checks, with a github actions config to run manually if I can't VPN into my home network.
Besides being more reliable, it's just faster to run on a machine you control.
One reason is it lets you access Windows build machines for free when you no longer have any around.
I would not underestimate how strategically important that will prove to be for Microsoft moving forward.
Right, the added value of being able to access, build, and publish releases for Windows, macOS, and Linux on BOTH x64 and ARM cannot be overstated
Maintaining Windows and macOS self hosted GH (or GL) runners is a hassle I really don't want to deal with anymore (been there done that).
Because I don’t want to maintain a server and pay for it.
I have a few GitHub actions that have ran for years without issue. You add a yaml file to your repo and it’s done.
As a hobbyist, I’m happy.
I would never do any real work with actions though.
If you're a small/FOSS project, it's not worth the time overhead to set up and maintain and troubleshoot.
If you're a large project, companies generally get more value paying for SaaS and letting devs do dev, rather than troubleshooting Jenkins for the billionth time.
The real value self-host brings is somewhere in the middle, where you need paid-level features with 0 fiat (and can pay in time instead).
Because that model does not scale for teams of developers who need a consistent view of CI health, build health, and PR readiness
It's probably the biggest, bestest, hardestest proof of 'worse is better' anyone could think of. The system is so dumb and inconsistent and simplistic it's actually a selling point: 'everyone in my team knows yaml' is an argument I kept hearing, but this conveniently ignores the fact that a) yaml is a serialization format, not a business logic description language b) the values in the yamls are a cursed mix of shell, whatever js variant it is c) actions themselves are mixes of scripts that accept completely schemaless values, so good luck figuring things out (at least in the pre-LLM era, anyway).
It's like everyone collectively agreed that assembly is bad, yes, but building CI/CD in assembly is actually good.
Even if you still use the actions the CI / CD minutes can still happen on a self hosted machine. https://docs.github.com/en/actions/concepts/runners/self-hos...
There’s a great deal to be said for having (inter alia) matrices, job dependencies, and cross-platform runners already working and available. Docker won’t immediately provide those things.
Forgejo actions with a self-hosted runner works amazing for me.
[dead]
I think it would be more efficient to post on HN when there is a day without GitHub issues.
Well, that's today. Also without actions and pull requests.
you misunderstood the comment you replied to. They were saying any day without issues/problems/failures, not Github Issues.
Thought it was funny, might be wrong about that :P
I thought it was clever!
I disabled CI on my part and need to make a separate "ci run" comment to start the CI. I found it increasingly annoying to have CI run "on every commit / push" because I only really need it to be green on the prod-release, not on every commit.
Why not just have CI on main/master commits?
I do want to check that a PR is green before merging, but I don't want to run CI on every pushed commit
I remember spending a long time looking for the weirdly named workflow_dispatch.
even better, run it locally and automatically update your github checks.
At this point just start charging and get rid of free repos. It's obviously killing the service.
Or rather rate limit CI or something. Free repos that don't use large scale LLM activity to trigger a humongous amount of workload onto the github servers don't have to be caught in the crossfire.
Last time there was an incident like this was 4 days ago.
There are no more excuses to self host. You can now ask Claude to do it for you, given that “coding is solved”.
Or even vibe code your own GitHub at this point.
I'm literally building my own Github right now
https://social.coop/@netopwibby/117418961623523555
> There are no more excuses to self host. You can now ask Claude to do it for you
Can’t tell if the irony is intentional or not
To me, the answer is in the part of the comment you stopped quoting "given that “coding is solved”."
For a small team I setup Forgejo with self hosted CI on Framework desktop. Connect via Tailscale. Runs amazingly well, feels good to have independence from this
I was just debating if I was going to side step actions altogether for a new build and deployment flow.
I'm at a point where I'm going to set two permanent branches on the remote, one pertaining to each environment. The targets will poll for the latest commit at their configured branch and perform a rebuild internally if a change is detected. Assuming git core ops are still alive, it is possible to keep this running. It's also easy to migrate to a different vendor.
How long have you been writing software? This is pretty much how we did it pre-EC2 except it was SVN + S/CP for the tip of the changes. There is a ton of work around this to build on!
Thankfully programming is solved so these are non issues.
Perhaps they need to make actions free and actions plus. Actions free does everything the current product does, but has deferred updates in the UI. And possibly have actions free/plus push updates to your own agentic infrastructure instead. Just configure a URL and all your actions updates go there
My company moved from Azure DevOps to GitHub because of the AI features. ADO was reliable and easy to use, and GitHub has been neither for us. Now Microsoft is adding AI to ADO too, so the main reason for the move is going away :) ADO is much better in my opinion
Whose agents have escaped this time?
How long until they increase their plans' pricing and reduce the free Actions minutes to drive bad users away? I would rather have 500 minutes/month of reliable GHA than this.
Given that Github seems to persistently have these scaling challenges, IMO either one of these two things must be true: 1) either they are less competent at running large scale infra vs other large tech companies 2) or git itself adds unique challenges, that make this more difficult to scale
I think 2) is much more likely than 1).
So why stick with git? Is it feasible to make a new VCS that scales better if you start from scratch?
Obviously getting adoption of something else would be a challenge. But Github has enough market-share that they could plausibly pull this off. Particularly using costs or reliability to incentivize migration.
(Not saying that this would necessarily be good for the community in general but I think good for Github.)
Is github built out of a house of cards? How does this stuff keep happening?
GitHub usually says it's because of the increased demand from AI. I didn't believe it until I saw the numbers: https://github.blog/news-insights/company-news/the-august-17...
Not really hard to understand code is a arbitrary commodity and anyone can grab a cloud model and push code and hook into ci, then have no idea theyre triggering rebuilds everytime they rename their todo apps title.
Basically yes, that's modern day cloud native large scale scalable software. Many smaller software components working together, and sometimes you get a perfect storm of issues or a single point of failure (like DNS) that fails.
Sort of.
It's on Microsoft Azure.
I have a noninformed thought on this... I think a lot of these issues are actually because they're not on modern Cloud yet.
I think GitHub was rolling everything themselves when Microsoft bought them. Ever since, MS has been working to strategically shift to Azure but that's a difficult and careful process that takes time.
In the meantime, the AI boom hit and usage went through the roof, crippling aging non-scaled infrastructure that hasn't had a chance to be modernized yet.
A lot of issues might very well be "we broke something while migrating", but I think a lot are also simply capacity issues on legacy.
None of that is true, sorry.
For someone the size of GitHub, the cloud is not infinite or infinitely scalable, and running their own datacenters is fine and completely normal and not at all "aging" or "crippled". Like the meme says, the cloud is just someone else's computer running in someone else's datacenter.
The traffic increases they cite as the reason are absolutely the reason. Those numbers are insane.
So a house of cards built on a foundation of shifting sand
Anything people are building with agentic coding could be called a "house of cards."
It should not surprise people when things start collapsing when there have been little falsehoods and random bits of senseless codes written by agents all over the place.
At this point they should post status alerts when the service is up, rather than down.
I'm genuinely amazed by anyone still using github CI for anything serious.
Which alternative do you like?
GitLab CI, BuildKite, Semaphore, Concourse (development started up again, last release was two weeks ago)...
GitHub Actions is only OK for relatively simple CI needs, or if your project is open-source and you enjoy having free CI. No major professional/enterprise CI need should depend on it.
I moved my team from GitHub Actions to RWX and I can't even express how much better it has been. It’s literally 10 times faster, qualitatively different.
Yep. We're a huge customer. It hurts. We are making exit plans.
So I see an interesting side effect of all of this in your comment. The groupthink seems to conclude that the root of GH's issues are: - A dramatic increase in its traffic due to vibecoding - Azure just generally not being very reliable
Now, if you're not a paying customer, maybe you just ignore this and chalk it up to "I'm getting what I paid for". And if you're a paying customer, you migrate away. What happens to GH then? Well, you (and other paying customers) are probably only a small part of the first of the identified causes, and not at all a contributor to the second. So it's not like you moving away will make GH more reliable, but it does lessen MS's motivations for fixing things: GH as a business unit works by having a free tier and a premium tier and that's fine as long as the profit from the premium tier exceeds the cost of the free tier.
THat's long winded way of saying: I wonder if you/other paying customers leaving is the real start of a GH death spiral. Less paying customers == less MS investment == less paying customers == less MS investment, etc.
I suspect the same.
For us it's not just about reliability, but risk exposure. The last couple of years really changed what our security model looks like. And it's not in public clouds, on SaaS products. LLMs are killing SaaS for sure, not by replacing them but making them a concentrated risk.
Oh again?
At this point, a status bar color when gh is down would be enough.
I think we can just use the orange status bar for that.
I am using Linear. Some component in Linear showed me the warning. I went to a GitHub PR, there was no warning. Hm.
Using CI as a gate does not scale. We really think you need to push that testing to the local system and use something like pushgate.dev to validate that the agents actually ran the tests and generated provenance.
You should probably disclose that you own pushgate.dev in your comment
Edit: just to be clear, i'm not saying this to be snarky, I'm saying it because the comment reads like he is a user of pushgate.dev, as opposed to the owner
Thank you. Agree, that was not the intention
There must a top 0.05% of users that account for the vast majority of this bs. GitHub should charge them and serve the majority.
Or just punitively rate limit them
just make actions a thing you have to self host and release an action plugin with github desktop
Microsoft should have never been allowed to buy github. A functioning government would have prevented the acquisition.
The entire US economy is tottering due to a lack of anti-trust enforcement.
Even The Economist, an oligarch-friendly magazine, has been hitting the alarm bell on this for over a decade: "America needs a giant dose of competition" https://www.economist.com/briefing/2016/03/26/too-much-of-a-...
The main concern is who should bought Github? Or maybe the founders should opened an IPO or went to the VC route.
[flagged]
doesnt surprise me
[flagged]
[dead]
[flagged]
[flagged]