It doesn't comply with one or more root store policies (which all incorporate the Baseline Requirements by reference, which incorporate various specs, such as RFC5280, by reference).
There are countless examples of non-compliant certificates documented in the Bugzilla component I linked above. A recent example: a certificate which was backdated by more than 48 hours, in violation of section 7.1.2.7 of the Baseline Requirements: https://bugzilla.mozilla.org/show_bug.cgi?id=2016672
Baseline requirements are not an imaginary problem. All of them have a legitimate reason for existing. You could argue that some "are not that big of a deal", but that's exactly the point, the overbearing and overly specific requirements serve both their own purpose and double as Van Halen's "no brown M&Ms" clause: if the CA screws them up, either by malice or incompetence and doesn't immediately catch them and self-report, then you know they have no way of telling what other things they are screwing up. And if you're in the business of selling trust, that instantly makes you untrustworthy.
There are countless Bugzilla reports of clearly unprofessional CAs trying to get away with doing whatever they want, get caught, say "it's no big deal", fail to learn the lesson and eventually get kicked out, much to the chagrin and bewilderment of their management, irate that some nerds on the Internet could ruin their business, failing to understand that following the scripture of the Internet nerds is the #1 requirement of the business they chose to run.
Yes. Brown M&M tests are exactly what's called for here. You want a strong psychological urge to obey rules just because they're rules. There are roles where this isn't the right thing, but operating a Certificate Authority isn't one of them.
In my experience every case in the Web PKI where we found what seems obviously to be either gross incompetence or outright criminality there were also widespread technical failures at the same CA. Principles who aren't obeying the most important rules also invariably don't care about merely technical violations, which are easier to identify.
For example, CrossCert had numerous technical problems to go along with the fact that obviously nobody involved was obeying important rules. I remember at one point asking, so, this paperwork says you issue only for (South) Korea, but, these certs are explicitly not for Korea, so, what technical measure was in place to ensure you didn't issue them and why did it fail? And obviously the answer is they didn't give a shit, they'd probably never read that paperwork after submitting it, they were just assuming it doesn't matter...
"There is an ongoing incident that will force issuance to be halted."
Feels like they were alerted to some current problem severe enough that "turn it off now" was the right move. Breaking the baseline requirements somehow maybe?
People went ballistic on me a few months ago for bringing this up, but this is exactly the kind of outage that makes me really, really worried about extremely short lived certificates. https://news.ycombinator.com/item?id=46118371
I'm not sure I follow. This outage seems like it occurred for less than 1 day. The post you link to is about having certificates expire after 45 days. What's the connection you see?
Some CAs are experimenting with shorter, 7 day certificates as well.
still not an outage that would endanger anyone's ability to renew in time, but for small or extremely shitty CAs (and there are a lot of those) such an outage may take enough time to cause issues in theory I guess?
It doesn't have to be small or more shitty than average. If Google has
a compliance issue and can meet it in 8 hours then its a pretty clear one.
They could have an issue that needs round trips of discussions with
auditors before resuming. etc. I'm not familiar with 24/7 auditor
services.
That's only if you delay renewal until the last day of the lifetime of the certificate. If you renew at day 30 you'd only get in trouble if there's more than two weeks of downtime.
You’re supposed to renew your cert way in advance of the expiration time. For 47-day certs the general expectation is that you renew them monthly, so in the worst case you’d need more than two weeks of CA outage before anything went wrong.
You're joking, but still: that's one very possible outcome of both requiring centrally issued certificates for security reasons and browsers refusing to display websites without.
Effectively certificates are now a license to publish.
Yes I had to, the v20.14.43 I patched a month ago broke just today; but updating it was pretty easy; just have to update[0] and repatch 20.14.43 with an updated GMS patch.
Yeah, if Heroku's cert rotation depends on Google's CA and it tried to renew during the outage window, that'd definitely cause problems. The 8-hour ETA is rough. This is why multi-CA fallback configs exist, but most platforms don't bother until they get burned by something like this. Worth checking if your apps are actually affected or if it's just the dashboard/API having issues.
I worked at RSADSI when I was a kid and supported the custom spin of TIPEM Hayden and Sophia used at Verisign. This brings back some very bad memories.
But... hopefully... people created overlapping windows of cert validity so there's always a valid cert available for their services and can tolerate the CA being out of action for 8(?) hours. Imagine if your TGS/Kerberos or AWS IAM IdP was down for 8 hours.
For persistent services using the affected ACME API, the window is usually 30 days.
But that didn’t stop Youtube and Youtube TV from going down hard. I imagine they’re provisioning ephemeral VMs or service instances and relying on them being able to get certs immediately, or something like that.
It is a well-known fact that the moment YouTube goes down, the collective productivity of Earth increases by approximately 4,000%, which is immediately squandered by everyone going to Hacker News to read comments about YouTube being down. I myself have taken to podcasts… an ancient medium in which people simply talk at you for ninety minutes without a single sponsorship for a mobile game, and this is considered a failure
VPN to Sweden to get the IP geolocated ads to retarget. The ads still exist but they're less obnoxious, and they're often in Swedish so you don't have to know what they're on about anyway.
Careful, I enjoyed this bonus (being in Japan and not being able to keep up with the ads)... so much so, that I started ignoring the Japanese. Including my wife. You can imagine how well that went.
Give it another 10-20 years and your 2 hour podcasts will be 30 minutes of morning zoo DJ banter, 10 minutes of guests, and 1.5 hours of ads.
We’ll have reached peak 90s all over again. With any luck we’ll avoid recreating the conditions for another Nickelback and can stay in the weird zone where Trip Hop and pop punk could chart at the same time.
The 00's podcasts I listened to were often in 2-3 hour episodes, rarely well scripted (or scripted at all?), but a lot of fun and very amateurish. I re-listened to several entire series recently and the episode lengths were the only thing I think was worse than in newer podcasts.
On the other hand, if ads etc gets too annoying, I already have run all my downloaded podcasts through whisper to get transcripts with timestamps. Running some LLM to find ranges to delete would probably be quite easy. As a bonus I would be happy to also cut out all the filler repetitions that seem popular these days ("yes, X, I absolutely agree, [repeats everything X just said]"). Could probably cut 1 hour episodes to 20 minutes without losing any content.
At least it is somewhat relevant. Hearing ads about Irish telecom operator ads at the other side of europe is pretty goofy. What's the actual point? Just worsening the podcast experience?
Well one must also argue the opposite. I myself have gained immense knowledge from YouTube. I have learned things like phone screen replacements or phone battery replacements. I call myself a mechanic from the school of YouTube and have saved myself at minimum $10k in repairs doing the work myself. I have learned to make endless food recipes or create things like giant bubbles or slime for my kids. My point is that I bet sure for some YouTube is a massive time sink waste of time. But I also wonder how much it has improved the knowledge, skills and ability of others.
My dad often mentions how had he had YouTube when he was younger how much it would have done for him. He talks about having to go to the library and if lucky there was a book that could show you the knowledge you were looking for. He says but now you can find not just the knowledge but for example specific knowledge like car make model and year and how exactly to do job xyz.
Ultimately I just can not imagine life without the wealth of knowledge YouTube has given me.
Personally, I just scroll through them. They break the feed into well defined "chapters" at the end of what I can decide to look into the next one or go somewhere else because there's nothing good there today.
Also there's this woman that makes very funny shorts about software development and good long videos that aren't as good. I look for her shorts too.
Lol I laughed out loud reading this comment. When shorts first came out they annoyed me to no end. I searched for how to block them through settings or other ways to just make them go away.
But now days I can admit there are a few, very few, content creators who create shorts that are very informative and straight to the point that can cover a topic and give you many facts and let you decide if you want to seek more. Sometimes it is nice to have the 30 seconds Coles notes verses a video stretched out to 10 minutes to be eligible for monetization.
BUT, and this is a big but, the shorts and similar video platform trends scare me as a parent. I can see how my kids find a 1.5 hour movie boring but can scroll endlessly through shorts. It might seem harmless letting your kid just scroll on YouTube from my perspective is like an addiction and kids are getting that dopamine hit watching a clip and seconds later watching something else. I've learned that it is very important to be aware of what your kids are being accustomed to and push them in the right direction.
I watched a movie, same late night talk show host, something like "welcome night owls".
I "loved" the style but I haven't found any actual radio on the internet of that style or a podcast. Not sure about name of movie but I do remember it being in the last 10-15 years.
"Undrain" is not idiomatic, at least outside of Google. One might drain a tank or creek to empty it, the reverse isn't "undraining" to fill it back up.
"issuance flow has been restored" might be a more widely understood phrasing.
Admittedly, a nitpick, however the tech industry has a tendency to invent new words when they could say the exact same thing in plain English and be better understood by a wider audience.
That feeling when you finally get the timelock safe open and have to do certificate work that shatters YouTube’s connection to the account personalization systems.
Yeah, this could end up as the actual root cause of The Great Oops that I've been raving about for years. And Google probably would be the right company to fuck it up in the worst way possible since Google Knows Best In All Situations.
It's inevitable that one of the major cloud providers will irrecoverably delete all customer data with one single fat-fingered command. Though in google's case I'll also consider the prophecy to be fulfilled if they delete their own data.
There are a few things that can cause tremendously widespread outages, essentially all of them network configuration changes. Actually deleting customer data is dramatically more difficult to the point of impossible - there are so many different services in so many different locations with so many layers of access control. There is no "one command" that can do such a thing - at the scale of a worldwide network of data centers there is no "rm -rf /".
Ah, but you fail to account for Google's incredible knack for building tools designed to do things at scale. Or put AI in things that don't need it.
The possibility Google will either manage to unleash a malicious AI on their infrastructure and/or develop a way to destroy a lot of data at scale quite efficiently or some combination of the two is far from zero.
"We deployed this private cloud with a missing parameter and it wasn't caught" is as different from "we wiped out all customer data" as hello world is from Kubernetes.
No one promised this "should be impossible". Did you confuse "we'll take steps to ensure this never happens again"?
You contend there's no global rm rf for a global cloud provider, but clearly a missing parameter can rm rf a customer in an irrecoverable manner.
The only half you're missing is... how every major cloud outage happens today... a bad configuration update. These companies have hundreds of thousands of servers, but they also use orchestration tools to distribute sets of changes to all of them.
You only need a command to rm rf one box, if you are distributing that command to every box.
Now sure, there are tons of security precautions and checks and such to prevent this! But pretending it's impossible is delusional. People do stupid stuff, at scale, every day.
The most likely scenario is a zero day in an environment necessitating an extremely rapid global rollout, combined with a plain, simple error.
And the most telling thing about most of these outages is that the provider later admits in their postmortem that they just didn't really understand how the system they made worked until it fell over and were forced to learn how it really works.
It's the sort of thing that used to keep me up at night.
This is the thing that gets me about postmortems. The document almost always reveals that the mental model the team had of the system was subtly wrong in some critical way. Not wrong in some dramatic fashion - just a quiet assumption about ordering, or a dependency they forgot existed, or a timeout they assumed would never actually fire.
I've seen this pattern repeatedly: the system works fine for months or years, and the postmortem reads like a mystery novel where the detective discovers the murder weapon was in the room the entire time. The failure mode was always there, just waiting for the right combination of load and timing.
The scariest variant is when the monitoring itself has the same blind spot. You build alerts based on your mental model, so of course the alerts don't fire for the failure mode you didn't know existed. The system fails silently until a human notices something downstream is wrong.
The release process, monitoring checks, etc. for a customer's private cloud is generally significantly different from the release process for a global product. I'm not going to get any more specific for all the standard NDA reasons, but having worked for Google and Microsoft among others....no, the risk you describe doesn't translate from one to the other.
I understand you believe the checks cannot fail that catastrophically, and I agree that the likelihood they do is quite low.
But it can happen, and it only has to happen once. (Also FYI, telling me your work history just tells me you've drunk the koolaid, ain't proof you know more.)
The idea that all customer data will be deleted is far fetched, but I feel like there have been some massive incidents. Crowdstrike comes to mind, but I feel its entirely possible that Apple/Google/etc could push out some kind of config update which bricks phones in a way they are unable to download another update to fix them.
Though I'm sure the major players are all over this risk which is why it hasn't happened.
There's at least five free ACME CAs, with failover it doesn't matter all that much if one of them falls over. If all of them fall over at once there's probably a more pressing issue like nuclear holocaust or alien invasion going on.
For one, Cloudflare uses four different CAs almost interchangeably. Caddy also makes it easy to configure ACME failover if you're self-hosting, and defaults to using two different CAs if you don't specify any.
Frankly even with no CA redundancy, downtime would have to drag on for weeks to actually disrupt renewals. ACME certs usually get rotated after about 2/3rds of their duration has expired, so the upcoming 45 day certs will still have about 15 days of wiggle room.
They aren't all drop in replacements for each other though. For example, Let's Encrypt offers free wildcard certs (with dns verification), but for ZeroSSL, it requires a paid subscription.
ZeroSSL is weird, if you use their classic non-ACME interface then the free tier is indeed limited to 3 active certs which can't be wildcards, but if you use ACME then there's no limits and wildcards are allowed.
> By using ZeroSSL's ACME feature, you will be able to generate an unlimited amount of 90-day SSL certificates at no charge, also supporting multi-domain certificates and wildcards.
So the question is why this hit Youtube and Youtube TV so hard. Presumably they’re relying on ephemeral instances being able to get certs immediately, or something like that.
I was thinking about the time some software influencer said that if you are afraid to deploy on Friday then there's something wrong with you. Eff that! Murphy's Law! (allen holub - https://x.com/allenholub/status/1637111242610610182)
I often deployed on Friday evening. Several factors contributed to this decision.
1. Sales volume was lowest on weekends so if something went wrong it would affect fewer customers.
2. If something went wrong and I needed to revert, nobody was at work on weekends so it would not disrupt coworkers.
3. I always made it so reverting would be easy.
4. Most of my weekends were just relaxing at home, mostly doing online stuff (games, reading, videos) or doing offline stuff at my computer (programming my personal projects). It wasn't much of a bother at all to have an ssh open to something at work monitoring the new deployment for problems for the rest of Friday night and Saturday.
OCSP is deprecated and basically dead at this point. Some clients still use it but I don't think many (any?) have actually enforced OCSP for years since it was notoriously fickle anyways.
Interesting. If you go to youtube.com it's all messed up; missing all the videos in the listings. But if you follow a video embedded in another site to youtube, it'll show and play fine. It'll break if you try to browse away from it.
Yeah, YouTube is not one server, it's hundreds of them. The videos are served mostly from CDNs (the Content Distribution Network). It's a different set of servers than handles account logins, routing, etc.
Some Google Services are also down at the moment, unrelated to YouTube, so probably a failure along some common infrastructure pipeline.
Your History, Subscriptions and search should all work. You should be able to see any creator's page if you go to it directly. The videos are all still watchable. It's primarily the home page and recommended videos that are having issues. Basically any place they recommend videos you haven't seen is broken right now, but the videos are still there and accessible.
I've tried via VPN from the U.S., U.K., Sweden, Germany, Russia, Colombia, etc. Same issue across the board.
Isn't that the thing that a bunch of YouTube creators pitch inside their channels along with VPNs and supplements? I would never consider it because the ads rub me the wrong way. Or is it some alternative frontend for YouTube that happens to have a similar sounding name?
It is a co-op where creators make videos without the threat of being demonetized or algorithmically punished - and it’s not garbage in the way you might expect people fearful of being demonetized might be.
Lots of excellent legal analysis, history, logistics, engineering content there.
It was initially founded by some of the most popular information YouTubers like CGPGrey, but he mysteriously left the project (I suspect one side wanted to be evil and the other side did not)
It's a place for creators to host long form content (that the google algorithm now disincentivizes) as well as history content that can't show a lot of history because of "violence" (like the holocaust).
Youtube is demonetizing channels left, right, and centre.
Oh I am more than happy to tell people how I took down entire Google Cloud 11 years ago. I mean, of course to the level of details Google is comfortable with to share externally :)
The CA outage is hitting a lot of services, but yeah, Heroku's been on a slow decline since the Salesforce acquisition. Free tier killed, pricing creep, stagnant innovation. Even when it's not their fault, you start wondering if it's worth the risk of being on a platform that feels like it's in maintenance mode.
The status history on the page makes it seem like this was intentional?
> 17 Feb 2026 11:32 PST A rollout is going to prevent issuance from occurring. We will provide an estimate on when issuance will stop.
> 17 Feb 2026 12:14 PST Issuance is beginning to stop. A fix to resolve the issue will roll out in about 8 hours
This usually indicates that the CA was issuing non-compliant certificates and needed to prevent further non-compliance. Will be interesting to watch Bugzilla for the incident report: https://bugzilla.mozilla.org/buglist.cgi?product=CA%20Progra...
https://bugzilla.mozilla.org/show_bug.cgi?id=2017747
What qualifies as a non-compliant certificate?
It doesn't comply with one or more root store policies (which all incorporate the Baseline Requirements by reference, which incorporate various specs, such as RFC5280, by reference).
Mozilla root store policy: https://www.mozilla.org/en-US/about/governance/policies/secu...
Chrome root store policy: https://googlechrome.github.io/chromerootprogram/
Apple root store policy: https://www.apple.com/certificateauthority/ca_program.html
Baseline Requirements: https://github.com/cabforum/servercert/blob/main/docs/BR.md
There are countless examples of non-compliant certificates documented in the Bugzilla component I linked above. A recent example: a certificate which was backdated by more than 48 hours, in violation of section 7.1.2.7 of the Baseline Requirements: https://bugzilla.mozilla.org/show_bug.cgi?id=2016672
Something is badly borked when the protections against an imaginary problem cause a real problem.
Baseline requirements are not an imaginary problem. All of them have a legitimate reason for existing. You could argue that some "are not that big of a deal", but that's exactly the point, the overbearing and overly specific requirements serve both their own purpose and double as Van Halen's "no brown M&Ms" clause: if the CA screws them up, either by malice or incompetence and doesn't immediately catch them and self-report, then you know they have no way of telling what other things they are screwing up. And if you're in the business of selling trust, that instantly makes you untrustworthy.
There are countless Bugzilla reports of clearly unprofessional CAs trying to get away with doing whatever they want, get caught, say "it's no big deal", fail to learn the lesson and eventually get kicked out, much to the chagrin and bewilderment of their management, irate that some nerds on the Internet could ruin their business, failing to understand that following the scripture of the Internet nerds is the #1 requirement of the business they chose to run.
Yes. Brown M&M tests are exactly what's called for here. You want a strong psychological urge to obey rules just because they're rules. There are roles where this isn't the right thing, but operating a Certificate Authority isn't one of them.
In my experience every case in the Web PKI where we found what seems obviously to be either gross incompetence or outright criminality there were also widespread technical failures at the same CA. Principles who aren't obeying the most important rules also invariably don't care about merely technical violations, which are easier to identify.
For example, CrossCert had numerous technical problems to go along with the fact that obviously nobody involved was obeying important rules. I remember at one point asking, so, this paperwork says you issue only for (South) Korea, but, these certs are explicitly not for Korea, so, what technical measure was in place to ensure you didn't issue them and why did it fail? And obviously the answer is they didn't give a shit, they'd probably never read that paperwork after submitting it, they were just assuming it doesn't matter...
The heading above that:
"There is an ongoing incident that will force issuance to be halted."
Feels like they were alerted to some current problem severe enough that "turn it off now" was the right move. Breaking the baseline requirements somehow maybe?
People went ballistic on me a few months ago for bringing this up, but this is exactly the kind of outage that makes me really, really worried about extremely short lived certificates. https://news.ycombinator.com/item?id=46118371
I'm not sure I follow. This outage seems like it occurred for less than 1 day. The post you link to is about having certificates expire after 45 days. What's the connection you see?
Some CAs are experimenting with shorter, 7 day certificates as well.
still not an outage that would endanger anyone's ability to renew in time, but for small or extremely shitty CAs (and there are a lot of those) such an outage may take enough time to cause issues in theory I guess?
It doesn't have to be small or more shitty than average. If Google has a compliance issue and can meet it in 8 hours then its a pretty clear one. They could have an issue that needs round trips of discussions with auditors before resuming. etc. I'm not familiar with 24/7 auditor services.
that's roughly 1/45th probable downtime window = 2.22% downtime probability (yeah, it's a figure not a real proba ;-) )
compared to say, roughly 1/365 probable downtime window for a 398 days cert lifetime = 0.25% downtime probability
let's pray you don't need to rotate when it's down...
Dan Geer famously said: "Dependency is the root cause of risk"...
PS: even stricter shortlived durations in some context:
Internal/Private 1 – 7 days Corporate VPNs, Internal apps
Ephemeral 5 mins – 1 hour Docker containers, CI/CD runners
That's only if you delay renewal until the last day of the lifetime of the certificate. If you renew at day 30 you'd only get in trouble if there's more than two weeks of downtime.
You’re supposed to renew your cert way in advance of the expiration time. For 47-day certs the general expectation is that you renew them monthly, so in the worst case you’d need more than two weeks of CA outage before anything went wrong.
You didn't read it or understand it.
You know there’s more than one CA?
but only one browser
Your license to website has been revoked.
You're joking, but still: that's one very possible outcome of both requiring centrally issued certificates for security reasons and browsers refusing to display websites without.
Effectively certificates are now a license to publish.
On a PC we atleast have an out.
On mobile, user certs are pretty much ignored unless opted in by apps. Even firefox allows user certs (for now) but only via an obscure hidden config.
This means we cannot use self-hosted services even using a VPN with official apps without getting a signed cert.
> This means we cannot use self-hosted services even using a VPN with official apps without getting a signed cert.
What do you mean by this? Any service that is designed to be self-hosted will have an app that accepts user-installed CAs. HomeAssistant, for example.
Thought my Revanced patch got outdated for a second. Phew.
Have you had to update microG yet?
Yes I had to, the v20.14.43 I patched a month ago broke just today; but updating it was pretty easy; just have to update[0] and repatch 20.14.43 with an updated GMS patch.
[0] https://github.com/ReVanced/GmsCore/releases/tag/v0.3.13.2.2...
Yeah, if Heroku's cert rotation depends on Google's CA and it tried to renew during the outage window, that'd definitely cause problems. The 8-hour ETA is rough. This is why multi-CA fallback configs exist, but most platforms don't bother until they get burned by something like this. Worth checking if your apps are actually affected or if it's just the dashboard/API having issues.
I worked at RSADSI when I was a kid and supported the custom spin of TIPEM Hayden and Sophia used at Verisign. This brings back some very bad memories.
But... hopefully... people created overlapping windows of cert validity so there's always a valid cert available for their services and can tolerate the CA being out of action for 8(?) hours. Imagine if your TGS/Kerberos or AWS IAM IdP was down for 8 hours.
For persistent services using the affected ACME API, the window is usually 30 days.
But that didn’t stop Youtube and Youtube TV from going down hard. I imagine they’re provisioning ephemeral VMs or service instances and relying on them being able to get certs immediately, or something like that.
It is a well-known fact that the moment YouTube goes down, the collective productivity of Earth increases by approximately 4,000%, which is immediately squandered by everyone going to Hacker News to read comments about YouTube being down. I myself have taken to podcasts… an ancient medium in which people simply talk at you for ninety minutes without a single sponsorship for a mobile game, and this is considered a failure
They've begun injecting obnoxious ads into the downloadable mp3s on a lot of podcasts I've found. Hyperlocal ads for tire shops and bakeries.
I don't want to buy tires, I want to learn about ______. The ads don't even make sense because they're irrelevant.
VPN to Sweden to get the IP geolocated ads to retarget. The ads still exist but they're less obnoxious, and they're often in Swedish so you don't have to know what they're on about anyway.
And what are we Swedes supposed to do?
VPN to Greece!
https://en.wikipedia.org/wiki/Greek_to_me#:~:text=Swedish
Careful, I enjoyed this bonus (being in Japan and not being able to keep up with the ads)... so much so, that I started ignoring the Japanese. Including my wife. You can imagine how well that went.
Welcome to radio 2.0.
Give it another 10-20 years and your 2 hour podcasts will be 30 minutes of morning zoo DJ banter, 10 minutes of guests, and 1.5 hours of ads.
We’ll have reached peak 90s all over again. With any luck we’ll avoid recreating the conditions for another Nickelback and can stay in the weird zone where Trip Hop and pop punk could chart at the same time.
The 00's podcasts I listened to were often in 2-3 hour episodes, rarely well scripted (or scripted at all?), but a lot of fun and very amateurish. I re-listened to several entire series recently and the episode lengths were the only thing I think was worse than in newer podcasts.
On the other hand, if ads etc gets too annoying, I already have run all my downloaded podcasts through whisper to get transcripts with timestamps. Running some LLM to find ranges to delete would probably be quite easy. As a bonus I would be happy to also cut out all the filler repetitions that seem popular these days ("yes, X, I absolutely agree, [repeats everything X just said]"). Could probably cut 1 hour episodes to 20 minutes without losing any content.
> 2 hour podcasts
You have high hopes. Next YT tool will be to split anything long in 30s reels as brains will be completely incapable of focusing for longer.
And it will all be AI generated specifically for you live.
At least it is somewhat relevant. Hearing ads about Irish telecom operator ads at the other side of europe is pretty goofy. What's the actual point? Just worsening the podcast experience?
I listen to multi-hour unsponsored content on Youtube almost exclusively.
Well one must also argue the opposite. I myself have gained immense knowledge from YouTube. I have learned things like phone screen replacements or phone battery replacements. I call myself a mechanic from the school of YouTube and have saved myself at minimum $10k in repairs doing the work myself. I have learned to make endless food recipes or create things like giant bubbles or slime for my kids. My point is that I bet sure for some YouTube is a massive time sink waste of time. But I also wonder how much it has improved the knowledge, skills and ability of others. My dad often mentions how had he had YouTube when he was younger how much it would have done for him. He talks about having to go to the library and if lucky there was a book that could show you the knowledge you were looking for. He says but now you can find not just the knowledge but for example specific knowledge like car make model and year and how exactly to do job xyz. Ultimately I just can not imagine life without the wealth of knowledge YouTube has given me.
Congratulations! You’ve successfully avoided YouTube Shorts.
YT shorts are up to 3 minutes now.
At this point it is just YT Vertical Videos.
Personally, I just scroll through them. They break the feed into well defined "chapters" at the end of what I can decide to look into the next one or go somewhere else because there's nothing good there today.
Also there's this woman that makes very funny shorts about software development and good long videos that aren't as good. I look for her shorts too.
I just stay on my subscriptions page. Most of them don’t do Shorts, and the few that do don’t do many so they’re easy to ignore.
Lol I laughed out loud reading this comment. When shorts first came out they annoyed me to no end. I searched for how to block them through settings or other ways to just make them go away.
But now days I can admit there are a few, very few, content creators who create shorts that are very informative and straight to the point that can cover a topic and give you many facts and let you decide if you want to seek more. Sometimes it is nice to have the 30 seconds Coles notes verses a video stretched out to 10 minutes to be eligible for monetization.
BUT, and this is a big but, the shorts and similar video platform trends scare me as a parent. I can see how my kids find a 1.5 hour movie boring but can scroll endlessly through shorts. It might seem harmless letting your kid just scroll on YouTube from my perspective is like an addiction and kids are getting that dopamine hit watching a clip and seconds later watching something else. I've learned that it is very important to be aware of what your kids are being accustomed to and push them in the right direction.
You can avoid the infinite scroll by taking the short video ID and inserting it into the regular player URL.
This comment sponsored by Vivo barefoot. I really do wear them myself. Honest.
I watched a movie, same late night talk show host, something like "welcome night owls".
I "loved" the style but I haven't found any actual radio on the internet of that style or a podcast. Not sure about name of movie but I do remember it being in the last 10-15 years.
Ah, so that’s probably why YouTube is also down (at the time of this comment)
I am playing a YouTube video (since the time of this comment) and it has not been interrupted.
I am too. But I just loaded up a new youtube page and it's completely white except for a few menu buttons.
It seems to be back, now.
You can still see your subscription videos, just not the homepage.
Searching also works. Actually it seems only the recommendation system is down, which I'd say isn't completely a bad thing.
It is pretty annoying for those of us for whom the recommendation system actually works well.
What do you recommend?
(i'm that old)
My subscriptions page just shows an error. And the app version won't load at all.
I'm able to play videos that are bookmarked in my browser, but the YouTube home page errors out.
> I am playing a YouTube video (since the time of this comment) and it has not been interrupted.
So you're using snakeoil certificates and MITM proxies at work?
Perhaps the same underlying cause, but there's no reason why Google's public CA being temporarily down would bring YouTube down.
If multiple services are affected, it's probably some underlying infrastructure issue.
It could prevent Google from rotating in new instances, because they aren't able to obtain a certificate.
Although, if that is the case, I would expect to to impact basically every google site.
Google uses mTLS for communications between systems and it could just be bad timing.
Yeah companies which also operate CAs can print as many certs as they want so it’s tempting to use a bunch everywhere with very short expiry.
youtube (recommendations/homepage) also seems down, I wonder if its relater.
I can see all the videos and play the ones in my subscription tab though.
Not sure but it is very strange i was served a strange tom And jerry video https://youtu.be/rilFfbm7j8k
You can watch any YT video by directly following a link or from history/playlist etc. Its just their homepage etc is down
> The fix has been rolled out and the issuance flow has been undrained. We again apologize for the inconvenience.
issuance flow has been undrained?
Draining is terminology they use for draining traffic from a service.
"Undrain" is not idiomatic, at least outside of Google. One might drain a tank or creek to empty it, the reverse isn't "undraining" to fill it back up. "issuance flow has been restored" might be a more widely understood phrasing.
Admittedly, a nitpick, however the tech industry has a tendency to invent new words when they could say the exact same thing in plain English and be better understood by a wider audience.
google sre speak
Eight hour estimated restoration time!
Time to go over my Watch Later list
Here's a direct link to the latest Veritasium. You're welcome!
https://www.youtube.com/watch?v=cMx139eTxoc
> A fix to resolve the issue will roll out in about 8 hours
oof
I guess it's good Google hasn't succeeded in forcing people to renew certificates every 8 hours (yet)
In theory 8 hours of downtime should be fine for a CA. Obviously not ideal, but the pki system is not meant to be a live system.
Fairly sure it used to be pretty much a manual process where someone had to actually process your request for a certificate on the other side.
Yes, and it's not that long ago, or I aged really quickly.
For code signing certificates and EV certificates, (and OV certificates, if they are even alive), this is still the case.
That feeling when you have to suspend production service until the time lock safe can be opened.
That feeling when you finally get the timelock safe open and have to do certificate work that shatters YouTube’s connection to the account personalization systems.
The same amount of time it feels like it takes for my google functions to deploy.
It's a good thing we have ever-shrinking certificate lifetimes and automation never breaks. That's what I've been told, anyway.
Yeah, this could end up as the actual root cause of The Great Oops that I've been raving about for years. And Google probably would be the right company to fuck it up in the worst way possible since Google Knows Best In All Situations.
I don't subscribe to your newsletter. What about the Oops?
Do you have a blog post on the oops? I'd love to read it.
I can't wait for the Great Oops.
Please tell me more about The Great Oops
It's inevitable that one of the major cloud providers will irrecoverably delete all customer data with one single fat-fingered command. Though in google's case I'll also consider the prophecy to be fulfilled if they delete their own data.
It will forever be known as The Great Oops.
It's not inevitable, it's essentially impossible.
There are a few things that can cause tremendously widespread outages, essentially all of them network configuration changes. Actually deleting customer data is dramatically more difficult to the point of impossible - there are so many different services in so many different locations with so many layers of access control. There is no "one command" that can do such a thing - at the scale of a worldwide network of data centers there is no "rm -rf /".
Delete a decryption key. Good luck! I'll see you at the end of time.
Break your control plane, and you can't stop the propagation of poison.
Propagate the wrong trust bundle... everywhere.
Also, it's not about the delete command. It's about the automatic cleanup following behind it that shreds everything, or repurposes the storage.
Children of the kubernetic line.
Cyclic infrastructure dependencies suck :(
Google accidentally deleted customer location history data from customer devices (after intentionally deleting it from Google servers) just last year.
If didn't back it up yourself, it is gone forever.
Ah, but you fail to account for Google's incredible knack for building tools designed to do things at scale. Or put AI in things that don't need it.
The possibility Google will either manage to unleash a malicious AI on their infrastructure and/or develop a way to destroy a lot of data at scale quite efficiently or some combination of the two is far from zero.
Bear in mind, this "Little Oops" should also have been impossible: https://www.techspot.com/news/103207-google-reveals-how-blan...
.....no?
"We deployed this private cloud with a missing parameter and it wasn't caught" is as different from "we wiped out all customer data" as hello world is from Kubernetes.
No one promised this "should be impossible". Did you confuse "we'll take steps to ensure this never happens again"?
It's pretty much half the puzzle actually.
You contend there's no global rm rf for a global cloud provider, but clearly a missing parameter can rm rf a customer in an irrecoverable manner.
The only half you're missing is... how every major cloud outage happens today... a bad configuration update. These companies have hundreds of thousands of servers, but they also use orchestration tools to distribute sets of changes to all of them.
You only need a command to rm rf one box, if you are distributing that command to every box.
Now sure, there are tons of security precautions and checks and such to prevent this! But pretending it's impossible is delusional. People do stupid stuff, at scale, every day.
The most likely scenario is a zero day in an environment necessitating an extremely rapid global rollout, combined with a plain, simple error.
And the most telling thing about most of these outages is that the provider later admits in their postmortem that they just didn't really understand how the system they made worked until it fell over and were forced to learn how it really works.
It's the sort of thing that used to keep me up at night.
This is the thing that gets me about postmortems. The document almost always reveals that the mental model the team had of the system was subtly wrong in some critical way. Not wrong in some dramatic fashion - just a quiet assumption about ordering, or a dependency they forgot existed, or a timeout they assumed would never actually fire.
I've seen this pattern repeatedly: the system works fine for months or years, and the postmortem reads like a mystery novel where the detective discovers the murder weapon was in the room the entire time. The failure mode was always there, just waiting for the right combination of load and timing.
The scariest variant is when the monitoring itself has the same blind spot. You build alerts based on your mental model, so of course the alerts don't fire for the failure mode you didn't know existed. The system fails silently until a human notices something downstream is wrong.
When was the last time it wasn't a cascading failure caused by Rube Goldberg levels of interdependency on their own systems.
The release process, monitoring checks, etc. for a customer's private cloud is generally significantly different from the release process for a global product. I'm not going to get any more specific for all the standard NDA reasons, but having worked for Google and Microsoft among others....no, the risk you describe doesn't translate from one to the other.
Do you not remember crowdstrike?
Again: an outage caused by a config change is different from data loss.
The remediation was painful but it was not data loss.
What if a machine was supposed to be running to capture data?
Yet.
I understand you believe the checks cannot fail that catastrophically, and I agree that the likelihood they do is quite low.
But it can happen, and it only has to happen once. (Also FYI, telling me your work history just tells me you've drunk the koolaid, ain't proof you know more.)
That seems unlikely. Is Google run by one Homer Simpson?
Yes.
I don’t know if you’re being serious but that’s laughable
The idea that all customer data will be deleted is far fetched, but I feel like there have been some massive incidents. Crowdstrike comes to mind, but I feel its entirely possible that Apple/Google/etc could push out some kind of config update which bricks phones in a way they are unable to download another update to fix them.
Though I'm sure the major players are all over this risk which is why it hasn't happened.
Google wiped all of UniSuper not too long ago by mistake, I don't see why such a occurrence couldn't happen more widely.
There's at least five free ACME CAs, with failover it doesn't matter all that much if one of them falls over. If all of them fall over at once there's probably a more pressing issue like nuclear holocaust or alien invasion going on.
How many servers are set up with CA redundancy? I've yet to see one let alone hear of this practice.
For one, Cloudflare uses four different CAs almost interchangeably. Caddy also makes it easy to configure ACME failover if you're self-hosting, and defaults to using two different CAs if you don't specify any.
Frankly even with no CA redundancy, downtime would have to drag on for weeks to actually disrupt renewals. ACME certs usually get rotated after about 2/3rds of their duration has expired, so the upcoming 45 day certs will still have about 15 days of wiggle room.
They aren't all drop in replacements for each other though. For example, Let's Encrypt offers free wildcard certs (with dns verification), but for ZeroSSL, it requires a paid subscription.
ZeroSSL is weird, if you use their classic non-ACME interface then the free tier is indeed limited to 3 active certs which can't be wildcards, but if you use ACME then there's no limits and wildcards are allowed.
https://zerossl.com/documentation/acme/
> By using ZeroSSL's ACME feature, you will be able to generate an unlimited amount of 90-day SSL certificates at no charge, also supporting multi-domain certificates and wildcards.
So the question is why this hit Youtube and Youtube TV so hard. Presumably they’re relying on ephemeral instances being able to get certs immediately, or something like that.
(Or an unrelated failure, of course)
I was thinking about the time some software influencer said that if you are afraid to deploy on Friday then there's something wrong with you. Eff that! Murphy's Law! (allen holub - https://x.com/allenholub/status/1637111242610610182)
I often deployed on Friday evening. Several factors contributed to this decision.
1. Sales volume was lowest on weekends so if something went wrong it would affect fewer customers.
2. If something went wrong and I needed to revert, nobody was at work on weekends so it would not disrupt coworkers.
3. I always made it so reverting would be easy.
4. Most of my weekends were just relaxing at home, mostly doing online stuff (games, reading, videos) or doing offline stuff at my computer (programming my personal projects). It wasn't much of a bother at all to have an ssh open to something at work monitoring the new deployment for problems for the rest of Friday night and Saturday.
Hmm why youtube does not work but google.com does.
Now I'm wondering if you rely on OCSP in a TLS client and the pki is Google does it still works?
OCSP is deprecated and basically dead at this point. Some clients still use it but I don't think many (any?) have actually enforced OCSP for years since it was notoriously fickle anyways.
Interesting. If you go to youtube.com it's all messed up; missing all the videos in the listings. But if you follow a video embedded in another site to youtube, it'll show and play fine. It'll break if you try to browse away from it.
Yeah, YouTube is not one server, it's hundreds of them. The videos are served mostly from CDNs (the Content Distribution Network). It's a different set of servers than handles account logins, routing, etc.
Some Google Services are also down at the moment, unrelated to YouTube, so probably a failure along some common infrastructure pipeline.
Your History, Subscriptions and search should all work. You should be able to see any creator's page if you go to it directly. The videos are all still watchable. It's primarily the home page and recommended videos that are having issues. Basically any place they recommend videos you haven't seen is broken right now, but the videos are still there and accessible.
I've tried via VPN from the U.S., U.K., Sweden, Germany, Russia, Colombia, etc. Same issue across the board.
Heroku having service issues, dependency related?
seeing heroku issues here too, had assumed it was salesforce's fault, bc of course they are eventually going to destroy heroku somehow, right?
https://status.pki.goog/
Welp, looks like they're back up. Home page and notifications are loading just fine now.
Is that what was happening with my youtube mid workout?
Correct. It's not youtube, it's themtube.
Good thing I have nebula.tv for when youtube breaks
Isn't that the thing that a bunch of YouTube creators pitch inside their channels along with VPNs and supplements? I would never consider it because the ads rub me the wrong way. Or is it some alternative frontend for YouTube that happens to have a similar sounding name?
It is a co-op where creators make videos without the threat of being demonetized or algorithmically punished - and it’s not garbage in the way you might expect people fearful of being demonetized might be.
Lots of excellent legal analysis, history, logistics, engineering content there.
It was initially founded by some of the most popular information YouTubers like CGPGrey, but he mysteriously left the project (I suspect one side wanted to be evil and the other side did not)
Not quite. It's a co-op, where the creators own the shares of the company.
Supposedly a more holistic approach to video hosting with less oversight from the platform itself.
It's a place for creators to host long form content (that the google algorithm now disincentivizes) as well as history content that can't show a lot of history because of "violence" (like the holocaust).
Youtube is demonetizing channels left, right, and centre.
Nebula is actually quite a decent alternative/supplement to YouTube and worth the subscription IMHO.
Did someone buy the google.com domain again?
I have the domain. If you want you cat videos back, you are going to have to pay me:
ONE MILLION DOLLARS!
Down here in Southeast Asia
Trust is down ;-)
All is down in eu too
While were all here does anyone want to launch a startup for a cloud security tool I built
Still down
Everyone loves to say they work at $FAMOUS_COMPANY, but when something like this happens, no-one will say that they did this.
Looking forward to the post-mortem.
Oh I am more than happy to tell people how I took down entire Google Cloud 11 years ago. I mean, of course to the level of details Google is comfortable with to share externally :)
I'll bite; ok, what'd you do? :)
I mean, with any sufficiently large project or system it’s rarely super accurate to say one person did something.
The CA outage is hitting a lot of services, but yeah, Heroku's been on a slow decline since the Salesforce acquisition. Free tier killed, pricing creep, stagnant innovation. Even when it's not their fault, you start wondering if it's worth the risk of being on a platform that feels like it's in maintenance mode.