Funny, similar patterns are super common in API traffic. A large spike at midnight UTC, a smaller one every hour, even smaller ones every 15 minutes, and a little bump every minute if your metrics have enough resolution.
Of course, if someone calls your API every minute you should probably consider some kind of streaming, or at least include make the request cheap by requiring a cursor of some kind.
Yeah, adding 'jitter' for periodic tasks that make remote requests was standard procedure when I worked for a large CDN. Every cron entry on every server would run with a delay of between 0 and 59 seconds, based on a hash of the server name. You basically 60x your internal infrastructure capacity by doing this (depending on how long the request takes to serve, of course, as well as a few other factors)
That, exponential backoffs, and local cache for external resources needed to operate the CDN (such as configuration) were the foundation of running a large network reliably.
> and a little bump every minute if your metrics have enough resolution
Amusingly, you can also observe periodic bumps if your metrics don’t have enough resolution, due to aliasing effects dominating signal (guestures vaguely at Nyquist frequency).
Sampling metrics is sampling and leads to aliasing.
Similarly, sample count/interval limits in tsdbs like prometheus are not downsampling. They're just skipping samples. This also creates aliasing. Can be very noticeable especially if you combine it with rate/derivative functions.
Customer wants every 12 hours; customer wants uniform load on the server. You do a refresh at startup, one after a random (0-12) interval, then every 12 hours. Is that a random shift?
Even more important if your devices are dialing into a modem bank. We spent a lot of time designing retry schedules and such when I worked on these (ironically for energy storage/demand management devices)
The press release is rather light on details. They found that it is every 60seconds (duh!) and generators (or such). So what exactly is it? (Make and model?)
Seems like they did not find the exact manufacturer or the exact devices and were not able to change firmware then?
"following our communication with the manufacturer, a software update was issued that removed the strict periodicity of the current peaks in one battery block of the investigated UPS system"
You see the same thing with e.g. cron jobs, where everyone schedules on xx:00 or xx:30. No communication needed, it's just natural. More senior people recognize this and add randomness to their start times or skew a bit.
I always recommend to use weird numbers for different tasks. That way, you can see "oh my system is under severe load at *:13 (or 17, 23, 42, ...). That must be this cron job"
I do a similar thing with ping payload sizes to be able to fingerprint the traffic: I’ll use prime numbered packet sizes so that they can be detected even when the packet sizes are summed.
Btw, one of those numbers is not like the others (has more prime factors that evenly divide 60).
It's often failing of the tools used, cron (and various other OS equivalents) just doesn't have good facilities to it.
Systemd timers have some, for example RandomizedOffsetSec=30m on 30m job will spread it nicely.
I'd also like if schedulers just had option "just fit it in with other jobs and fire it with this rough interval when there is least amount of other jobs running"
Depends. If you want a breadcrumb trail, use primes. If you want no noticeable effect at all, or cannot imagine a traffic bump that you'll want to decipher, use random offsets.
The water-hammer effect, or in this case electron-hammer effect, is why you don't reboot every Windows server in the data center at exactly 2am, to stagger the load shift over time. Surprised that the power guys didn't realize that switching a presumably multi-hundred-MW load in and out at the top of each minute would cause a problem.
In computer/systems/distributed engineering this is called request smoothing or traffic shaping, and it's implemented by adding jitter, which is a random time offset/pad added to the interval.
The funny thing is, modern electrical codes already account for this phenomenon. In order to implement Net Metering (selling your power to the grid), modern standards implement two kinds of jitter:
- Sandia Frequency Shift: a minute shift in the power frequency generated by inverters. If the grid is connected, this is effectively eliminated; if the inverter is disconnected from the grid, it detects the frequency has shifted and disconnects the inverter from the grid.
- Slip-Mode Frequency Shift: basically the same thing, but phase shift (which I don't understand).
Both of these are used for anti-islanding (don't create an island of power). When the inverter detects it's isolated and stops outputting current, it then has to check every 5 minutes, with jitter, for whether the grid is restored or not. It has to add random jitter, because otherwise all the mini battery/solar installs will come online at the same time and swamp the grid, causing another outage.
Sounds like they've been set up as cron tasks. I've always been unhappy with the 1 minute granularity of cron tasks. Only thing I could come up with was putting a sleep(random()) type of line to avoid a thundering herd type of issue.
The Google internal equivalent to cron has a ? placeholder in the scheduling spec syntax letting the scheduler deterministically pick an arbitrary valid value within the field range. So ? 2 ... will pick any minute between 02:00 and 02:59 (and I suspect any second within that minute). The value is derived from a hash of the job definition, so overall this spreads the cron load within the hour.
Obviously cron jobs are not all equal in terms of load and duration, and hours are not all equal in terms of usage, but that still helps prevent trivial spikes similar to the one mentioned in the article (disclaimer: I'm not aware of a recent counterfactual experiment proving this, but that seems a fair assumption).
I suppose one would end up with checks happening on a random second instead of a random minute - reducing the size of the spike by a factor of 60, but still spiky.
Timers also let you logical-AND elements within one simple declaration. Combine that with a random delay, and you can easily define a trigger such as "the 2nd Saturday of the month, anytime between midnight and 6AM":
You can tell a timer whether to re-use that same randomly-selected time during subsequent runs (FixedRandomDelay), or to re-roll the time for every trigger. They can also persist their last-run history across reboots, and trigger if they missed their previous time-slot.
The improved logic vs. cron is one of the features which makes my life easier, and so I started using systemd more often. IMHO some of the newer additions aren't quite as solid; but the basic units have proven to be very handy. Another example: The 'path' unit, which can trigger whenever a defined file is updated.
It's a human problem. Noone "expects" their own "minimal"
checks to compound. And everyone (re)implements periodic checks in ever-more-inventive ways.
BSD cron solves this ? Alright. Does prometheus use it to schedule its periodic polling ? Do OTel collectors use it to smear their backend metrics uploads ? Does your own code use cron jobs for its "periodic" heartbeating ?
If the answer to that is all-yes, then my compliments to you. Alas ... at least my experience says that's not how things are done.
The "forced to use systemd" meme is a weird inversion. No one complained about being "forced" to use sysvinit when their was no choice, but when distros started to use systemd, which still allows to use sysvinit style services, that's suddenly framed as an infringement of free choice.
journalctl is good idea executed terribly. It's builtin data format is massively wasteful and not even good at searching for data you want (often need to go thru all the files anyway)
Journalctl with SQLite or a columnar embedded database would be amazing for sysadmins. Current format is entirely useless
That's interesting as it is also visible with a crontab -l so that new scripts can avoid using the same second. However, if you have a farm of machines with the same hardcoded value, they will all still stampede at the same time just not at the top of the minute. By allowing a random value, their start times are slightly offset with in the minute. Is the precise 60s consistent interval really that critical or just that each machine checks once per minute-ish?
Yes, using prime numbers are a good way to reduce the risk of multiple timers coinciding, for example a 10s, 30s, 60s period on a freshly restarted node / process.
By default, systemd timers activate at a random interval within 60 seconds of when the timer is actually set in order to solve that potential issue. It's weird when you set a 60s timer and it activates twice within a few seconds.
power guys got surprised as rest of the grid load is human randomised and even huge factories will come on one chonk at a time, vs miilisecond co ordinated load switching at scale, with batteries as the load of the initial surge into even fully charged batteries can be almost infinite, and the whole calculation around grid scale turbines is that they run at very precise rpm's for stability and longevity, but worse, and a big fail, as battery storage is supposed to help with that, not be the problem.
bet it gets fixed realy fast as those grid guys definitly get listened to.
> as the load of the initial surge into even fully charged batteries can be almost infinite
There was an post a couple of days ago about a telco debrief of an interesting outage that happened. Several things went wrong like the audible bell was manually silence from being heard as a step in routine working but was forgotten to put back into normal mode once work was complete. The actual loss in power triggered the automatic switch to the batteries, but something else seemingly benign happened at the same time so that when power was restored, the batteries had been depleted to the point the charging process could not start because of something similar to this. I wish I could remember the write up, as it was really well written about the cascades of failures that occurred.
> grid scale turbines is that they run at very precise rpm's for stability and longevity
The turbines/generators set the grid frequency itself.
Having the batteries disconnect/connect would cause the generators to slow down and then speed up (or vice versa) which makes the rest of the grid frequency fluctuate as well.
It may also trigger the FCAS part of the grid that is paid to keep the frequency within the required range which is +/- 10mHz-200mHz.
It seems more like one or multiple suppliers of battery systems not taking into account that the decentralized deployments of their system could have that effect. If it's not regulated and an open market, this was bound to happen sooner or later.
i'm confused, why would a system simply checking its own power state be taking so much electricity as to show up on grid measurements? unless they mean that the systems are choosing to start charging the batteries, but then it's a continuous load and not a spike at the start of the minute, right?
The Frequency is actually increasing at every event which means grid load is reduced for a short while every minute. Those UPS installations basically briefly switch to Battery every Minute to test Capacity/internal resistance and so on. This so called "load dump" is visible in grid frequency since a lot of these systems work synchronous thanks to NTP
It can be that part of battery is disconnected, to estimate cells voltage. You can not measure voltage while another source is chsrging battery. Or capacitors are charging.
I think you can. RC toys do that all the time. Maybe the measurement is less precise, but you usually don't need super precise voltage, especially if you know your capacitance and measure current.
Current doesn't tell you if the battery is leaking charge, or otherwise failing to store it though. Vastly increased losses could be a sign of failure, for example. In any case it's important to be sure.
If they are checking the charging at exactly the minute, then start of charge will happen then too, which will create a spike if it's synchronized with many others doing the same thing.
Doesn't seem that complicated from the article though. They say that they immediately knew it was clock based due to the strict interval. With REA etc. it might become easier to proof, but I guess it's a measurement problem that only appears in large networks?
Can someone with more understanding explain why this was apparently so difficult to find out?
When you monitor a power line, you don't get a tcpdump showing who's producing and who's consuming the power. You can't just jump straight to the culprit.
It's just bulk flows, so when a weird flow shows up, you've got to move up or down the line, past a few producers or consumers, and see if the flow changes. For a subtle effect, that might not even be so clear.
Worse, for an effect that's most visible at the medium-voltage transmission level, simply placing a monitor might take weeks. These are planned installations with incredible safety considerations -- you don't just plug in.
Finally they found a research campus with a bunch of battery storage systems, AND the ability to install a bunch of monitors, and persuaded them to install dozens of monitoring points. Finally a high-resolution view of the effect allowed them to confirm the behavior.
> Can someone with more understanding explain why this was apparently so difficult to find out?
Because the grid as a whole is dumb as fuck. Sure, you got measurement stations doing very fine frequency analysis, to the point you can use grid hum to timestamp audio recordings [1].
But you rarely have realtime insight at individual busbar level for each substation or heaven forbid consumers. Best chance you get is 15-minute consumption pings and voltage/frequency metering at large plants.
That means if you want to track down bad actors like the batteries here or the Kosovo/Serbia bullshit [2] that dragged down the frequency on the entire ENTSO-E grid (which is quite the challenge!), it takes months if not years.
> When faults happen though, recordings are automatically made available.
Depends on the fault. Without dedicated measurement in place it's still a lot of (literal) legwork to install current and voltage loggers.
> Some of these measurements can have surprisingly high frequency sampling, even though the grid itself feels "slow".
Sure. But especially in a world dominated by more and more solar and wind, where something like a cloud passing by can easily shed hundreds of kilowatts in a matter of seconds, the grid needs much, much more data across the board and everywhere - and much, much more battery storage, both centralized at substations and decentralized in people's basements.
That's not really telling us anything. The way you convert a time domain signal to a frequency domain signal, that is fourier analysis. There are other uses for it too, but that is the primary use.
By increasing the length of the FFT you lower the apparent noise floor in the FFT magnitude plot and are able to detect smaller signals. That's called "FFT processing gain". The formula is:
FFT gain = 10*log(M/2)
where M is the number of points processed in the FFT. Doubling the number of points, lowers the apparent noise floor by 3 dB.
(I don't know if that's what the researchers actually did. I'm just explaining what the comment presumably meant.)
That's what they did. See [1], Fig. 3. (Which, amazingly for Nature, has a free download of the PDF.) This isn't something that shows up in a plain load vs time graph.
>> explain why this was apparently so difficult to find out?
Likely because the source of the noise was a class of energy providers that themselves react to fluctuation in the grid. Electricity may travel at the speed of light but all the little switches and transistors that connect/disconnect suppliers do not. It would have been noise within noise.
Well ya, nothing on earth moves that fast, but the speed of power/signal in a grid is a speed described using direct reference to c, as opposed to the frequency of AC current which is measurable in frequencies we can actually appreciate with our own ears/eyes. They are in different leagues.
If it were not a substantial research effort spanning years, it would not get funded. This motivates academic researchers to ensure that every topic is explored thoroughly and at great length.
The grid frequency is set directly by 3 phase generators. So if the grid frequency drops, there is indeed a very strong torque trying to decelerate the turbines.
To some extent, this is a good thing. Because it means the spinning mass of all turbines in the grid actually provide a buffer for sub-second mismatches in demand and supply. But it does mean that big demand spikes (or drops) put some wear on the systems. They are made for this, but if it gets very periodic it might be more than designed for.
Note that this means that all turbines in a power network are perfectly synchronized. And if they are not, they are subject to massive torques. This is why a cold start is so hard. You need to bring on generators from one central point to keep everything in sync. And as you bring on new generation capacity you also need to bring on new consumption, that electricity has to go somewhere.
The reason for this synchronization is because the generators are spinning a magnet past 3 coils, and those coils are directly wired into the power network. If your magnets are out of phase then effectively the entire power network will use those coils to generate a field to get that magnet back in sync. All of those coils on all those generators really are all hard-wired together, it still kind of blows my mind.
When the battery storage systems all temporarily pause providing grid inertia due to the state of charge check they’re performing, grid synchronized spinning mass has to compensate, which slows it down (transferring the kinetic energy to electrical energy), and then is sped back up as battery storage resumes inertia support.
Broadly speaking, it’s a collective momentum problem. If part of the system inertia sags, another part needs to step in to compensate for the inertia loss, otherwise you have brownouts and eventually, a blackout.
Oh interesting that got reformed, too! Strange choice to split it in two instead of three words (the latter which would have mimicked English: “all too”).
Well, then it seems we don’t have any single German word for all too regular anymore :(
Until the reforms in 1996, allzu… was one word: allzuoft, allzuschön, allzugroß, allzuviel, etc. But yeah, probably would have written such a long word in two words even then.
My new proposal would be überregelmäßig. Über+adjective is still one word only, post-reform. Sounds strange, but grammatically correct, and still conveys the “unusual” part.
Not a scholar of languages, but maybe something like "mit unheimlicher Regelmäßigkeit" (with un-home-ly regular-like-ness, read "home" like in "family" -> "familiar").
There's no such thing as sysadmins anymore. I would be expected to solve this in an afternoon back in the day, and if it was really that complicated, maybe a week, and I would get to loop in some of the dev team.
If I took a bit more than that, I would be constantly harassed about how much the dev team's hours cost. The same team that has become table tennis experts and shows up 2 days a week.
no, you hash your node ID or get a random number, and then just add offset. That guarantees on average uniform load, not synchronized-wave-pattern-that-happens-to-be-59.
You have to consider that when power goes down and everything goes back up at same time you get same pattern back, just spike every 59 seconds instead of 60.
Yeah I thought about that, but I just assumed it would be very unlikely that everything would be initialized at the same time. I suppose your scenario of power being lost simultaneously makes sense when that could happen.
Funny, similar patterns are super common in API traffic. A large spike at midnight UTC, a smaller one every hour, even smaller ones every 15 minutes, and a little bump every minute if your metrics have enough resolution. Of course, if someone calls your API every minute you should probably consider some kind of streaming, or at least include make the request cheap by requiring a cursor of some kind.
Package managers like apt will delay automatic updates for a random time to avoid this problem:
https://wiki.debian.org/PeriodicUpdates#Modify_the_schedule_...Opscode built in this feature into Chef when they first released it, oh so many years ago. They learned their lesson the hard way.
Systemd timers also have randomization features. They're very useful.
Yeah, adding 'jitter' for periodic tasks that make remote requests was standard procedure when I worked for a large CDN. Every cron entry on every server would run with a delay of between 0 and 59 seconds, based on a hash of the server name. You basically 60x your internal infrastructure capacity by doing this (depending on how long the request takes to serve, of course, as well as a few other factors)
That, exponential backoffs, and local cache for external resources needed to operate the CDN (such as configuration) were the foundation of running a large network reliably.
> and a little bump every minute if your metrics have enough resolution
Amusingly, you can also observe periodic bumps if your metrics don’t have enough resolution, due to aliasing effects dominating signal (guestures vaguely at Nyquist frequency).
By definition, if you can see effects then the metrics have enough resolution.
The specific resolution is the Nyquist frequency - that part is accurate.
Sampling metrics is sampling and leads to aliasing.
Similarly, sample count/interval limits in tsdbs like prometheus are not downsampling. They're just skipping samples. This also creates aliasing. Can be very noticeable especially if you combine it with rate/derivative functions.
I've just convinced my team to be nice and add a random delay for all those periodic checks in our software last week :)
Yes of course, there was the check at 2 am local time before ...
Just adding random shift of the duration of the check already will get you uniform distribution
Customer wants every 12 hours; customer wants uniform load on the server. You do a refresh at startup, one after a random (0-12) interval, then every 12 hours. Is that a random shift?
Same here. We see this kind of spikes in our tunnel traffic also - every hour and specially at midnight UTC. Cron jobs everywhere :)
Jenkins server has the feature to randomize job start time since at least 15 years. One would've thought this is a well known problem.
I wished cron had a similar feature. I missed that on acouple of occasions.
Even more important if your devices are dialing into a modem bank. We spent a lot of time designing retry schedules and such when I worked on these (ironically for energy storage/demand management devices)
The paper has more details: https://www.nature.com/articles/s41560-026-02144-0
The press release is rather light on details. They found that it is every 60seconds (duh!) and generators (or such). So what exactly is it? (Make and model?)
Seems like they did not find the exact manufacturer or the exact devices and were not able to change firmware then?
The paper has a little more details:
https://www.nature.com/articles/s41560-026-02144-0Given the issue is so widespread (international, affecting pretty much every grid), it's likely that multiple manufacturers use similar approaches.
You see the same thing with e.g. cron jobs, where everyone schedules on xx:00 or xx:30. No communication needed, it's just natural. More senior people recognize this and add randomness to their start times or skew a bit.
I always recommend to use weird numbers for different tasks. That way, you can see "oh my system is under severe load at *:13 (or 17, 23, 42, ...). That must be this cron job"
I do a similar thing with ping payload sizes to be able to fingerprint the traffic: I’ll use prime numbered packet sizes so that they can be detected even when the packet sizes are summed.
Btw, one of those numbers is not like the others (has more prime factors that evenly divide 60).
Good point. 7,11,13... do not repeat every hour.
It's often failing of the tools used, cron (and various other OS equivalents) just doesn't have good facilities to it.
Systemd timers have some, for example RandomizedOffsetSec=30m on 30m job will spread it nicely.
I'd also like if schedulers just had option "just fit it in with other jobs and fire it with this rough interval when there is least amount of other jobs running"
You ought to use RandomizedDelaySec=xxx
Depends. If you want a breadcrumb trail, use primes. If you want no noticeable effect at all, or cannot imagine a traffic bump that you'll want to decipher, use random offsets.
It’s entirely possible the battery controllers are running linux too
The water-hammer effect, or in this case electron-hammer effect, is why you don't reboot every Windows server in the data center at exactly 2am, to stagger the load shift over time. Surprised that the power guys didn't realize that switching a presumably multi-hundred-MW load in and out at the top of each minute would cause a problem.
In computer/systems/distributed engineering this is called request smoothing or traffic shaping, and it's implemented by adding jitter, which is a random time offset/pad added to the interval.
The funny thing is, modern electrical codes already account for this phenomenon. In order to implement Net Metering (selling your power to the grid), modern standards implement two kinds of jitter:
- Sandia Frequency Shift: a minute shift in the power frequency generated by inverters. If the grid is connected, this is effectively eliminated; if the inverter is disconnected from the grid, it detects the frequency has shifted and disconnects the inverter from the grid.
- Slip-Mode Frequency Shift: basically the same thing, but phase shift (which I don't understand).
Both of these are used for anti-islanding (don't create an island of power). When the inverter detects it's isolated and stops outputting current, it then has to check every 5 minutes, with jitter, for whether the grid is restored or not. It has to add random jitter, because otherwise all the mini battery/solar installs will come online at the same time and swamp the grid, causing another outage.
Sounds like they've been set up as cron tasks. I've always been unhappy with the 1 minute granularity of cron tasks. Only thing I could come up with was putting a sleep(random()) type of line to avoid a thundering herd type of issue.
The Google internal equivalent to cron has a ? placeholder in the scheduling spec syntax letting the scheduler deterministically pick an arbitrary valid value within the field range. So ? 2 ... will pick any minute between 02:00 and 02:59 (and I suspect any second within that minute). The value is derived from a hash of the job definition, so overall this spreads the cron load within the hour.
Obviously cron jobs are not all equal in terms of load and duration, and hours are not all equal in terms of usage, but that still helps prevent trivial spikes similar to the one mentioned in the article (disclaimer: I'm not aware of a recent counterfactual experiment proving this, but that seems a fair assumption).
I suppose one would end up with checks happening on a random second instead of a random minute - reducing the size of the spike by a factor of 60, but still spiky.
That still lands it at the top of the minute, which is AFAIK the pattern the researchers found.
Yes it’s a real effect. I’ve just hardcoded offsets on various jobs:
etc.That way I still maintain a per-job 60-second cadence consistently (which random would not), but don’t do it right on the top of the minute.
Systemd timers do this natively btw, no need to hardcode And imho are also friendlier.
Timers also let you logical-AND elements within one simple declaration. Combine that with a random delay, and you can easily define a trigger such as "the 2nd Saturday of the month, anytime between midnight and 6AM":
You can tell a timer whether to re-use that same randomly-selected time during subsequent runs (FixedRandomDelay), or to re-roll the time for every trigger. They can also persist their last-run history across reboots, and trigger if they missed their previous time-slot.The improved logic vs. cron is one of the features which makes my life easier, and so I started using systemd more often. IMHO some of the newer additions aren't quite as solid; but the basic units have proven to be very handy. Another example: The 'path' unit, which can trigger whenever a defined file is updated.
https://wiki.archlinux.org/title/Systemd/Timers
BSD cron has supported jitter for decades. This is strictly a linux problem.
It's a human problem. Noone "expects" their own "minimal" checks to compound. And everyone (re)implements periodic checks in ever-more-inventive ways. BSD cron solves this ? Alright. Does prometheus use it to schedule its periodic polling ? Do OTel collectors use it to smear their backend metrics uploads ? Does your own code use cron jobs for its "periodic" heartbeating ?
If the answer to that is all-yes, then my compliments to you. Alas ... at least my experience says that's not how things are done.
But systemd bad :[
People slowly are getting used to it. (Less so with journalctl)
More like people were forced to use systemd and gave up due to lack of choice, along with an entire generation not old enough to know SysVinit.
The "forced to use systemd" meme is a weird inversion. No one complained about being "forced" to use sysvinit when their was no choice, but when distros started to use systemd, which still allows to use sysvinit style services, that's suddenly framed as an infringement of free choice.
journalctl is good idea executed terribly. It's builtin data format is massively wasteful and not even good at searching for data you want (often need to go thru all the files anyway)
Journalctl with SQLite or a columnar embedded database would be amazing for sysadmins. Current format is entirely useless
That's interesting as it is also visible with a crontab -l so that new scripts can avoid using the same second. However, if you have a farm of machines with the same hardcoded value, they will all still stampede at the same time just not at the top of the minute. By allowing a random value, their start times are slightly offset with in the minute. Is the precise 60s consistent interval really that critical or just that each machine checks once per minute-ish?
Our configuration management system(Puppet) has function for that, fqdn_rand
give it number and you will get random value up to that number, same every run for a given host, but different per host (basically, hashed host fqdn)
add a label and you will get different number but similarly consistent. Makes it real easy to randomize run across the fleet.
Would be easy script to make.
Also systemd timers have the randomization builtin, just need to specify time
Yes, using prime numbers are a good way to reduce the risk of multiple timers coinciding, for example a 10s, 30s, 60s period on a freshly restarted node / process.
Reminds me of hunting tooth gearsets. Perhaps these systems should be checking battery every 59, or 61 seconds, as those numbers are coprime with 60.
https://shoppress.dormanproducts.com/hunting-tooth-gearsets/
By default, systemd timers activate at a random interval within 60 seconds of when the timer is actually set in order to solve that potential issue. It's weird when you set a 60s timer and it activates twice within a few seconds.
Yeah, and it's not cron, it's "everything running on a digital schedule"
Analog systems naturally have a dispersion to them, digital ones, especially with everybody on NTP, much less so
power guys got surprised as rest of the grid load is human randomised and even huge factories will come on one chonk at a time, vs miilisecond co ordinated load switching at scale, with batteries as the load of the initial surge into even fully charged batteries can be almost infinite, and the whole calculation around grid scale turbines is that they run at very precise rpm's for stability and longevity, but worse, and a big fail, as battery storage is supposed to help with that, not be the problem. bet it gets fixed realy fast as those grid guys definitly get listened to.
> as the load of the initial surge into even fully charged batteries can be almost infinite
There was an post a couple of days ago about a telco debrief of an interesting outage that happened. Several things went wrong like the audible bell was manually silence from being heard as a step in routine working but was forgotten to put back into normal mode once work was complete. The actual loss in power triggered the automatic switch to the batteries, but something else seemingly benign happened at the same time so that when power was restored, the batteries had been depleted to the point the charging process could not start because of something similar to this. I wish I could remember the write up, as it was really well written about the cascades of failures that occurred.
I believe this is what you're referring to.
https://news.ycombinator.com/item?id=50153229&hl=en-US
Yes, thanks for allowing me to offload my RAM to yours!
Gotta give Gemini the credit on this one! I didn't see the original, but it was interesting!
> grid scale turbines is that they run at very precise rpm's for stability and longevity
The turbines/generators set the grid frequency itself.
Having the batteries disconnect/connect would cause the generators to slow down and then speed up (or vice versa) which makes the rest of the grid frequency fluctuate as well.
It may also trigger the FCAS part of the grid that is paid to keep the frequency within the required range which is +/- 10mHz-200mHz.
https://mainsfrequency.com/frequ_info_en.php
It seems more like one or multiple suppliers of battery systems not taking into account that the decentralized deployments of their system could have that effect. If it's not regulated and an open market, this was bound to happen sooner or later.
It's not a multi-hundred-MW load
It's thousands of few kW's of load
https://news.ycombinator.com/item?id=50314617
It had both ended up in the second chance pool.
i'm confused, why would a system simply checking its own power state be taking so much electricity as to show up on grid measurements? unless they mean that the systems are choosing to start charging the batteries, but then it's a continuous load and not a spike at the start of the minute, right?
The Frequency is actually increasing at every event which means grid load is reduced for a short while every minute. Those UPS installations basically briefly switch to Battery every Minute to test Capacity/internal resistance and so on. This so called "load dump" is visible in grid frequency since a lot of these systems work synchronous thanks to NTP
It can be that part of battery is disconnected, to estimate cells voltage. You can not measure voltage while another source is chsrging battery. Or capacitors are charging.
I think you can. RC toys do that all the time. Maybe the measurement is less precise, but you usually don't need super precise voltage, especially if you know your capacitance and measure current.
Current doesn't tell you if the battery is leaking charge, or otherwise failing to store it though. Vastly increased losses could be a sign of failure, for example. In any case it's important to be sure.
temperature is also a problem
Temperature probes shouldn't require disconnecting cells, though.
no the temperature changes the voltage reading
A few microvolts difference, well under 1 milli-volt. I'm pretty sure power grid storage doesn't need that precision.
That wouldn't require dropping from power network, just disconnecting battery from load/charging
If they are checking the charging at exactly the minute, then start of charge will happen then too, which will create a spike if it's synchronized with many others doing the same thing.
Excellent sleuthing. Witthaut these fellas we'd still be in the dark
> It really was years of detective work
Doesn't seem that complicated from the article though. They say that they immediately knew it was clock based due to the strict interval. With REA etc. it might become easier to proof, but I guess it's a measurement problem that only appears in large networks?
Can someone with more understanding explain why this was apparently so difficult to find out?
When you monitor a power line, you don't get a tcpdump showing who's producing and who's consuming the power. You can't just jump straight to the culprit.
It's just bulk flows, so when a weird flow shows up, you've got to move up or down the line, past a few producers or consumers, and see if the flow changes. For a subtle effect, that might not even be so clear.
Worse, for an effect that's most visible at the medium-voltage transmission level, simply placing a monitor might take weeks. These are planned installations with incredible safety considerations -- you don't just plug in.
Finally they found a research campus with a bunch of battery storage systems, AND the ability to install a bunch of monitors, and persuaded them to install dozens of monitoring points. Finally a high-resolution view of the effect allowed them to confirm the behavior.
> Can someone with more understanding explain why this was apparently so difficult to find out?
Because the grid as a whole is dumb as fuck. Sure, you got measurement stations doing very fine frequency analysis, to the point you can use grid hum to timestamp audio recordings [1].
But you rarely have realtime insight at individual busbar level for each substation or heaven forbid consumers. Best chance you get is 15-minute consumption pings and voltage/frequency metering at large plants.
That means if you want to track down bad actors like the batteries here or the Kosovo/Serbia bullshit [2] that dragged down the frequency on the entire ENTSO-E grid (which is quite the challenge!), it takes months if not years.
[1] https://en.wikipedia.org/wiki/Electrical_network_frequency_a...
[2] https://www.entsoe.eu/news/2018/04/03/frequency-deviations-c...
"But you rarely have realtime insight at individual busbar level for each substation"
This is kind of understandable, the amount of data would be huge. When faults happen though, recordings are automatically made available.
Some of these measurements can have surprisingly high frequency sampling, even though the grid itself feels "slow".
> When faults happen though, recordings are automatically made available.
Depends on the fault. Without dedicated measurement in place it's still a lot of (literal) legwork to install current and voltage loggers.
> Some of these measurements can have surprisingly high frequency sampling, even though the grid itself feels "slow".
Sure. But especially in a world dominated by more and more solar and wind, where something like a cloud passing by can easily shed hundreds of kilowatts in a matter of seconds, the grid needs much, much more data across the board and everywhere - and much, much more battery storage, both centralized at substations and decentralized in people's basements.
The effect is apparently small enough that they had to use Fourier analysis over many minutes to see it in the noise.
That's not really telling us anything. The way you convert a time domain signal to a frequency domain signal, that is fourier analysis. There are other uses for it too, but that is the primary use.
By increasing the length of the FFT you lower the apparent noise floor in the FFT magnitude plot and are able to detect smaller signals. That's called "FFT processing gain". The formula is:
FFT gain = 10*log(M/2)
where M is the number of points processed in the FFT. Doubling the number of points, lowers the apparent noise floor by 3 dB.
(I don't know if that's what the researchers actually did. I'm just explaining what the comment presumably meant.)
That's what they did. See [1], Fig. 3. (Which, amazingly for Nature, has a free download of the PDF.) This isn't something that shows up in a plain load vs time graph.
[1] https://www.nature.com/articles/s41560-026-02144-0
>> explain why this was apparently so difficult to find out?
Likely because the source of the noise was a class of energy providers that themselves react to fluctuation in the grid. Electricity may travel at the speed of light but all the little switches and transistors that connect/disconnect suppliers do not. It would have been noise within noise.
Not quite the speed of light (in a vacuum) if your watching signals propagate along a conductor:
https://en.wikipedia.org/wiki/Velocity_factor#Typical_veloci...
Well ya, nothing on earth moves that fast, but the speed of power/signal in a grid is a speed described using direct reference to c, as opposed to the frequency of AC current which is measurable in frequencies we can actually appreciate with our own ears/eyes. They are in different leagues.
There are many millions of devices connected that draw power, and increasingly so, supply power.
If it were not a substantial research effort spanning years, it would not get funded. This motivates academic researchers to ensure that every topic is explored thoroughly and at great length.
> turbines slow down momentarily before accelerating back to their original rotational speed. This increases wear and tear.
Why though? Due to increased torque?
The grid frequency is set directly by 3 phase generators. So if the grid frequency drops, there is indeed a very strong torque trying to decelerate the turbines.
To some extent, this is a good thing. Because it means the spinning mass of all turbines in the grid actually provide a buffer for sub-second mismatches in demand and supply. But it does mean that big demand spikes (or drops) put some wear on the systems. They are made for this, but if it gets very periodic it might be more than designed for.
Note that this means that all turbines in a power network are perfectly synchronized. And if they are not, they are subject to massive torques. This is why a cold start is so hard. You need to bring on generators from one central point to keep everything in sync. And as you bring on new generation capacity you also need to bring on new consumption, that electricity has to go somewhere.
The reason for this synchronization is because the generators are spinning a magnet past 3 coils, and those coils are directly wired into the power network. If your magnets are out of phase then effectively the entire power network will use those coils to generate a field to get that magnet back in sync. All of those coils on all those generators really are all hard-wired together, it still kind of blows my mind.
Practical Engineering did a great video on power grid black starts: https://www.youtube.com/watch?v=uOSnQM1Zu4w
When the battery storage systems all temporarily pause providing grid inertia due to the state of charge check they’re performing, grid synchronized spinning mass has to compensate, which slows it down (transferring the kinetic energy to electrical energy), and then is sped back up as battery storage resumes inertia support.
Broadly speaking, it’s a collective momentum problem. If part of the system inertia sags, another part needs to step in to compensate for the inertia loss, otherwise you have brownouts and eventually, a blackout.
https://neutrino-energy.com/technology/grid-inertia/
I wish that article wasn't written entirely in Claude speak, it has some useful information, but the obvious AI voice will repel readers.
Stuxnet damaged gas centrifuges by introducing a subtle change in rotational speed over time, eventually the motors failed mechanically.
https://en.wikipedia.org/wiki/Stuxnet
Because a variation in force is a cycle of stress, which is more destructive than constant force. It's called hysteresis, leading to material fatigue.
>unusually regular
Oh I’m sure there is a German phrase for this.
I would go with “allzuregelmäßig”.
Similar to “allzuschön um wahr zu sein”, “allzugroß” etc.
However this is not a single word, but would be split as "allzu regelmäßig" cf. https://www.korrekturen.de/wortliste/allzuoft.shtml
Oh interesting that got reformed, too! Strange choice to split it in two instead of three words (the latter which would have mimicked English: “all too”).
Well, then it seems we don’t have any single German word for all too regular anymore :(
This never was one word actually.
Until the reforms in 1996, allzu… was one word: allzuoft, allzuschön, allzugroß, allzuviel, etc. But yeah, probably would have written such a long word in two words even then.
My new proposal would be überregelmäßig. Über+adjective is still one word only, post-reform. Sounds strange, but grammatically correct, and still conveys the “unusual” part.
Not a scholar of languages, but maybe something like "mit unheimlicher Regelmäßigkeit" (with un-home-ly regular-like-ness, read "home" like in "family" -> "familiar").
Thinking about it tactically, I’ll go with taktisch. :) Takt-isch.
Surely you mean Tick-Tacktisch?
rather tak-tisch if we are talking hyphenation
verdächtig !
uhrwerksmäßig
That doesn’t capture the “unusually” in “unusually regular”
Ungewöhnlichregelmäßig or Hyperregelmäßig works for me, but then there is Regelmäßigkeitsmonsterhaft.
Still, I think we can push the limits to Bürokratieapokalyptischhyperregelmäßigkeitsgötterungeheuerhaft and feel pretty comfortable with it.
While grammatically feasible, none of these sound natural
Yes true, but they sure are fun, and well .. a lot of GermanConjunctionSentences sound completely unnatural to us non-native speakers ..
Thanks for sharing, very cool!
It's really surprising to me this took years to solve.
Cha...ya know neuron, Dave is out and Jenny is on maternity leave - let's reschedule for 2 week okay?
No good. I am out that week. How about the 3rd week of November?
That's no good. Second week of December should work for all of us.
--near actual conversation with (State) government coordination team to have a 20 minute discussion.
Shows they didn't had sysadmin to look at it, it's very common pattern on IT systems
There's no such thing as sysadmins anymore. I would be expected to solve this in an afternoon back in the day, and if it was really that complicated, maybe a week, and I would get to loop in some of the dev team.
If I took a bit more than that, I would be constantly harassed about how much the dev team's hours cost. The same team that has become table tennis experts and shows up 2 days a week.
You'd think they'd randomise offsets to avoid a thundering herd effect, like anyone scheduling multiple processes does?
To anyone herding a herd this is obvious. To people crafting animals less so.
A better solution might be to make the intervals coprime with 60. 59 or 61 seconds should work fine.
https://shoppress.dormanproducts.com/hunting-tooth-gearsets/
no, you hash your node ID or get a random number, and then just add offset. That guarantees on average uniform load, not synchronized-wave-pattern-that-happens-to-be-59.
You have to consider that when power goes down and everything goes back up at same time you get same pattern back, just spike every 59 seconds instead of 60.
Yeah I thought about that, but I just assumed it would be very unlikely that everything would be initialized at the same time. I suppose your scenario of power being lost simultaneously makes sense when that could happen.
Apparently not everyone does ;)