Readit News logoReadit News
xenadu02 · 10 months ago
Generators must synchronize with the grid. Huge spinning rotor masses that will experience tremendous forces to coerce them into matching an RPM that corresponds to the grid's frequency.

Frequency is also impacted by load: the greater the load on the generator the more torque required at its input shaft to maintain the same RPM. If the generator's input engine is already at max torque then RPM must decrease all else equal. That in turn requires that every other generator on the grid also slow down to match.

When a huge chunk of generating capacity disappears there isn't enough power feeding the remaining generator input shafts (all else equal) to maintain RPM so the grid frequency must drop. That tends to destroy customer equipment among other problems.

Generators are motors and motors are generators. If the capacity disappears too quickly the grid _drives the generator as a motor_ potentially with megawatts of capacity all trying to instantly make that 100 ton rotor change from 3600 RPM to 2800 RPM or whatever. Inertia puts its $0.02 and the net result is a disintegrating rotor slinging molten metal and chunks of itself out while the bearings turn into dust.

Protective equipment sees this happening and trips the generator offline to protect it. Usually the coordinating grid entity keeps spare capacity available at all times to respond to loss of other capacity or demand changes. This is also the point of "load shedding": if spare capacity drops below a set level loads are turned off.

If spare capacity is not maintained or transmission line choke points present problems then capacity trip outs can cause progressive collapse as each generator sees excessive load, trips, and in turn pushes excess load to the next generator. If your grid control systems are well designed they can detect this from a central location and command parts of the grid to "island" into balanced chunks of load/capacity so the entire grid does not fully collapse.

Of course when you want to reconnect the islands it takes careful shifting of frequency to get them aligned before you can do that.

If all generators collapse you end up in a black start situation that requires careful staging lest more load than you expected jumps on the grid (maybe due to control devices being unpowered or stuck somewhere), triggering a secondary collapse.

Caveat: not a grid engineer so I may have gotten some of this wrong but hopefully it helps anyone who wonders why load shedding exists or how a grid can "collapse" and what the consequences are if you don't do those things and just let it ride.

raggles · 10 months ago
Am grid engineer. You nailed it. It can get incredibly complex modelling this stuff, and reading all the armchair observers banging on about 'single points of failure' is amusing.
wglb · 10 months ago
My professor did that for Bonneville Power Administration. I worked on a tiny piece of his modelling software. A short writeup of it https://ciex-software.com/fmcmx.html
xenadu02 · 10 months ago
Oh yeah. These posts are the limit of my knowledge. Real-world power plants, switchgear, and grid management is way way more complex! And if you want to talk about "hard real time requirements".... it doesn't get much harder or real time than keeping the grid going :)

Thanks to you, the operators, and the linemen out there keeping it all going. When it works correctly no one notices.

apex_sloth · 10 months ago
An interesting side effect of that is one can use the grid frequency to coordinate emergency power response - individual nodes (batteries, peaker plants, etc.) can react directly to the frequency measurement with generation or load, thus stabilizing the grid. Too much energy is equally an issue. Usually it's called fast frequency response these days.
xenadu02 · 10 months ago
Good point... oversupply lightens the load on the generator meaning not as much angular momentum is converted to magnetic flux then to an electric field. Unopposed the torque is able to increase the rotor's speed which directly determines grid frequency.

My grid tie solar system does exactly what you say. It monitors the frequency of grid power and matches it dynamically. There are defined parameters for how out of spec it can get and for how long. I don't recall the exact numbers but imagine 0.2Hz for 100ms, 0.5Hz for 1ms, 1Hz for 500ns. Same thing for voltage though that allows a much wider range.

In CA all grid tie solar also requires communication with the utility (through the manufacturer) with a backup connection source (Internet and LTE in my case). This is so if the grid is nearing capacity or going unstable the utility can command the inverters to allow a wider band of voltage and frequency. The last thing the grid needs in an unstable scenario is everyone's solar panels tripping off at the same time.

Technically the utility can also command the panels to stop production if there is an excess supply but they are limited in how long and how often they can do that.

Fun fact: The interconnect operators used to keep track of the average frequency over time and would run the grid slightly fast or slow to ensure the grid averaged 60Hz over time. This allowed clocks and such to maintain time by relying on the grid. That is no longer the case though. I think they still roughly aim for a 60Hz average but if they're behind by 0.01Hz over the past week they no longer run the grid at 60.05Hz for a while to "catch up".

Fun fact number 2 I just learned recently: Southern California used to be on 50Hz! That's right, the USA had split cycle just like Japan. Most of the country on 60Hz, SoCal on 50Hz. Right after WW2 they made the switch apparently. I guess a lot of stuff was dual frequency capable at the time but the utility provided assistance where required.

Fun fact number 3: Ever wonder why we have 110/220 or 115/230 or 120/240? Because every local utility picked their own standard: 110 (from Edison's DC system carried over into AC world), 115, or 120. It was not until relatively recently that we really standardized on 120/240 (+/- 5% which is 114 - 126 but with brief excursions allowed). That's why some old appliances might say 110 or 115 on them.

Fun fact number 4: 120/240 is a backwards compatibility hack. It was too late to change to 240 and 120 is (for physical reasons) better for electric lighting applications (thicker less fragile filament for same light output). How to solve this? Change your MV-LV transformers to 240 but center tap them. Instead of Line-Neutral you provide customers Line 1 - Neutral - Line 2. Connections across L1/L2 give you 240 volts, connections across L1-N or L2-N give you 120 volts! Everyone's happy! There is a NEMA plug standard for low-amp 240V. It has both blades horizontal (looks like the unimpressed smiley face). I wish it were more popular in kitchens for boiling water and such.

shepherdjerred · 10 months ago
> the grid _drives the generator as a motor_

Can you expand on this/link a relevant article? I'm a layman and this sounds really interesting!

wglb · 10 months ago
If you have two AC generators connected to a load and one of them shifts is phase any amount ahead of the other one, the second one becomes a motor. Power flows in proportion to the difference in phase angle.
xenadu02 · 10 months ago
Take a little hobby DC motor and hook it to a battery. It will spin. Now hook it to a small LED and turn the shaft. The LED will light up. In fact hook both up. The battery will spin the motor and light the LED based on its voltage/charge state. But if you start spinning the shaft you will begin taking over from the battery. Spin even faster and the battery will start charging. Stop trying to spin the shaft and watch the LED dim and the motor slow down as the battery takes over again.

Fundamentally electromagnetism is a unified force.

A changing electric field induces a magnetic field: the electricity creates a magnetic field inside the motor-generator causing the rotor's electromagnet or permanent magnet to spin to align with that field. For a DC motor halfway through the rotation the polarity switches causing the rotor to keep spinning to align with the new magnetic field (AC doesn't need this because it is varying on its own). The motor-generator is operating in "motor" mode here because the magnetic field changes first and the rotor is trying to catch up to it ultimately consuming energy.

Now spin a motor's shaft putting energy into the system. The rotor has a magnetic field either because its an electromagnet or a permanent magnet. You are spinning the rotor shaft so by definition the magnetic field coming off the rotor also spins. Thus the stator sees a continuously varying magnetic field which - you guessed it - induces an electric field in the stator windings.

In reality this is a continuously varying effect based on whether the rotor's motion is leading or lagging the magnetic field (really it is trying to lead or lag but never gets far otherwise it would burn up).

The short version is a generator at "idle" but synchronized is a motor. The grid is providing the power to spin that motor (or the attached engine/steam turbine is providing just enough rotation of the shaft to keep the generator synchronized but no more).

As the generator begins to "provide" power more torque is put into the shaft's rotation. This causes the magnetic field to want to lead the electric field, transferring energy into the electric field. The entire grid strongly resists the shaft actually spinning faster though. The mechanical energy that wants to make the rotor go faster is siphoned off by the magnetic flux and electric fields. Electric field steals energy from the magnetic flux. Reduced magnetic flux steals energy from the rotor's angular momentum to replenish itself (if it couldn't the rotor would slow down). The rotor gets angular momentum from the connected source of torque like a steam turbine. The Cycle of Life ... or something.

If you suddenly cut the transmission lines and took no action all that torque would overspeed the generator very quickly though because no electric current could flow so the electric field just builds up (stealing energy from the rotor) then gives the energy right back as more magnetic flux (speeding up the rotor). This would let the rotor just go faster and faster while also heating everything up.

rightbyte · 10 months ago
>If the capacity disappears too quickly the grid _drives the generator as a motor_

Essentially braking, right?

appleorchard46 · 10 months ago
I don't know the first thing about electric infrastructure, but for there to be a single point of failure like this just seems bonkers. I suppose the country's shape is partially to blame. I wonder what could have caused this.
toomuchtodo · 10 months ago
Broadly speaking, electrical grids are the largest machines in the world and are held together by systems to keep everything in harmony at a target frequency (enormous spinning mass that slows when load is added, and has to spin back up, failsafes that have to decouple parts of the grid when maintaining voltage and frequency becomes impossible with committed capacity, etc). It’s actually wild it works flawlessly most of the time and is a testament to robust systems keeping them operational. There’s always room for improvement between here and some nebulous point of diminishing returns. That cost/resiliency discovery process is constant.

In this case, there is room for improvement, at some cost to be determined from a post mortem.

Galatians4_16 · 10 months ago
Giant flywheels exist, in some systems, which can easily start or restart backup diesel generators on the fly, though probably not for whole grids.

A bit of decentralization goes a long way.

VBprogrammer · 10 months ago
In general failures like this aren't so much because of a single point of failure but because they trigger cascading failures across the network.

One piece of infrastructure trips offline, causing an abnormal situation at another, which then trips etc.

ziofill · 10 months ago
Isn’t this the same as a single point of failure, just “spread out”?
feldrim · 10 months ago
In finance, these are called systemic risks. Some nodes in the overall network of financial institutions that can trigger cascading failures, hence affecting the whole system.
cmrdporcupine · 10 months ago
Are you old enough to remember the great northeast blackout of 2003? A single software bug brought the whole grid down for 55 million people and we didn't have power fully restored here in southern Ontario until 3 days later.

It was nice to see the stars though.

jcranmer · 10 months ago
It wasn't a single software bug. Broadly, what happened was this:

* A power company failed to trim trees from its power lines properly. Four of their major transmission lines failed in one afternoon due to shorting out via tree.

* The software bug you mentioned caused a failure to alarm the power company of two of their line failures. The first and fourth failures were alarmed in real time.

* A separate issue rendered the regional grid operator's software modeling real-time grid instability inoperable for most of the afternoon. Crucially, if this had been running, the operator would have realized that the failure of one more line would have created grid instability requiring immediate action.

* The final line trip set off a cascade of overloads trips until Sammis-Star overloads and trips. This takes about 15 minutes, and analysis suggests that the Sammis-Star trip is when the blackout became inevitable.

* Sammis-Star triggers lines to fail one-by-one from east to west, until it severed every line in Ohio and Michigan. This causes a large power surge to go from Michigan through southern Ohio, Pennsylvania, New York, Ontario, back into Michigan and then into the final demand of Cleveland.

* Essentially every link along this spiral fails in the course of a few seconds, creating several grid islands. Whether these grid islands black out is dependent on the local mismatch between generation and consumption.

(Full story from https://web.archive.org/web/20120318081212/http://www.nerc.c...).

greenie_beans · 10 months ago
just gave me a good ambiance idea for a fiction, ty
Maxion · 10 months ago
Electricity infrastructure is incredibly expensive. If a main transmission line goes down you will have issues both sides of it, one side experiencing sudden increased supply which will raise the frequency which will cause power plants to trip offline.

On the other side you will see frequency drops, which will do the same.

I do not know what Chiles power production looks like, but it would be a huge challenge for any power network to deal with one of the main lines suddenly dropping out.

oliwarner · 10 months ago
It's likely not a single fault, but a fault and a subsequently failed protection mechanism. Elements of a grid usually work together, tied through frequency synchronisation with protection breakers to separate them if a part gets overloaded or fails. If a required generator or transmissions like did go down, and it and its users remained connected to the national grid, that load can very easily pull the rest down.

Many modern countries have experienced rolling black- and brown-outs in the past ~25 years. It's not an easy thing to sidestep without a lot of spare capacity and that takes money that nobody has the appetite to spend.

uzbit · 10 months ago
I was in Santiago when this happened. While working in the basement of my hostel, all the power went out, and it was pitch black. I didn't really think much of it at the time because there had been several minutes-long outages before in various places in Santiago/Chile. When I got upstairs, I noticed that I had no cellular coverage, nor did anyone else, which is when I realized that this one was different. Took about 7hrs to get the power back on in my part of town.

It's very interesting when there is ZERO internet or any other form of external communication. In hind-sight I'd say it was an interesting short simulation for the zombie apocalypse.

lenerdenator · 10 months ago
Consider this your reminder to swing by the home improvement store and buy that backup generator you've been thinking about getting for a while.

Gonna be a lot of discarded food in Chile over the next few days...

jillesvangurp · 10 months ago
Getting fuel for your generator might be a challenge. Petrol stations tend to require electricity to operate their pumps.

A less reactive and better long term plan would be investing in tech that simultaneously reduces your monthly electricity bills and makes you more resilient against grid failures: solar + battery. It's not a solution in the middle of a black out. But a great one for dealing with the next one.

I have no idea how reliable the grid in Chile is. But even the southern tip is at -52 degrees latitude, which means their winters are about as dark as at +52 latitude, like Berlin, Germany where I live where solar power is very popular Anything in between (most of the US, except Alaska) gets enough sun hours to make solar panels a very useful fallback (with seasonal limitations obviously); and you can also keep your batteries topped up with grid power too using e.g. cheap off-peak power.

A cheaper alternative could be to pick up a few portable batteries and solar panels from Amazon. You can get a nice plug and play setup that will power some essential things for a while for a few thousand dollar/euro. Running the AC off such a setup is not going to be a thing. But you could run a small fridge for a half a day or so, keep phones laptops charged, and keep the lights on. Some people buy generators just so they can top up their batteries when solar falls short and run completely off-grid otherwise.

svachalek · 10 months ago
Note that just having solar doesn't necessarily protect you in a blackout. A lot of systems won't work without having power on the main line to sync to.
mindslight · 10 months ago
Batteries are great if you can use them all of the time - eg V2G, grid load shifting, or if your utility won't buy generated solar power. Otherwise the capital cost is too high to have significant storage batteries sitting around idle waiting for an emergency.

> You can get a nice plug and play setup that will power some essential things for a while for a few thousand dollar/euro

A petrol generator that will do this can be had for like $400.

bdcravens · 10 months ago
A good hedge against not being to access petrol is a multi-fuel generator, that can use residential natural gas or propane tanks in addition to liquid fuel.
asah · 10 months ago
there's fuel stabilizer you can buy to store fuel for a year+, just calendar a reminder to use/replace once a year (good idea for other reasons).
AdamN · 10 months ago
This seems pretty wasteful. Going without electricity for a day really isn't such a burden and a freezer can last pretty long or if it doesn't there's rarely much of a problem with waste if you just eat the food in order.
lenerdenator · 10 months ago
The problem isn't going without electricity for a day, it's that you don't know if you will only have to do a day.

Also keeping people out of the damn fridge. But that can be modified through other means.

canadiantim · 10 months ago
Risk management isn't a waste
SamPatt · 10 months ago
A backup generator is wasteful?

Where do you live?

noisy_boy · 10 months ago
Or a lot of impromptu parties...
ourmandave · 10 months ago
By candle light.
BLubliulu · 10 months ago
Get an EV instead. Use it for bi-directional charging, buy PV to reduce your dependency and energy bill and a heat pump to be independent of oil/gas companies.
aftbit · 10 months ago
Hardware store backup generator => $500 - $2000

vs

EV => $8000 - $40000

Bidirectional charging infra => $2000 - $5000

PV => $5000 - $15000

Heat pump => $5000 - $10000

That's an awful lot of money that you're proposing people spend in order to cover a rare occurrence. Of course there's day-to-day value in having all of that which a backup generator cannot provide, but in a power outage, you'd probably rather just have the cheap gas generator, and maybe a $1000-ish "solar generator" (i.e. battery pack with inverter) that you can use to load-shift the generator. Run the genny during the day to charge the battery; run the fridge, lights, and phone charger from the battery overnight.

mikestew · 10 months ago
Get an EV instead

Be choosy about the EV, not all of them have this feature (Tesla, for instance, doesn’t last I checked). That said, after we got our Hyundai (and the vehicle-to-load adapter), I sold our generator to a neighbor. Less than a year later, we both got to test our electrical backups.

Hopefully you were going to buy an EV anyway, because a nice generator is about $1000. A Hyundai Ioniq 5 is considerably more than that.

officialchicken · 10 months ago
That's a terrible suggestion. A car is 20x the price of a generator while and at least 10x larger. A car can't power an entire household for days on end using a few gallons of gasoline per day, etc. A car is a transportation device, not a stationary energy generation machine designed as a backup in case of power failure.
kibwen · 10 months ago
Remember that having PV isn't enough, you also need to have the circuitry to disconnect your house from the grid, because your personal panels probably don't have the capacity to power the entire grid on their own (plus it wouldn't be safe for line workers to have random pockets of energy in a grid that's supposed to be down).
ASalazarMX · 10 months ago
The backup generator that needs periodic testing and maintenance whether you use it or not, and accommodations to store fuel safely if you want to be in full prep mode?

If the electrical grid fails to the point that you're days or weeks without power, having your food unfrozen is going to be the least of your problems.

lenerdenator · 10 months ago
> The backup generator that needs periodic testing and maintenance whether you use it or not, and accommodations to store fuel safely if you want to be in full prep mode?

Yep.

> If the electrical grid fails to the point that you're days or weeks without power, having your food unfrozen is going to be the least of your problems.

Happens several times a decade on the Atlantic coast of the US. Well, the days do, at least. Weeks, probably not.

iAMkenough · 10 months ago
When my family had a windstorm take out our lines for an entire week and we had to toss food, we opted to go for a battery-operated cooler/freezer that charges off 100W solar. Bonus is that we can take it camping too.
Galatians4_16 · 10 months ago
Feed it to chickens and hogs!
bob1029 · 10 months ago
A 2200W inverter generator + 2000WH battery can carry most households for a week before you need to do any serious maintenance on the unit. About 3 gallons of fuel per day and a mid-week oil change is all you need during operation. You can often run off the battery alone while everyone sleeps at night.

The battery you can keep charged, and the generator you can keep new in the box until you actually think you'll be in a prolonged outage. You can run fridge, internet, lamps off the battery for a solid day. The #1 thing that screws people with the small engines is maintenance after some initial amount of usage. Gasoline turns into jello in the fuel system after a ~year in storage. Oil needs to be changed frequently. Don't run gas through that small engine unless you really think you'll need it. LP/LNG is much cleaner, but can be harder to obtain and use.

I think of these small inverter generators as a one use/emergency item. Once I fire it up, it's on a 200 hour death timer. It only costs around $600 for a brand new unit. Catastrophic, multi-day outages don't happen often.

asah · 10 months ago
meh - "By Wednesday, the government said that 90% of homes and businesses affected by the blackout had had their electricity restored, according to the Chilean National Electric Coordinator."
TZubiri · 10 months ago
Simultaneously Argentina is like:

"There's too much bureocracy in customs for electrical imports, make it laxer, also no restriction on plug types

eric__cartman · 10 months ago
Honestly I don't see much of a problem if this is applied to imports of single items by an end user. I used to be that I had trouble importing some device partly because the power supply was not certified by the local regulatory entities. Most of what people import in single quantities are electronics with switch mode power supplies that work from 100-240v and at 50/60hz. I doubt many people are importing a hairdryer or a toaster. Personally if a power supply is approved by the FCC or some other important entity I consider it good enough for my personal use, even if it has a foreign plug.

It is a problem for importing large quantities to resell though, I'm not defending the ability to import 100s of death traps and sell them to people.

TZubiri · 10 months ago
As I understand it, the most frequent type of importation by item is wholesale, we can be talking about the import of 500k phone chargers.

I think plug types are not a great risk as users will usually not want that. But in my head the risk is that we import 500k of something that technically works, but is off spec by 10V or 10hz, or the tolerance specs are too wide or too small. It's obvious how too small of a tolerance can cause issues, but too wide isn't ideal either, as there's tradeoffs, you end up importing swiss knife products. Which makes sense for big expensive electronics, but stuff like phone chargers? Subterranean or aerial cabling?

The task of verifying the quality of something is distinct from the task of verifying that it conforms to the local standards. And I wouldn't put it past the cargo culting governments to figure that if it's good enough for the US it's good enough for us.

IncreasePosts · 10 months ago
Well, no country-wide black outs so far.
polsaker · 10 months ago
We had one in 2018 tho :-)
tqi · 10 months ago
Going to be some incredible long exposure night photography in urban areas that comes out of this
CSMastermind · 10 months ago
Does anyone know what the root cause is?
dietr1ch · 10 months ago
The transmission company that triggered this claims a safety mechanism misfired, taking down the main and backup lines. After this 200km section shut down it triggered cascading failures. I heard on the radio that one substation exploded, but didn't find news about it[^1].

Power was partially restored within 44 minutes, but it was more like 2-6hrs before it was back and stable depending on the area.

---

I'm not too mad at the initial outage, this kind of thing happens, but I'm ashamed of how many emergency contingencies didn't work great. In Chile we kind of expect a natural disaster to take down power or communications in a large area without much warning due to earthquakes. For instance power is meant to go down locally after a ~7 M_{W} earthquake as a safety feature, so it's going to go down even if it's no one's fault, but the protocols and safety nets didn't work great. Traffic was a mess in cities, particularly Santiago which heavily relies on the subway to get people around, some critical infrastructure had no backup power (Mobile antennas, few Hospitals). Some people reacted poorly IMO, many went to fill up their gas tanks when it made absolutely no sense to me. I guess we really need solar to spread more so people don't even think about using their cars to charge their phones.

I'm also annoyed that most modern cellphones have no AM/FM receivers. Mobile coverage and networks are good, but they stands no chance if everyone suddenly tries to use them.

[^1]: I guess a mix of a power surge together with high temperatures triggered this. I've seen a small transformer explode as it got shorted and light up the cooling oil.

webdoodle · 10 months ago
I'm sure the solar proton storm that hit right as Chile was sun facing had nothing to do with it. /s

https://spaceweather.com/archive.php?view=1&day=25&month=02&...

FYI - The 3rd worlds power grid, mostly near the equator isn't as resilient to solar storms as the U.S. and other nations in the Northern Hemisphere who've had to harden there grids against such things due to repeated solar incidents. Often because they have no redundancy when a line surges.

bigtimesink · 10 months ago
Chile is further south than you think and more developed than you think. Unlike most of Latin America, the tap water in Chile is generally safe to drink, so their electric grid might also be better than you expect.

Dead Comment