This sounds big enough to require a black start. Unfortunately, those are slow and difficult.
If an entire nation trips offline then every generator station disconnects itself from the grid and the grid itself snaps apart into islands. To bring it back you have to disconnect consumer loads and then re-energize a small set of plants that have dedicated black start capability. Thermal plants require energy to start up and renewables require external sources of inertia for frequency stabilization, so this usually requires turning on a small diesel generator that creates enough power to bootstrap a bigger generator and so on up until there's enough electricity to start the plant itself. With that back online the power from it can be used to re-energize other plants that lack black start capability in a chain until you have a series of isolated islands. Those islands then have to be synchronized and reconnected, whilst simultaneously bringing load online in large blocks.
The whole thing is planned for, but you can't really rehearse for it. During a black start the grid is highly unstable. If something goes wrong then it can trip out again during the restart, sending you back to the beginning. It's especially likely if the original blackout caused undetected equipment damage, or if it was caused by such damage.
In the UK contingency planning assumes a black start could take up to 72 hours, although if things go well it would be faster. It's one reason it's a good idea to always have some cash at home.
In another life I worked as an engineer commissioning oil rigs and I’ve seen how tricky even a small-scale black start can be. On a rig, we simulate total power loss and have to hand-crank a tiny air compressor just to start a small emergency generator, which then powers the compressors needed to fire up the big ~7MW main generators. It's a delicate chain reaction — and that's just for one isolated platform.
A full grid black start is orders of magnitude more complex. You’re not just reviving one machine — you’re trying to bring back entire islands of infrastructure, synchronize them perfectly, and pray nothing trips out along the way. Watching a rig wake up is impressive. Restarting a whole country’s grid is heroic.
I remember talking to my ex's dad about his job, which involved planning refuels of a large nuclear-powered generation station in the Lower Midwest.
The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true. It's a system basically built to produce "common accidents". It's amazing that it doesn't on a regular basis.
I love the "analog" handcranked air compressor to 7MW generator escalation, it really captures human ingenuity.
I wonder however how being part of the "continental Europe synchronous grid" affects this, and how it isolates to Portugal and Spain like this.
But yeah there are a lot of capacitors that want juice on startup that happily kills any attempt to restore power. My father had "a lot" of PA speakers at home and when we tripped the 3680w breaker (16A 220v) we had to kill some gear to get it back up again. I'm also very sure we had 230v because I lived close to the company I worked for and we ran small scale DC operations so I could monitor input voltage and frequency on SNMP so through work I had "perfect amateur" monitoring of our local grid. Just for fun I got notifications if the frequency dropped more than .1 and it happened, but rarely. Hardly ever above though since that's calibrated over time like Google handle NTP leap seconds.
> have to hand-crank a tiny air compressor just to start a small emergency generator
Similarly, the US Navy maintains banks of pressurized air flasks to air-start emergency diesels. Total Capacity being some multiple of the required single-start capacity
I can appreciate the ability to revert to hand cranking an air compressor, yet I can't help but feel that the 99.99% of events, you'd be better served with keeping a two stroke gas engine ready to go. Air compressors tend to have parts just as or more vulnerable to environmental factors, and you get a lot more power for less elbow grease out of a two stroke.
Black out on a rig or ship is very different to black start of a national electricity grid.
Most vessels will experience a blackout periodically and the emergency generator start fine, normally on electric or stored air start, and then the main generators will come up fine. It's really not delicate, complex or tricky - some vessels have black outs happen very often, and those that don't will test it periodically. There will also be a procedure to do it manually should automation fail.
There are air starters on some emergency generators that need handling pumping. These will also get tested periodically.
The most complex situation during black out restoration would be manual synchronisation of generators but this is nothing compared to a black start.
bringing islands together requires one to synchronize both -- frequency and phase. It is super difficult for large generators and transmission lines. transient heat dissipation can be a real bummer.
The fewer resources we dedicate to grid resilience and modernization, the harder black starts become. And as grids get more complex and interdependent, recovering from total failure becomes exponentially harder.
A rare but sobering opportunity to reflect on something we usually take for granted: electricity.
We live in societies where everything depends on the grid — from logistics and healthcare to communications and financial systems. And yet, public awareness of the infrastructure behind it is shockingly low. We tend to notice the power grid only when it breaks.
We’ve neglected it for decades. In many regions, burying power lines is dismissed as “too expensive.” But compare that cost to the consequences of grid collapse in extreme weather, cyberattacks, or even solar storms — the stakes are existential. High-impact, low-frequency events are easy to ignore until they’re not.
Just to highlight this: the last significant power outage in Western/middle Europe was 2003. [1]
That's 20 years without any significant problems in the grid, apart from small localized outages.
It's not hard to start taking things for granted if it works perfectly for 20 years.
Many people don't even have cash anymore, either in their wallet or at home. In case of a longer power outage a significant part of the population might not even be able to buy food for days.
Yeah, this is the turkey’s dilemma - life on a farm is a lot better than life in the wild for 51 out of the 52 weeks of the year.
Most of our modern economy and systems are built to reduce redundancy and buffers - ever since the era of “just in time” manufacturing, we’ve done our best to strip out any “fat” from our systems to reduce costs. Consequently, any time we face anything but the most idealized conditions, the whole system collapses.
The problem is that, culturally, we’re extremely short-termist- normally I’d take this occasion to dunk on MBAs, and they deserve it, but broadly as a people we’re bad at recognizing just how far down the road you need to kick a can so you’re not the one who has to deal with it next time and we’ve gotten pretty lazy about actually doing the work required to build something durable.
Honest question, are we better off in the long run, and is it a better solution, to decentralize energy generation and make more smaller grids rather than linking them all up? This isn't to say completely getting rid of the ability to transfer between the smaller grids to assist with power disruptions but to decouple and make it less likely for catastrophic "global" failures like this.
We're slowly reaching this point with the internet too.
I feel like to many technologists, the internet is still "the place you go to to play games and chat with friends", just like it was 20 years ago. Even if our brains know it isn't true, our hearts still feel that way.
I sometimes feel like the countries cutting off internet access during high school final exams have a point. If you know the internet will be off and on a few days a year, your systems will be designed accordingly, and if anything breaks, you'll notice quickly and during a low-stakes situation.
Maybe a good reason (in parts of the world where this is practical) to have some solar + battery storage. Doesn't even need to fully replace grid power, just enough to run the barebones when the grid goes out.
Interestingly it seems that the black start drill is considering a smaller zone of impact than what has happened here.
Also I suspect there is far more renewables on the grid now than in 2016.
This is potentially the first real black start of a grid with high renewable (solar/wind) penetration that I am aware of. Black starts with grids like this I imagine are much more technically challenging because you have generation coming on the grid (or not coming on) that you don't expect and you have to hope all the equipment is working correctly on "(semi)-distributed" generation assets which probably don't have the same level of technical oversight that a major gas/coal/nuclear/hydro plant does.
I put in another comment about the 2019 outage which was happened because a trip on a 400kV line caused a giant offshore wind farm to trip because its voltage regulator detected a problem it shouldn't have tripped the entire wind output over.
Eg: if you are doing a black start and then suddenly a bunch of smallish ~10MW solar farms start producing and feeding back in "automatically", you could then cause another trip because there isn't enough load for that. Same with rooftop solar.
This is potentially the first real black start of a grid with high renewable (solar/wind) penetration that I am aware of.
The South Australia System Black in 2016 would count - SA already had high wind and rooftop solar penetration back then. There's a detailed report here if you're interested:
Practical Engineering did a really great video a few years ago on why black starts are hard, complete with a tabletop demo about the physics of synchronizing large spinning generators: https://www.youtube.com/watch?v=uOSnQM1Zu4w
Yes because they have to bring it all back up in phases so that they only face the load spike* from one interconnect at a time, which can take some time and can fail if there’s unknown damage like the GP said.
It really depends on the region though because almost all large hydroelectric dams are designed to be primary black-start sources to restore interconnects and get other power plants back up quickly in phase with the dam. i.e. in the US 40% of the country has them so it’s relatively easy to do. The hardest part is usually the messy human coordination bit because none of this stuff is automated (or possible even automatable).
* the load spike from everyone’s motors and compressors booting up at the same time
Spain is but Portugal is only connected to Spain and they are currently doing a full black start.
For Spain the external power and synchronization can come from France rather than generators which will help, but the process and complexities are still mostly the same. Call it a dark start, perhaps.
> A black start is the process of restoring an electric power station, a part of an electric grid or an industrial plant, to operation without relying on the external electric power transmission network to recover from a total or partial shutdown.[1]
The frequency aspect of a black start is presumably a bit easier in Europe because there's an interconnected synchronous grid so they can bootstrap it from France essentially.
It's far more problematic for the UK because all the interconnects are DC.
I was recently told by an electrical engineering lecturer that the black start plan here in Ireland is to use the DC interconnectors with the UK to provide startup power to a synchronous generator.
If your Factory uses too much power, theres not enough energy to run the power plants generation, which decreases your power production. Death spiraling until theres no power.
You have to disconnect the factory, and independently power your power plants back up until you have enough energy production to connect your factory up again.
Capacitor circuit network warning for alarm for "main grid is drawing on this bank - bad things may happen if capacity is not increased."
Another "trick" is those burner inserters are black start capable. They can pick up fuel and feed themselves to keep running without an electrical network.
I also tend to put Schmitt triggers in low priority areas. They've got a battery on the main grid next to them and if the battery drops below 50% power they remain off until it goes back above 75% power.
In my server I hooked up a sound alarm to a set of capacitors. Too low of a charge indicates higher power consumption than production, allowing you to unplug certain low priority loads. I also have some emergency coal generators ready to go at the flick of a switch if needed.
Same with Satifactory, The larger powerplants need a lot of energy for their infrastructure to run and an overload will trip breakers and shut the whole grid down, a naively designed grid death spirals very easy. My factory was needing increasingly complicated black start systems so I started putting the powerplant infrastructure on a self contained islands. something a factory overload would not trip, it was something like one coal power plant can run the machinery needed for itself and 8 grid power plants.
One of the Administrators of the REN, the Portuguese electric supplier is currently giving a press conference. Confirmed they are in scenario of restart from black start.
- Cause of event not known yet.
- They noticed power oscillations from the Spanish grid that tripped safety mechanisms in the Portuguese grid. At the time, due to the cheaper prices, the Portuguese grid was in a state of importing electricity from Spain.
- They are bringing up multiple power systems and the Portuguese grid is able to supply 100% of needs if required. It was not configured in such a state at the moment of event.
- They had to restart the black start more than once, since while starting, noticed instabilities in some sectors that forced them to restart the process.
- Time for full recovery unknown at this time, but it will take at least 24 hours.
We are beginning to recover power in the north and south of the peninsula, which is key to gradually addressing the electricity supply. This process involves the gradual energization of the transmission grid as the generating units are connected.
I see load dropping to zero on that graph, or rather, load data disappears an hour ago.
If the grid frequency goes too far out of range then power stations trip automatically, it's not an explicit decision anyone takes and it doesn't balance load, quite the opposite. A station tripping makes the problem worse as the frequency drops even further as the load gets shared between the remaining stations, which is why grids experience cascading failure. The disconnection into islands is a defense mechanism designed to stop equipment being too badly damaged and to isolate the outage.
It's not just about the power. System components cannot be brought to operating temperatures, speeds and pressures faster than mechanical tolerances allow. If a thermal plant is cold & dark, it can take days to ramp it to full production.
That's true of some kinds of thermal generators, but not all. Simple cycle gas turbines can come up very quickly (think jet engines). Or your car's engine.
A true black start has several factors (which make it difficult and notable):
1. The grid has to fully collapse with no possibility of being rescued by interconnection
2. As a result, a generation asset has to be started without external power or a grid frequency to synch to
3. An asset capable of this is usually a small one connected to a lower voltage network that has to then backfeed the higher voltage one
4. Due to the difficulty of balancing supply/demand during the process, the frequency can fluctuate violently with a high risk of tripping the system offline again
None of this applies in yesterday's case:
The rest of the European synchronous grid is working just fine.
News reports stated Spain restored power by reconnecting to France and Morocco.
By reestablishing the HV network first, they can directly restart the largest generation asset with normal procedures.
As they bring more and more load or generation online, there's little risk of big frequency fluctuations because the wider grid can absorb that.
Just to add, I was at a University campus when the entire building's electrics went out and there was a significant pull due to relatively powerful computers in every room. They initially tried to bring the building back online altogether and failed. Then they tried to bring it back in sections and failed too. In the end they ended up going into each lab, turning every computer off at the wall to bring each lab's power back, and then turn each computer on one by one.
I can only imagine the difficulty of bringing large parts of the grid back online, that rush current must be immense.
Yup. I used to work with a factory that had a bunch of really big machines. Turn everything on at once and the transformer out on the pole self-destructed. Note that the breakers didn't pop--the startup transient was short enough. The power company wasn't happy.
Or look at Apollo 13. The astronauts had turned off everything possible because they had lost their generator and only had their batteries. And it took a lot of furious planning by the guys on the ground to come up with a sequence of turning things back on that didn't cause the peak draw to go too high. Can't go too fast or it trips. Can't start too early because the power is limited, but can't start too late because the systems have to be up when they hit the atmosphere.
I did a project where we predicted transformer failure (they blow up!) from changes in the oil that they have in them (its insulative properties suddenly change). This was 30 years ago so it's all a bit fuzzy, but the one thing that really stuck with me was the story that the SME that we were working with had about the UK grid teetering on the edge of failure during the "great storm" of 1987. His telling was that they really were unsure if they would get it back at all!
> It's one reason it's a good idea to always have some cash at home.
More than cash it was important yesterday to have the following in case it would have lasted longer:
- a battery powered am/fm radio with spare new batteries
- some candles and matches
- food reserves for a few days that don't need refrigeration: bread, anything in can, pasta, rice...
- some kind of gaz or alcohol stove, dry wood or bbq charcoal: you can always make a fire in the middle of the street where there is no risk of burning things around.
- water reserve (I always have like 24L of drinking water) and since I hate waste I regularly fill jerrycans when waiting for hot water in the shower that I use for manual washes (kitchenware
or gears).
Does solar power make this process easier or harder? I know that with thermal plants you have a spinning mass that you have to synchronize, and phase shift is used to assess how hard the plant is working (and whether to trip a disconnect as we see here)
But with solar, how is the synchronization provided? In like a giant buck? Or in software somehow? Does the phase shift matter as much as in the electromechanical systems?
My intuition is that solar would make the grid harder to keep stable (smaller mass spinning in sync) but also may offer more knobs to control things (big DC source that you can toggle on/off instantly.. as long as sun is out). But I don’t actually know.
Mike_hearn's comment was grey but was correct: phase following is indeed done through software in the inverter. Phase matching is still required, wherever the phase difference is not zero there is a deadweight loss of power as heat.
Currently the main driver of battery deployments is not so much energy price time arbitrage as "fast frequency fresponse": you can get paid for providing battery stabilization to the grid.
Most solar and wind plants follow the inertial lead of the thermal plants. They can't synchronize without enough thermal generation being online. Supposedly there are efforts to change that, I don't know enough about grid engineering to say how far along that might be in Spain.
> But with solar, how is the synchronization provided? In like a giant buck? Or in software somehow? Does the phase shift matter as much as in the electromechanical systems?
If you mean how does solar act to reinforce the grid: search for terms like "grid forming inverter vs. grid following inverter" though not all generators are the same in terms of how much resilience they add to the grid, esp. w.r.t. the inertia they do or do not add. See e.g. https://www.greentechmedia.com/squared/dispatches-from-the-g...
Harder mostly, See the frequency is set by huge rotating masses in the form of generators, and when the supply and demand is matched the frequency and voltage are stable, when demand dramatically increases it pulls the frequency and voltage down, which is effectively slowing the generators down as load / magnetic drag increases with current drawn. Having large inertial masses spinning actually helps smooth out frequency changes. whilst large solar farms can and do syncronise with the grid, they are reactive and do not add the same smoothing effect as humungous spinning masses.
Low Grid frequency & voltage can cause an increase in current & heating of transmission lines and conductors and can damage the expensive things, this is why these systems trip out automatically at low frequency or low voltage, and why load shedding is necessary
I'd say a little harder to negligible now, but potentially way easier in the future.
The main difficulty is that the software of grid-following inverters tend to make them trip out very suddenly if the grid parameters get too far out of spec (they will only follow the grid so far), but once the grid is good they basically instantly synchronise.
But all large solar farms are likely to be mandated to switch from grid following to grid forming inverters eventually which will make them beneficial for grid security because they will help provide 'virtual intertia' that looks exactly the same to the rest of the grid as spinning mass does.
As the press release you linked points out, the black start plan Spain trains on uses nuclear energy supplied from France to re-energize the power plants.
"Luckily", France is at an historically high level of production capacity at the moment and the connection between the two countries was reestablished fast.
According to RTE (French network manager), the interconnection was maxed yesterday at around 3GW of power.
Sadly, while Spain is part of CESA, it's not very well connected. I wouldn't be surprised if one the takeaway from the whole incident is that more interconnections are needed.
Ukraine is interesting in this context because there are so many generators. In the richer parts of Odesa I've even found it hard sometimes to tell whether or not the grid power is on as literally every single building had sufficient backup generators to keep the lights on (also, many big businesses seemed to run their generators even when the power was on, I presume as a civic-minded way to add generation capacity overall and avoid interruptions when the power flips on and off).
And if you are a mere mortal in this world, play Factorio : Space age expansion on the planet Aquilo. To learn the precious importance of reliable multi stage power bootstrapping
Power never went out in a country completely. At the lowest point consumption was ~40% of normal for that time of day.
Ukraine went through many black starts in the first winter of Russian strikes against energy. I guess they built a skill of recovering it quickly enough that it started happening faster and easier every next time.
I would think that renewable infrastructure could be the fix, at least if you start installing larger battery capacity to meet renewable store and usage shifts, the grid essentially is installing the resources that can also be used to respond to & contain sudden source losses and prevent cascades.
I wonder if someone could build a realistic scenario into a game -- let's say some sort of smaller scale black start for, say, a space station. And throw in a unknown computer architecture for the in-game computers so that players need to RTFM to figure out how it works.
I took down the servers though, so you probably can't easily try it. I don't know if I added a way to configure the lobby server. I should have! It's open source though. And there is a video about that thing on my YouTube: https://www.youtube.com/watch?v=6TPgfa7LbiI
The game is bad and nothing of what we planned on doing actually made it into the game. The video is long and boring too. But maybe someone finds this cool and is inspired by this and makes a game like this.
The first 15 minutes of the game were actually about getting the ship moving, first by reading the manuals of half a dozen different ship systems and then following some procedure outlined in those manuals (parts of which were simply incorrect), maybe having to do some things in sync with your other players and stuff like that. I think it would have been cool to add multiple reactors and start them up in sync and stuff. The different ship systems were actually Lua programs that interacted via a message bus. So kind of a unknown computer architecture?
For maybe the first 24 hours at a grocery store, and then not so sure. Would your neighbors sell you supplies and food? Maybe not? And so many places now depend on cashless transactions and doubtful they have pen, paper, lockbox, and safe as a contingency plan.
It would be essentially the same thing as a grid black start, except that the first breaker to close has the European grid on its primary side, instead of a freshly started generator under your control.
The complex process of configuring the transmission network to bring grid power to each power plant in succession is the same.
The continental part of the EU runs on one synchronized grid. The Nordics (except half of Denmark and, uh, Iceland, Greenland etc) runs on a separate synchronized grid.
I'm confused. Would the start ever have to be truly black? Wouldn't water always be driving hydroelectric turbines, generating some electricity? Solar panels generate electricity without requiring input. I understand that synchronizing AC is not trivial, I'm only questioning the part about whether the start is truly black.
The tills have keys to manually open them, and you can just record transactions on pen-and-paper if needed and enter them later. I've seen plenty of businesses do exactly that during power outages. Not as fast. But totally doable.
Stores with tills and freezers etc will have power for the tills but the backing network for payments probably won't be up. That's the concern with being cashless. They can accept cashxl, but no one has any.
I was able to buy some groceries and pay with card. The tills had a battery backup and the network infrastructure that supports card payments was apparently working.
That said, lots of people hit the cafés and had to resource to cash payments. There was also lots of people buying bottled water at the shops.
So basically, you could divide people in two groups. Those that took it like an extra Sunday, and those that took it like the beginning of a war or something :-D
>It's one reason it's a good idea to always have some cash at home.
Most places are so dependent upon electricity that they can't even take cash during a blackout. And they don't even have the mechanical machines to take a credit card imprint anymore.
The last of my raised number credit cards went away last month. Those old machines will only get a blank rectangle from me. Sad, because I did actually use one of those about 3 years ago when a rural gas station had power but no network.
I actually have no real clue how and where Spain/Portugal is connected to rest of Europe but could they also restart North to south with help from the grid from France?
The grid uses AC (not DC), running at 50 Hz (cycles per second). So the voltage is going up and down at that frequency, in a sine-wave pattern.
If you try to connect another generator to the grid, it needs to be at the same point (phase) in the sine-wave cycle, so that its power contribution is added, not subtracted.
If it's not in sync, huge currents can flow, causing damage. Sort of like connecting jumper cables backwards.
The bulk of the power grid is alternating current (AC), and the frequency of the resulting sine wave needs to be synchronized with the other parts of the grid it is connected to.
I'm going with: never attribute to malice what can be explained by ... an incredibly complex system that can fall over even if no-one's being stupid. I would want very strong evidence before I believe this is an attack.
I remember the day when the Swiss railway power network went down for a day (in 2005) because one power line was down for maintenance and someone pressed the wrong button and produced a short circuit somewhere else. It's a bit like the incidents in planes were one engine has a problem and the crew shut down the other one by mistake.
That wasn't tree falling in Ohio, that was overloaded line sagged and shorted into a tree, compounded with several other factors that contributed to the grid instability and the inability of the grid operator to realize how unstable the grid was.
Having lived through the Texas electricity fiasco in 2021, I would blame cost cutting, the reckless drive for “efficiency”, and maximizing shareholder value.
In Texas, the electric providers cut staff and maintenance to maximize shareholder value. They will not have redundant systems and redundant plants out of the goodness of their hearts. The Texas marketplace actually allowed them in the odd event of an outage to charge astronomical spot prices thinking this will incentivize them to have redundant systems. This was a foolish fantasy.
Probably didn't help that before the outage hit, Spain was running its grid with very little dispatchable spinning generation, and therefore not much inertia.
Might as well mean that they suspect the culprit, but they're not going to do anything anyway for this specific culprit, so let's all pretend it was an accident. Why make yourself look like a fool?
They already became a laughing stock once for promising the "strongest possible response" for the Nord Stream 2 sabotage [1].
> but I thought they were incredibly hardened, with backups and contingencies in depth
Some are harder than others, and some have random flaws which nobody can really predict.
Spain seems in the transition to renewables, so it's possible that they have some flaws because they are still in the process, or because it's something which never happened before and is unknown territory. Also, Spain had some economic problems in the last decade, maybe someone build to cheap or was even cheating somewhere.
> Are the grids at this scale really this brittle? Would there be a death toll from this?
Hospitals should have backup-systems. Traffic should be able to stop in time. I guess the most problematic parts are people stuck in elevators and other spaces which only open electrical, as also the loss of cellular phone-connections for calling helpers.
Hardening focuses first on not damaging equipment and second on providing energy. If things go wrong quickly enough you don't have time to react, because after a power plant disconnects you get sudden bumps in load that can trigger a chain failure near the original point. The last time it happened in Europe was in 2003, which isn't too shoddy.
What Spain's PM is saying (and is being reported by Spanish newspapers), 15 gigawatts of energy production went down all in 5 seconds. Hardening to tolerate that much of a change, that fast is a more extreme event than a grid the size of Spain's preps for.
> I'm going with: never attribute to malice what can be explained by ... an incredibly complex system that can fall over even if no-one's being stupid. I would want very strong evidence before I believe this is an attack.
I think we should prepare for the worst though. It's wrong to assume it's not an attack too, and until we can conclude it's not an attack we should be prepared to deal with the possible consequences and act accordingly.
I think James Burke's classic talking about the fragility of our complex interdependent systems starting the episode from the 1965 Northeast blackout is still relevant and an interesting watch: https://www.youtube.com/watch?v=XetplHcM7aQ
Every single retrospective I've seen on major power outages is full of miscommunication and people who would be fired if they worked in fast food. As well as blaming the tech boogeyman ("Our systems were on the fritz this morning")
A few months ago I was hit by a blackout literally the second I was about to start delivering a company-wide talk on AI. Everything went out - Internet, mobile networks, street lighting, the lot.
We're a remote business so it seemed like I'd just rudely dropped off the call, but as everything was down I couldn't let people know what'd happened.
Apparently it was caused by botched maintenance work affecting 30,000 houses, but the timing was so perfect I can't help thinking it was because our AGI overlords really didn't want me to deliver that talk for some reason.
I don't think we're able to tell from the data if one is the cause of the other, are we? Since if production was lost, load would have to be shedded to balance the grid, and if load was lost (e.g. due to a transmission failure), production would have to be disconnected to balance the grid.
That started from a combination of a lightning strike and generator trip, but turned into a local cascade failure as lots of distributed generation noticed that the frequency was under 49Hz and disconnected itself. I suspect the Spanish situation will be similar - inability to properly contain a frequency excursion, resulting in widespread generator trips.
(I suspect this is going to restart a whole bunch of acrimony about existing pain points like grid maintenance, renewables, domestic solar, and so on, probably with the usual suspects popping up to blame renewables)
Wow that's interesting, solar was absolutely peak at the time. Maybe related to poor frequency leading from solar powerplants coupled with some other spike that drew it out of sync?
Interesting - when you realize that nuclear energy is not as reliable as solar (nuclear disappears completely around 1 pm, while solar keeps a stable input share).
Looks like they're using French nuclear to help black start the grid [1]. Solar and wind systems could have caused the instability issues that led to the Spanish Nuclear requiring isolation. It will be interesting to see the final investigation, but my bet is that "induced atmospheric vibration" is a PR deflection from a badly designed and operated system [2].
[1] https://transparency.entsoe.eu/transmission-domain/physicalF...
[2] I'm not necessarily blaming the engineers, but the politicians who force those engineers to put square pegs in round holes. For example, I can imagine politicians making a short term decision to skimp on energy storage while increasing renewable penetration. Surely renewable systems must be less reliable without storage given the lack of rotational inertia?
Apparently a local grid overload near France and a cascading failure down the Spanish network, but radio and newspapers don’t agree on root cause. Of course there is a lot of noise.
For instance, one reporter asked one of the government flunkies whether it could be a cyberattack and they turned his noncommittal “maybe, we don’t know” into “government says cyberattack may be ongoing”.
Be careful of idiot reporters out there.
Edit: I’m listening to another radio interview where they are outlining the plans to bring online Portuguese dams and thermal generators over the next few hours, progressively unplugging from the Spanish supply (fortunately we have enough of those, apparently).
It should take 3-4 hours to get everything balanced with only national supplies, and they will restore power from North to South.
Key points that started it were (you can see the chain of events in the doc):
2.4.1. At 16:52:33 on Friday 9 August 2019, a lightning strike caused a fault on the Eaton Socon – Wymondley 400kV line. This is not unusual and was rectified
within 80 milliseconds (ms)
2.4.2. The fault affected the local distribution networks and approximately 150MW of distributed generation disconnected from the networks or ‘tripped off’ due to a safety mechanism known as vector shift protection
2.4.3. The voltage control system at the Hornsea 1 offshore wind farm did not
respond to the impact of the fault on the transmission system as expected and
became unstable. Hornsea 1 rapidly reduced its power generation or ‘deloaded’
from 799MW to 62MW (a reduction of 737MW).
Curious question for someone familiar with power at grid-scale -- How granular is load shedding? And how is this measured / tracked?
In my head, I'm thinking of generators/plants, connected by some number of lines, to some amount of load, where there are limited disconnection points on the lines.
So how do grid operators know what amount of load will be cut if they disconnect point A123 (and the demand behind it) vs point B456?
Is this done sort-of-blind? Or is there continual measurement? (e.g. there's XYZ MW of load behind A123 as of 2:36pm)
I was going to say something similar. I live in Portugal and I've heard a lot of panic/fear mongering, mainly from the techies in the co-working space I was working on and expats.
(apologies for singling out these specific groups of people - my point is that it might be worth to put down news sources like xitter, and read AP/translated local Portuguese news)
I experienced it first hand in Madrid. This was much scarier than I would have imagined.
News travelled extremely slow: phone coverage was just barely enough to receive a couple text messages every 15 minutes or so. News spread on the street, I even saw a group of 20 people hunched around someone owning a hand-held radio in the streets.
Just before power was restored, things started to get worse, as the phone coverage went completely out (presumably batteries were depleted). People were in between enjoying the work-free day, and starting to worry about how tomorrow would look like if power didn't come back.
When the northeast blackout hit in 2003 in NYC I dont remember any panic. We still had house phones and they still worked in the blackout thanks to telcos being legally obligated to give a shit about reliability.
I stopped by a friends house and we then went on a walk. Some stores were open and cash was accepted. We hung out later that night and had a few beers. The sky was amazing as there was next to no light pollution. Next day was totally in the dark as well and again, no panic. More beers were enjoyed.
The choice to move to electronic everything without having to give a shit about reliability is a failure of modern government. Move fast and break society for a dollar.
> thanks to telcos being legally obligated to give a shit about reliability.
Yeah, they don't need to do that anymore. Around me, enough towers have battery backups that I can count on 2 hours of coverage when utility power goes out (if it goes out late at night or early morning, there's usually coverage until 6-7 am when people start waking up and use up the rest of the power). I don't have a real landline, but the telco DSL would drop instantly with utility power so I don't have big hopes and I wasn't willing to pay $60/month to find out.
Around when I moved, stores would pull out the credit card imprint machines, but those don't work anymore because cards are flat. Cash might work, and I've got some, but I don't think many people in my community do; people don't have cash for the snack shack we run at my kid's sports, so I doubt they have it for restaurants and stores either. And we get frequent 2-4 hour power outages, at least one, usually two or three per year; and ~ 24 hour outages every few years. The snack shack runs during summer where electricity is most reliable, but I doubt people stock up on cash in the fall and use it all up before spring/summer; they probably just don't have any.
2003 was over two decades ago. Most people didn’t have mobile phones let alone smart phones. I’ve been in Lisbon today and it’s surreal being 100% cut off from friends and family back home and a big relief power is back on, we’ve become very used to and reliant on seamless instant connection. Our mindsets and how we live our lives with that instant communication is totally different to 22 years ago.
That sounds like the sensible reaction, at the time at least.
It's interesting to think about and realize how much things have changed now though, and how reliant people are on everything, and especially their tiktoks etc. working all the time.
Some of the panic is likely related to the war in Europe too, and especially the general talk about war
Back in the 90's I worked at Nortel and visited a modest size Captive Office in Los Angeles. It supported maybe 20k or 30k people. I was amazed by the field of lead-acid batteries, 1.5m high x 50m^2.
Like 10-15 years ago before VOIP I sold internet/landline services door to door for a summer. My biggest selling point was explaining to people (usually who had children) that the VOIP service they switched to would not work in an emergency where power wasn't available... Worked like a charm to get people to switch back.
Similar experience in a town in Madrid's metropolitan area.
Electricty went down (something kind of frequent). My UPS kept PC up, and alarm system with sim and small UPS mantained wifi up for an hour or so.
Scary moments started when people I was in a call with in Portugal texted 'Grid is out'. Later no phone signal nor data.
At first, it might seem people running towards supermarkets an overreaction on being without TikTok for a couple of hours, but you have to live how scary it is to experience this in Europe's current political status to know 60 million people (plus industry) in three countries are out of the grid.
If you see Snowden's film (this might not be the most trustworthy source) it is exactly how CIA's agent describes the feasable attack towards these countries. Again, not a valid source, but I'd love to understand if that could be feasable.
Experience from Portugal, near Lisbon: fake news and made-up stories traveled fast! My wife called me (before phones went out) saying someone heard on the radio that Portugal was on red alert, it was WW3 (world war 3) and I think she even mentioned "missiles"! Also someone said it was a cyberattack and all Europe was off. Lots of panic reactions, many people buying toilet paper, water, candles, sausages and other canned food.
All gas stations closed because they could not sell gasoline/diesel. Today there are lines on all gas stations, people filling their car tanks and bottles..
Oh, let me tell you about electric cars! Many people had to spend the night somewhere away from home because they could not charge their cars.. My sister (with her job's electric car) had to stay the night some 200km away from home, and since the ATMs (Multibanco) didn't work, she didn't have physical money to pay for food. Luckily a stranger paid for the food (yogurt and some cookies).
Petrol cars, because of their range, had better luck!
Pure fear and panic..
I can only blame the authorities (Portuguese/European) for not having contingency plans for keeping people informed, and thus letting fear spread like wildfire.
During a bad storm here many people used their electric cars to power their house. There were still places to charge it and then come back to the house to power it until the house was re-connected.
I think this is where F-droid and Briar have a (short-lived in this instance) chance to shine. Since Briar allows communication between phones without access to the Internet, and F-droid allows to direct transmit apps between phones as well.
I wonder what similar solutions exist in the iOS ecosystem.
we tried briar during the recent mass protests in Serbia and the bluetooth connection never worked, it was unable to connect to any of the contacts that were previosly added using the normal 4G connection.
My son was at the university outside of Barcelona, and I lost contact with him for 6 hours. He was traveling home, and I knew that the roads had no working semaphores, and there were dozens of incidents...
I lost some years today.
My son is fine, thanks to a random person (“the man with a rabbit”) who just decided to give a lift to my son and his friend to the edge of Barcelona.
I come from a world where I faced power outage for 17 hours for weeks. Each day there would be power at random time of the day. Life is just bleak. And when I see these kind of posts, I sympathize with them. You can't do anything in that situation. Modern world itself is so complex, you are left to thoughts. If this continues for a month, most people will go insanse especially those who rely on technology to stay sane.
no one has emergency fm radios there? I thought that was pretty much ubiquitous. that's the first I'd go to check what was going on I think, other than my phone
From my personal experience, mostly just in the car. Here in Portugal there was a rather well-timed rail strike today, so a lot of people who would've taken the train drove instead and had a way to get back and listen to the radio. From what I could tell from the radio reporting, handheld radios and batteries were rather quickly sold out.
I've got to give massive props to Antena 1 too, which is the national broadcaster's main radio station, who stopped all normal programming and did an all-day massive report on this situation to keep people informed. From what I could tell they didn't even run any ads during that period, just all-day reporting continuously repeating key information for people who'd just tuned in.
I did not see much fear. I was at work and it took about two hours for us to realize the outage was not just local. The cafeteria had gas burners and served everything they could to empty the refrigerators. We all at lunch and discussed whether those who lived far away (train trip) would need to sleep at work (they might have, I don't know what happened to them). I made the relatively short 75 minute walk home across the city.
The atmosphere was quasi-festive and most people were quite relaxed, enjoying an unexpected afternoon off. Younger people filled the bars which were serving everything they could. There were long lines at supermarkets and an occasional fellow toting a box of supplies, but mostly there were just huge numbers of people in the street and completely collapsed traffic flow (the police were out in force almost immediately, directing traffic). In the part of Madrid I was in about 1/4-1/3 of the population is from South America and I suspect most of them have seen this all before anyway. The only real stress I saw was from people that need a train to get home (because the trains weren't running) and a had a walk of more than 2-3 hours.
I got cell phone signal when I was near two hospitals which were fully operational.
It was interesting that almost immediately, while I was still at work, everyone said power was out in Portugal and France too. After an hour or two some were claiming problems in Germany, but this seemed already to be unfounded rumors.
Some younger people couldn't walk home because they didn't have google maps ...
What caused it?
The Portuguese prime minister, Luís Montenegro, said that the issue originated in Spain. Portugal’s REN said a “rare atmospheric phenomenon” had caused a severe imbalance in temperatures that led to the widespread shutdowns.
REN said: “Due to extreme temperature variations in the interior of Spain, there were anomalous oscillations in the very high voltage lines (400 kV), a phenomenon known as ‘induced atmospheric vibration’. These oscillations caused synchronisation failures between the electrical systems, leading to successive disturbances across the interconnected European network.”
I'm seeing some reports saying that a significant frequency oscillation happened, which triggered automatic shut-downs, which cascaded. Could an event like this have that effect?
I suppose it makes sense that it was an automatic shutdown rather than infrastructure failing on such a wide area. And then once it's shut down, a black-start is a logistical challenge as other comments have explained.
I'm also seeing some reports about it being more likely that something happened on the east side, somewhere like the Ebro valley or north across the Pyrenees. Catalonia seems to have been particularly affected, and it's on the path of important lines coming from France. High heat at noon could have caused a line to fail and short against a tree, which would be similar to the 2003 nation-wide outage in Italy.
The grid is supposed to tolerate any single failure, even under full load. Of course sometimes the first failure is a fire or equipment malfunction and the second failure is a planning failure or someone pressing the wrong button.
From Le Monde live feed, RTE (French electricity network manager) declared the issue unrelated to this fire.
"Le gestionnaire français souligne par ailleurs que cette panne n’est pas due à un incendie dans le sud de la France, entre Narbonne et Perpignan, contrairement à des informations qui circulent."
The 2003 US Northeast blackout was caused by the failure of only a few lines that shorted into trees. These line failures created grid instability that resulted, ~5 minutes later, in most of the Northeast losing power in a cascading failure.
Redundant systems have failures in one path quite often that you never know anything about. We get headlines when the failures correlate in the same timeslot.
Few years ago nearly entire day European network was sitting on N-0 due to multiple issues in Poland, caused by a heat wave and deeper root causes. There are many power plants and power lines where any further issue would cause Europe-wide blackout.
you are right, but the emphasis could use a tune-up. In California, home of world-leading tech.. there are sensors and information networks, extensive electrical power lines, heavy equipment and budgets, a lot of dry and dead tress, a history of fire. So you see that California in a way is a world-quality testing lab. and the way the information travels, and the way the information is applied, could also be world-quality .. or, world-theater for government imbecility..
Readit News
If an entire nation trips offline then every generator station disconnects itself from the grid and the grid itself snaps apart into islands. To bring it back you have to disconnect consumer loads and then re-energize a small set of plants that have dedicated black start capability. Thermal plants require energy to start up and renewables require external sources of inertia for frequency stabilization, so this usually requires turning on a small diesel generator that creates enough power to bootstrap a bigger generator and so on up until there's enough electricity to start the plant itself. With that back online the power from it can be used to re-energize other plants that lack black start capability in a chain until you have a series of isolated islands. Those islands then have to be synchronized and reconnected, whilst simultaneously bringing load online in large blocks.
The whole thing is planned for, but you can't really rehearse for it. During a black start the grid is highly unstable. If something goes wrong then it can trip out again during the restart, sending you back to the beginning. It's especially likely if the original blackout caused undetected equipment damage, or if it was caused by such damage.
In the UK contingency planning assumes a black start could take up to 72 hours, although if things go well it would be faster. It's one reason it's a good idea to always have some cash at home.
Edit: There's a press release about a 2016 black start drill in Spain/Portugal here: https://www.ree.es/en/press-office/press-release/2016/11/spa...
A full grid black start is orders of magnitude more complex. You’re not just reviving one machine — you’re trying to bring back entire islands of infrastructure, synchronize them perfectly, and pray nothing trips out along the way. Watching a rig wake up is impressive. Restarting a whole country’s grid is heroic.
The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true. It's a system basically built to produce "common accidents". It's amazing that it doesn't on a regular basis.
I wonder however how being part of the "continental Europe synchronous grid" affects this, and how it isolates to Portugal and Spain like this.
But yeah there are a lot of capacitors that want juice on startup that happily kills any attempt to restore power. My father had "a lot" of PA speakers at home and when we tripped the 3680w breaker (16A 220v) we had to kill some gear to get it back up again. I'm also very sure we had 230v because I lived close to the company I worked for and we ran small scale DC operations so I could monitor input voltage and frequency on SNMP so through work I had "perfect amateur" monitoring of our local grid. Just for fun I got notifications if the frequency dropped more than .1 and it happened, but rarely. Hardly ever above though since that's calibrated over time like Google handle NTP leap seconds.
I love infrastructure
Similarly, the US Navy maintains banks of pressurized air flasks to air-start emergency diesels. Total Capacity being some multiple of the required single-start capacity
Is that what Dr. Sattler is doing in this scene from Jurassic Park?
https://www.youtube.com/watch?v=FoW4vXnkhJw
Most vessels will experience a blackout periodically and the emergency generator start fine, normally on electric or stored air start, and then the main generators will come up fine. It's really not delicate, complex or tricky - some vessels have black outs happen very often, and those that don't will test it periodically. There will also be a procedure to do it manually should automation fail.
There are air starters on some emergency generators that need handling pumping. These will also get tested periodically.
The most complex situation during black out restoration would be manual synchronisation of generators but this is nothing compared to a black start.
Dead Comment
A rare but sobering opportunity to reflect on something we usually take for granted: electricity.
We live in societies where everything depends on the grid — from logistics and healthcare to communications and financial systems. And yet, public awareness of the infrastructure behind it is shockingly low. We tend to notice the power grid only when it breaks.
We’ve neglected it for decades. In many regions, burying power lines is dismissed as “too expensive.” But compare that cost to the consequences of grid collapse in extreme weather, cyberattacks, or even solar storms — the stakes are existential. High-impact, low-frequency events are easy to ignore until they’re not.
That's 20 years without any significant problems in the grid, apart from small localized outages.
It's not hard to start taking things for granted if it works perfectly for 20 years.
Many people don't even have cash anymore, either in their wallet or at home. In case of a longer power outage a significant part of the population might not even be able to buy food for days.
[1] https://en.wikipedia.org/wiki/2003_Italy_blackout
Most of our modern economy and systems are built to reduce redundancy and buffers - ever since the era of “just in time” manufacturing, we’ve done our best to strip out any “fat” from our systems to reduce costs. Consequently, any time we face anything but the most idealized conditions, the whole system collapses.
The problem is that, culturally, we’re extremely short-termist- normally I’d take this occasion to dunk on MBAs, and they deserve it, but broadly as a people we’re bad at recognizing just how far down the road you need to kick a can so you’re not the one who has to deal with it next time and we’ve gotten pretty lazy about actually doing the work required to build something durable.
I feel like to many technologists, the internet is still "the place you go to to play games and chat with friends", just like it was 20 years ago. Even if our brains know it isn't true, our hearts still feel that way.
I sometimes feel like the countries cutting off internet access during high school final exams have a point. If you know the internet will be off and on a few days a year, your systems will be designed accordingly, and if anything breaks, you'll notice quickly and during a low-stakes situation.
Also I suspect there is far more renewables on the grid now than in 2016.
This is potentially the first real black start of a grid with high renewable (solar/wind) penetration that I am aware of. Black starts with grids like this I imagine are much more technically challenging because you have generation coming on the grid (or not coming on) that you don't expect and you have to hope all the equipment is working correctly on "(semi)-distributed" generation assets which probably don't have the same level of technical oversight that a major gas/coal/nuclear/hydro plant does.
I put in another comment about the 2019 outage which was happened because a trip on a 400kV line caused a giant offshore wind farm to trip because its voltage regulator detected a problem it shouldn't have tripped the entire wind output over.
Eg: if you are doing a black start and then suddenly a bunch of smallish ~10MW solar farms start producing and feeding back in "automatically", you could then cause another trip because there isn't enough load for that. Same with rooftop solar.
The South Australia System Black in 2016 would count - SA already had high wind and rooftop solar penetration back then. There's a detailed report here if you're interested:
https://www.aemo.com.au/-/media/Files/Electricity/NEM/Market...
Non tied solar won't affect the grid at all. So this is a non-issue.
Grid tie requires the grid to tie to, otherwise it can't synchronize. So it stays disconnected.
You need to calculate for it but I don't think this would be a problem
Does this really qualify as "black start" when they can rely on the bigger EU grid?
It really depends on the region though because almost all large hydroelectric dams are designed to be primary black-start sources to restore interconnects and get other power plants back up quickly in phase with the dam. i.e. in the US 40% of the country has them so it’s relatively easy to do. The hardest part is usually the messy human coordination bit because none of this stuff is automated (or possible even automatable).
* the load spike from everyone’s motors and compressors booting up at the same time
For Spain the external power and synchronization can come from France rather than generators which will help, but the process and complexities are still mostly the same. Call it a dark start, perhaps.
> A black start is the process of restoring an electric power station, a part of an electric grid or an industrial plant, to operation without relying on the external electric power transmission network to recover from a total or partial shutdown.[1]
https://en.wikipedia.org/wiki/Black_start
Then again they might be less prepared precise because of the euro grid is available
It's far more problematic for the UK because all the interconnects are DC.
The UK keeping its own time just makes things easier for it IMO.
If your Factory uses too much power, theres not enough energy to run the power plants generation, which decreases your power production. Death spiraling until theres no power.
You have to disconnect the factory, and independently power your power plants back up until you have enough energy production to connect your factory up again.
Another "trick" is those burner inserters are black start capable. They can pick up fuel and feed themselves to keep running without an electrical network.
I also tend to put Schmitt triggers in low priority areas. They've got a battery on the main grid next to them and if the battery drops below 50% power they remain off until it goes back above 75% power.
Man I need to go play some more.
- Cause of event not known yet.
- They noticed power oscillations from the Spanish grid that tripped safety mechanisms in the Portuguese grid. At the time, due to the cheaper prices, the Portuguese grid was in a state of importing electricity from Spain.
- They are bringing up multiple power systems and the Portuguese grid is able to supply 100% of needs if required. It was not configured in such a state at the moment of event.
- They had to restart the black start more than once, since while starting, noticed instabilities in some sectors that forced them to restart the process.
- Time for full recovery unknown at this time, but it will take at least 24 hours.
Would this suggest the grid hasn't snapped apart, or is it just not possible to tell from the data?
Coal, pumped hydro, and nuclear generation all went to 0 around the same time, but presumably that's those sources being disconnected from the grid to balance demand? https://transparency.entsoe.eu/generation/r2/actualGeneratio...
https://x.com/RedElectricaREE/status/1916818043235164267
We are beginning to recover power in the north and south of the peninsula, which is key to gradually addressing the electricity supply. This process involves the gradual energization of the transmission grid as the generating units are connected.
I see load dropping to zero on that graph, or rather, load data disappears an hour ago.
If the grid frequency goes too far out of range then power stations trip automatically, it's not an explicit decision anyone takes and it doesn't balance load, quite the opposite. A station tripping makes the problem worse as the frequency drops even further as the load gets shared between the remaining stations, which is why grids experience cascading failure. The disconnection into islands is a defense mechanism designed to stop equipment being too badly damaged and to isolate the outage.
It's not just about the power. System components cannot be brought to operating temperatures, speeds and pressures faster than mechanical tolerances allow. If a thermal plant is cold & dark, it can take days to ramp it to full production.
1. The grid has to fully collapse with no possibility of being rescued by interconnection
2. As a result, a generation asset has to be started without external power or a grid frequency to synch to
3. An asset capable of this is usually a small one connected to a lower voltage network that has to then backfeed the higher voltage one
4. Due to the difficulty of balancing supply/demand during the process, the frequency can fluctuate violently with a high risk of tripping the system offline again
None of this applies in yesterday's case:
The rest of the European synchronous grid is working just fine.
News reports stated Spain restored power by reconnecting to France and Morocco.
By reestablishing the HV network first, they can directly restart the largest generation asset with normal procedures.
As they bring more and more load or generation online, there's little risk of big frequency fluctuations because the wider grid can absorb that.
I can only imagine the difficulty of bringing large parts of the grid back online, that rush current must be immense.
Or look at Apollo 13. The astronauts had turned off everything possible because they had lost their generator and only had their batteries. And it took a lot of furious planning by the guys on the ground to come up with a sequence of turning things back on that didn't cause the peak draw to go too high. Can't go too fast or it trips. Can't start too early because the power is limited, but can't start too late because the systems have to be up when they hit the atmosphere.
More than cash it was important yesterday to have the following in case it would have lasted longer:
- a battery powered am/fm radio with spare new batteries
- some candles and matches
- food reserves for a few days that don't need refrigeration: bread, anything in can, pasta, rice...
- some kind of gaz or alcohol stove, dry wood or bbq charcoal: you can always make a fire in the middle of the street where there is no risk of burning things around.
- water reserve (I always have like 24L of drinking water) and since I hate waste I regularly fill jerrycans when waiting for hot water in the shower that I use for manual washes (kitchenware or gears).
But with solar, how is the synchronization provided? In like a giant buck? Or in software somehow? Does the phase shift matter as much as in the electromechanical systems?
My intuition is that solar would make the grid harder to keep stable (smaller mass spinning in sync) but also may offer more knobs to control things (big DC source that you can toggle on/off instantly.. as long as sun is out). But I don’t actually know.
Currently the main driver of battery deployments is not so much energy price time arbitrage as "fast frequency fresponse": you can get paid for providing battery stabilization to the grid.
(for the UK not Spain: https://www.axle.energy/blog/frequency )
If you mean how does solar detect phase and synchronize to the grid: https://en.wikipedia.org/wiki/Phase-locked_loop
If you mean how does solar act to reinforce the grid: search for terms like "grid forming inverter vs. grid following inverter" though not all generators are the same in terms of how much resilience they add to the grid, esp. w.r.t. the inertia they do or do not add. See e.g. https://www.greentechmedia.com/squared/dispatches-from-the-g...
Low Grid frequency & voltage can cause an increase in current & heating of transmission lines and conductors and can damage the expensive things, this is why these systems trip out automatically at low frequency or low voltage, and why load shedding is necessary
The main difficulty is that the software of grid-following inverters tend to make them trip out very suddenly if the grid parameters get too far out of spec (they will only follow the grid so far), but once the grid is good they basically instantly synchronise.
But all large solar farms are likely to be mandated to switch from grid following to grid forming inverters eventually which will make them beneficial for grid security because they will help provide 'virtual intertia' that looks exactly the same to the rest of the grid as spinning mass does.
"Luckily", France is at an historically high level of production capacity at the moment and the connection between the two countries was reestablished fast.
According to RTE (French network manager), the interconnection was maxed yesterday at around 3GW of power.
Sadly, while Spain is part of CESA, it's not very well connected. I wouldn't be surprised if one the takeaway from the whole incident is that more interconnections are needed.
Ukraine went through many black starts in the first winter of Russian strikes against energy. I guess they built a skill of recovering it quickly enough that it started happening faster and easier every next time.
I took down the servers though, so you probably can't easily try it. I don't know if I added a way to configure the lobby server. I should have! It's open source though. And there is a video about that thing on my YouTube: https://www.youtube.com/watch?v=6TPgfa7LbiI
The game is bad and nothing of what we planned on doing actually made it into the game. The video is long and boring too. But maybe someone finds this cool and is inspired by this and makes a game like this.
The first 15 minutes of the game were actually about getting the ship moving, first by reading the manuals of half a dozen different ship systems and then following some procedure outlined in those manuals (parts of which were simply incorrect), maybe having to do some things in sync with your other players and stuff like that. I think it would have been cool to add multiple reactors and start them up in sync and stuff. The different ship systems were actually Lua programs that interacted via a message bus. So kind of a unknown computer architecture?
> have some cash at home
For maybe the first 24 hours at a grocery store, and then not so sure. Would your neighbors sell you supplies and food? Maybe not? And so many places now depend on cashless transactions and doubtful they have pen, paper, lockbox, and safe as a contingency plan.
The entire EU runs on one synchronised grid so from that perspective a single 'province' went offline, not the grid.
The complex process of configuring the transmission network to bring grid power to each power plant in succession is the same.
https://en.wikipedia.org/wiki/Continental_Europe_Synchronous...
https://en.wikipedia.org/wiki/Spain-Morocco_interconnection
That said, lots of people hit the cafés and had to resource to cash payments. There was also lots of people buying bottled water at the shops.
So basically, you could divide people in two groups. Those that took it like an extra Sunday, and those that took it like the beginning of a war or something :-D
Most places are so dependent upon electricity that they can't even take cash during a blackout. And they don't even have the mechanical machines to take a credit card imprint anymore.
If you try to connect another generator to the grid, it needs to be at the same point (phase) in the sine-wave cycle, so that its power contribution is added, not subtracted.
If it's not in sync, huge currents can flow, causing damage. Sort of like connecting jumper cables backwards.
Dead Comment
There is precedent for major power outages, a huge majority of which are not malicious: https://en.wikipedia.org/wiki/List_of_major_power_outages
I remember the day when the Swiss railway power network went down for a day (in 2005) because one power line was down for maintenance and someone pressed the wrong button and produced a short circuit somewhere else. It's a bit like the incidents in planes were one engine has a problem and the crew shut down the other one by mistake.
https://en.wikipedia.org/wiki/Northeast_blackout_of_2003
https://edition.cnn.com/2025/02/13/asia/sri-lanka-power-outa...
Deleted Comment
In Texas, the electric providers cut staff and maintenance to maximize shareholder value. They will not have redundant systems and redundant plants out of the goodness of their hearts. The Texas marketplace actually allowed them in the odd event of an outage to charge astronomical spot prices thinking this will incentivize them to have redundant systems. This was a foolish fantasy.
Now in Texas, discussion of how to cost share redundancy have taken place. But no one wants to pay for it. https://www.texastribune.org/2023/03/01/texas-power-market-p...
If you want a no fail grid you need to incentivise a no fail grid.
So frequent it even has its own wikipedia page: https://en.wikipedia.org/wiki/Category:Airliner_accidents_an...
Solar PV/thermal + wind: ~78%
Nuclear: 11.5%
Co-generation: 5%
Gas-fired: ~3% (less than 1GW)
They already became a laughing stock once for promising the "strongest possible response" for the Nord Stream 2 sabotage [1].
[1] https://www.reuters.com/business/energy/eu-sees-sabotage-nor...
I only have a layman's understanding of power grids, but I thought they were incredibly hardened, with backups and contingencies in depth
Are the grids at this scale really this brittle? Would there be a death toll from this?
I also wouldn't blame malice without corroborating evidence
Some are harder than others, and some have random flaws which nobody can really predict.
Spain seems in the transition to renewables, so it's possible that they have some flaws because they are still in the process, or because it's something which never happened before and is unknown territory. Also, Spain had some economic problems in the last decade, maybe someone build to cheap or was even cheating somewhere.
> Are the grids at this scale really this brittle? Would there be a death toll from this?
Hospitals should have backup-systems. Traffic should be able to stop in time. I guess the most problematic parts are people stuck in elevators and other spaces which only open electrical, as also the loss of cellular phone-connections for calling helpers.
I think we should prepare for the worst though. It's wrong to assume it's not an attack too, and until we can conclude it's not an attack we should be prepared to deal with the possible consequences and act accordingly.
Very poorly explained right now by Space Weather News. I am waiting for an updated explanation.
Deleted Comment
We're a remote business so it seemed like I'd just rudely dropped off the call, but as everything was down I couldn't let people know what'd happened.
Apparently it was caused by botched maintenance work affecting 30,000 houses, but the timing was so perfect I can't help thinking it was because our AGI overlords really didn't want me to deliver that talk for some reason.
Three quarter of the production disconnects from the grid between 12:30 and 13:00, with only a bit of solar and onshore wind sticking around.
I don't think we're able to tell from the data if one is the cause of the other, are we? Since if production was lost, load would have to be shedded to balance the grid, and if load was lost (e.g. due to a transmission failure), production would have to be disconnected to balance the grid.
That started from a combination of a lightning strike and generator trip, but turned into a local cascade failure as lots of distributed generation noticed that the frequency was under 49Hz and disconnected itself. I suspect the Spanish situation will be similar - inability to properly contain a frequency excursion, resulting in widespread generator trips.
(I suspect this is going to restart a whole bunch of acrimony about existing pain points like grid maintenance, renewables, domestic solar, and so on, probably with the usual suspects popping up to blame renewables)
[1] https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...
[1] https://transparency.entsoe.eu/transmission-domain/physicalF... [2] I'm not necessarily blaming the engineers, but the politicians who force those engineers to put square pegs in round holes. For example, I can imagine politicians making a short term decision to skimp on energy storage while increasing renewable penetration. Surely renewable systems must be less reliable without storage given the lack of rotational inertia?
For instance, one reporter asked one of the government flunkies whether it could be a cyberattack and they turned his noncommittal “maybe, we don’t know” into “government says cyberattack may be ongoing”.
Be careful of idiot reporters out there.
Edit: I’m listening to another radio interview where they are outlining the plans to bring online Portuguese dams and thermal generators over the next few hours, progressively unplugging from the Spanish supply (fortunately we have enough of those, apparently).
It should take 3-4 hours to get everything balanced with only national supplies, and they will restore power from North to South.
Key points that started it were (you can see the chain of events in the doc):
2.4.1. At 16:52:33 on Friday 9 August 2019, a lightning strike caused a fault on the Eaton Socon – Wymondley 400kV line. This is not unusual and was rectified within 80 milliseconds (ms)
2.4.2. The fault affected the local distribution networks and approximately 150MW of distributed generation disconnected from the networks or ‘tripped off’ due to a safety mechanism known as vector shift protection
2.4.3. The voltage control system at the Hornsea 1 offshore wind farm did not respond to the impact of the fault on the transmission system as expected and became unstable. Hornsea 1 rapidly reduced its power generation or ‘deloaded’ from 799MW to 62MW (a reduction of 737MW).
In my head, I'm thinking of generators/plants, connected by some number of lines, to some amount of load, where there are limited disconnection points on the lines.
So how do grid operators know what amount of load will be cut if they disconnect point A123 (and the demand behind it) vs point B456?
Is this done sort-of-blind? Or is there continual measurement? (e.g. there's XYZ MW of load behind A123 as of 2:36pm)
(apologies for singling out these specific groups of people - my point is that it might be worth to put down news sources like xitter, and read AP/translated local Portuguese news)
Dead Comment
News travelled extremely slow: phone coverage was just barely enough to receive a couple text messages every 15 minutes or so. News spread on the street, I even saw a group of 20 people hunched around someone owning a hand-held radio in the streets.
Just before power was restored, things started to get worse, as the phone coverage went completely out (presumably batteries were depleted). People were in between enjoying the work-free day, and starting to worry about how tomorrow would look like if power didn't come back.
I stopped by a friends house and we then went on a walk. Some stores were open and cash was accepted. We hung out later that night and had a few beers. The sky was amazing as there was next to no light pollution. Next day was totally in the dark as well and again, no panic. More beers were enjoyed.
The choice to move to electronic everything without having to give a shit about reliability is a failure of modern government. Move fast and break society for a dollar.
Yeah, they don't need to do that anymore. Around me, enough towers have battery backups that I can count on 2 hours of coverage when utility power goes out (if it goes out late at night or early morning, there's usually coverage until 6-7 am when people start waking up and use up the rest of the power). I don't have a real landline, but the telco DSL would drop instantly with utility power so I don't have big hopes and I wasn't willing to pay $60/month to find out.
Around when I moved, stores would pull out the credit card imprint machines, but those don't work anymore because cards are flat. Cash might work, and I've got some, but I don't think many people in my community do; people don't have cash for the snack shack we run at my kid's sports, so I doubt they have it for restaurants and stores either. And we get frequent 2-4 hour power outages, at least one, usually two or three per year; and ~ 24 hour outages every few years. The snack shack runs during summer where electricity is most reliable, but I doubt people stock up on cash in the fall and use it all up before spring/summer; they probably just don't have any.
It's interesting to think about and realize how much things have changed now though, and how reliant people are on everything, and especially their tiktoks etc. working all the time.
Some of the panic is likely related to the war in Europe too, and especially the general talk about war
Electricty went down (something kind of frequent). My UPS kept PC up, and alarm system with sim and small UPS mantained wifi up for an hour or so.
Scary moments started when people I was in a call with in Portugal texted 'Grid is out'. Later no phone signal nor data.
At first, it might seem people running towards supermarkets an overreaction on being without TikTok for a couple of hours, but you have to live how scary it is to experience this in Europe's current political status to know 60 million people (plus industry) in three countries are out of the grid.
If you see Snowden's film (this might not be the most trustworthy source) it is exactly how CIA's agent describes the feasable attack towards these countries. Again, not a valid source, but I'd love to understand if that could be feasable.
All gas stations closed because they could not sell gasoline/diesel. Today there are lines on all gas stations, people filling their car tanks and bottles..
Oh, let me tell you about electric cars! Many people had to spend the night somewhere away from home because they could not charge their cars.. My sister (with her job's electric car) had to stay the night some 200km away from home, and since the ATMs (Multibanco) didn't work, she didn't have physical money to pay for food. Luckily a stranger paid for the food (yogurt and some cookies). Petrol cars, because of their range, had better luck!
Pure fear and panic..
I can only blame the authorities (Portuguese/European) for not having contingency plans for keeping people informed, and thus letting fear spread like wildfire.
Dead Comment
I wonder what similar solutions exist in the iOS ecosystem.
I lost some years today.
My son is fine, thanks to a random person (“the man with a rabbit”) who just decided to give a lift to my son and his friend to the edge of Barcelona.
I hope this comes back ASAP.
Deleted Comment
I've got to give massive props to Antena 1 too, which is the national broadcaster's main radio station, who stopped all normal programming and did an all-day massive report on this situation to keep people informed. From what I could tell they didn't even run any ads during that period, just all-day reporting continuously repeating key information for people who'd just tuned in.
The atmosphere was quasi-festive and most people were quite relaxed, enjoying an unexpected afternoon off. Younger people filled the bars which were serving everything they could. There were long lines at supermarkets and an occasional fellow toting a box of supplies, but mostly there were just huge numbers of people in the street and completely collapsed traffic flow (the police were out in force almost immediately, directing traffic). In the part of Madrid I was in about 1/4-1/3 of the population is from South America and I suspect most of them have seen this all before anyway. The only real stress I saw was from people that need a train to get home (because the trains weren't running) and a had a walk of more than 2-3 hours.
I got cell phone signal when I was near two hospitals which were fully operational.
It was interesting that almost immediately, while I was still at work, everyone said power was out in Portugal and France too. After an hour or two some were claiming problems in Germany, but this seemed already to be unfounded rumors.
Some younger people couldn't walk home because they didn't have google maps ...
REN said: “Due to extreme temperature variations in the interior of Spain, there were anomalous oscillations in the very high voltage lines (400 kV), a phenomenon known as ‘induced atmospheric vibration’. These oscillations caused synchronisation failures between the electrical systems, leading to successive disturbances across the interconnected European network.”
https://www.theguardian.com/business/2025/apr/28/spain-and-p...
https://en.m.wikipedia.org/wiki/Conductor_gallop
Dead Comment
https://www.euronews.com/my-europe/2025/04/28/spain-portugal...
The similarity between that event and this early-on report is striking.
[es language]: https://www.lavozdegalicia.es/noticia/espana/2021/07/24/aver...
I suppose it makes sense that it was an automatic shutdown rather than infrastructure failing on such a wide area. And then once it's shut down, a black-start is a logistical challenge as other comments have explained.
I'm also seeing some reports about it being more likely that something happened on the east side, somewhere like the Ebro valley or north across the Pyrenees. Catalonia seems to have been particularly affected, and it's on the path of important lines coming from France. High heat at noon could have caused a line to fail and short against a tree, which would be similar to the 2003 nation-wide outage in Italy.
"Le gestionnaire français souligne par ailleurs que cette panne n’est pas due à un incendie dans le sud de la France, entre Narbonne et Perpignan, contrairement à des informations qui circulent."
Called N-1 criterion. See https://en.wikipedia.org/wiki/Contingency_(electrical_grid)#...
And it depends. During https://en.wikipedia.org/wiki/2006_European_blackout N-1 criterion was supposed to be holding, in practice not even N-0 was holding and network crashed.
Few years ago nearly entire day European network was sitting on N-0 due to multiple issues in Poland, caused by a heat wave and deeper root causes. There are many power plants and power lines where any further issue would cause Europe-wide blackout.
https://radar.cloudflare.com/es?dateRange=1d
https://radar.cloudflare.com/pt?dateRange=1d
Portugal nearly reached zero.