This article starts by saying "we dont know what actually happened here, or why" and then goes on to make various insinuations and suppositions, and then proposes policy based on that.
We haven't yet seen the sequence of breaker trip events.
We haven't yet seen the graphs of power flows at tie points, and of frequency and voltage error.
The closest thing to a useful fact visible from Google seems to be: "The network lost 15 gigawatts of electricity generation in five seconds at around 1033 GMT, the Energy Ministry said on Monday evening, without explaining the reason for the loss." That's not a cause. Losing 15GW all at once, more power than any one plant generates, indicates some previous event had caused breaker trips somewhere. No indication of the previous event.
We do know this wasn't a supply shortage. There was plenty of generating capacity online.
Here's an analysis of the US Northeast blackout of 2003.[1] Until we start to see that level of detail, it's just blithering.
Unless someone diverted all that power for some purpose.
Classic scifi might be a time traveller trying to get home, or someone opening an interdimensional gateway, or maybe an AI becoming conscious and needing the power to ascend.
This unusual situation points to a perfect storm of poor grid management and inadequate connections of solar facilities to the grid, as well as other unknown faults. In my opinion, there is a good chance ..."
We don't really know, I don't really know, but I'm going to write a long post about it.
It makes me realize how ignorant about power grid I am. It is kinda unfathomable to me, that 2 countries completely black out for 10 hours, then somewhat "magically" go up again, and a week later nobody knows what went wrong and how it got fixed. Or maybe someone knows but doesn't want to say? Anyway, it makes an impression that the whole power grid is this magical black box that somehow works on its own, breaks, fixes itself, and humans can only pray gods to let them have good harvest and uninterrupted electricity next summer. Maybe it's time to renew sacrifices at the temple of Jupiter or somebody else.
I don't agree. You do not need a thorough root cause analysis to acknowledge the fact that a) some invariants in the system were violated, b) some of the people accountable are already making claims that fly in the face of reason.
Also, Spain has a regrettable track record of covering up the responsibilities of state institutions in major disasters.
> At that time the price of electricity on the official market was in the negative at around -1€/MWh.
> Current evidence therefore points to a problem in the synchronisation of the grid. All sources feeding power into the grid must be synchronised at the same frequency, 50 Hertz. To facilitate this synchronisation, stable base-load power is required, which is normally provided by nuclear and other large gas and hydroelectric facilities. These sources act as a natural buffer against disturbances
So:
- price is negative, so solar automatically disconnects not to pay for providing electricity
- nuclear is overloaded at unexpected time, it also disconnect due to safety.
Seems like a bug in accounting software for solar power plants. It disconnected too many power plants too quickly! I bet like 40% of solar plants are using the same software for managing connectivity.
I've seen a few articles of The Conversation/Europe on HN, and it seems an interesting news source. Wikipedia also gives some insight. I've just subscribed.
Does HN know about their agenda's, biases, blind spots. Not necessarily to shoot them down, just to be aware of them.
> However, variable renewable sources, such as solar photovoltaic, do not have this capability. They generate direct current which is converted to alternating current at 50 Hertz, but they cannot react automatically to frequency variations.
This part didn't make sense to me.
The rotating machinery of hydroelectric, or nuclear's steam generators, produce the frequency of AC in synchronization with their mechanical rotation, but the inverters converting solar's DC into AC are making a purely synthesized waveform under control of high speed digital electronics.
I would think they should be the most able to modify the AC sine wave they are generating.
I remember reading an article about exactly that some time last year: https://spectrum.ieee.org/electric-inverter The technology is grid forming inverters, unlike the normal grid following converters. I'm definitely not an expert, but I think the part you're quoting is hinting at the following:
> Grid-following inverters operate only if they can “see” an existing voltage and frequency on the grid that they can synchronize to. They rely on controls that sense the frequency of the voltage waveform and lock onto that signal, usually by means of a technology called a phase-locked loop. So if the grid goes down, these inverters will stop injecting power because there is no voltage to follow. A key point here is that grid-following inverters do not deliver any inertia.
From the linked ieee articale ..... Not yet commercially available ....
For photovoltaics and wind, grid-forming inverters are not yet commercially available at the size and scale needed for large grids, but they are now being developed by GE Vernova, Enphase, and Solectria......
You are very right about electronics being much more capable than electromechanical systems. We can sychronize the frequency much better with GPS for example. This wasn't an issue for small scale solar, so apparently nobody implemented it to keep things low cost. We are at a point that we need these.
The beauty of it is the electronics can be connected and controlled much better with each other too, so more optimized control algorithms for supply and demand balancing is possible. Bad thing is, this will incentivize to solve problems by over the air updates. As we know software reliability has gotten so much worse, which makes it more prone to bugs being pushed. Obe should expect weird outages due to someone pushing something buggy.
You’re not wrong, it’s not that they can’t, it’s just that they currently don’t do it well. The concept is known as as grid forming if you want to read about it.
I'm having a hard time understanding how we can move the bulk of our energy to renewables without some kind of massive storage system. The wind can die for long periods. It can get abnormally cloudy for long periods. There's nighttime and winter.
I'm no expert, but is there any storage system that is practical that can store the amount of energy we'd need to have most (all) electricity come from renewables?
That’s why Grid Storage is a major topic in power generation and renewables. The technology and economics are evolving. A lot of people are working on finding a storage solution that is both durable and economical. One solution is probably not going to solve for all situations.
In the meantime, load sharing is used to distribute power across a power grid for cases where one area has more power and another area needs power. For instance, wind may die out in one area but over larger areas there are usually still areas with wind. It’s harder with solar, of course.
RE "....load sharing is used to distribute power across a power grid ....." TO distribute power across the grid - on a large scale fashion , would require a new power ( very expensive) power distribution network. The base problem with renewable solar and wind is its not reliable. an have outages for SEVERAL days
We can do the bulk of it, i.e. more than 50%, by geographic and modal aggregation fairly easily. Doing all of it from renewables is still an unsolved problem that would require better energy storage.
We have very detailed continental weather readings for over a century and can build very good statistical models. So you just ask a supercomputer to find the lowest cost solution for 99.99% electrical coverage without using any carbon sources.
It's quite possible -- over the previous 30 years we've never had an hour where there was no wind & sun anywhere in Europe. So we could theoretically solve it without storage, but that would require an insane amount of overbuild and interconnect. Imagine supplying all of Europe's electricity needs with just wind power from the south shore of Ireland.
Adding a few days worth of storage reduces the interconnect & overbuild needs dramatically, putting 99.99% carbon free within the realm of the possible.
> So you just ask a supercomputer to find the lowest cost solution for 99.99% electrical coverage without using any carbon sources.
Sure... but my expectation is that that 90% solution would include an unreasonable amount of storage (assuming you want most/all renewables). I guess was my point.
> It's quite possible -- over the previous 30 years we've never had an hour where there was no wind & sun anywhere in Europe.
But "some" wind or sun is not enough to support the whole grid.
Having enormous synchronized grids helps a bit with that.
It’s highly unlikely to be abnormally cloudy/calm over the whole of Europe, for example, so Europe’s large grid (https://en.wikipedia.org/wiki/Continental_Europe_Synchronous...) can be used to move electricity from where there’s excess energy from solar or wind to where there’s a deficit.
That requires large capacity connections, though that aren’t everywhere yet.I understand electricity was restored in the south of France much more rapidly last week than in Spain and Portugal because it’s much better connected to the rest of Europe.
The reason why the Iberian grid is not better connected to France might have something to do with French Nuclear not wanting to be bankrupted by cheap renewables from southern countries, this was the word on the street a ehi'e back, when the push for renewables in Spain in Portugal was starting up.
This is a political problem, having Mitteleuropa depend on power from Putin is apparently OK, but on lazy southerners is verboten. Hopefully this blackout changes that (but I'm not holding my breath).
Some hydroelectric plants can pump water back up and both absorb excess supply and provide stabilization during demand spikes.
I have no idea if it’s feasible to fill longer-term gaps during extended cloudy/windless days and nights using that alone, though. Other than that, there are already large-scale battery plants deployed in some cities.
Another approach is to control the demand side: When air conditioning or heating with electricity, minutes usually don’t matter, and dropping/providing extra load at very short notice should be feasible in a smart grid.
It is entirely infeasible to used pumped hydro alone to solve our energy storage problems. They’re a great component of the solution but they’re big expensive environmentally disruptive projects and there are limited suitable sites.
> To facilitate this synchronisation, stable base-load power is required, which is normally provided by nuclear and other large gas and hydroelectric facilities. These sources act as a natural buffer against disturbances, helping to keep the frequency stable in the face of sudden changes in generation or demand.
In theory, it seems like you could instrument a photovoltaic array to carry some "inertia" with the right control system.
If you need to feed power, you run some power point tracking algorithm, and if you need to consume power, you just overbias the cells and heat them up.
In a system of micro-inverters, they need something to synchronize with. There needs to be a "truth" reference, so they can push their power onto the grid by slightly leading the phase of that.
If some critical mass of PV micro-inverters exceeds the traditional generators, they'll push so hard that the grid itself will change phase, and blackouts are the result.
One possible solution might be to use a better oscillator in the micro-inverter and limit the rate of phase shift. Unfortunately, the grids of the world have been moving in the opposite direction and now allow more drift than in the past, so where do you draw the line?
> they'll push so hard that the grid itself will change phase
That's a thundering herd problem: where all invertors have the same set-points and they all are synchonised to push in the same direction at once.
In networks, thundering herd problems are fixed by given each sender different random delays.
For power networks we could choose statistical methods to get individual solar generators to lead or lag so that the frequency becomes an aggregate vote.
Given that part of the blackout was due to large amounts of solar going offline at the same time, it's possible that all invertors with common software were tripped at the same set-point.
So trip conditions also need to be randomly fuzzy. E.g. if frequency drops below 49Hz +/- random spread of 0.5Hz.
Although it's difficult to match financial incentives against random variations (individual generators are incentivised to power outside of boundaries to keep getting paid, and a trip event can be expensive - due to restart costs).
Electricity market design is hard because the design needs to be resilient to perverse incentives.
Yep it’s called synthetic inertia. Solar and wind can both offer it. Wind can actually offer loads of it it’s just not as easy since it’s not synchronous.
you have panels connected to grid thru inverter and that can modulate output in any way needed in few milliseconds.
same as inverter in electric car providing power to motors.
or inverter providing power to coils in your loudspeaker/ headphones.
inverter can adjust phase, voltage, frequency. it can means it is job of inverter to provide that in normal operation. that is why it is there in first place.
> much of the available energy was being used to pump water from low lying river basins into reservoirs – the only practical way to store energy on a large scale. However, this capacity has a limit and, with the reservoirs almost full, it cannot continue to be stored indefinitely.
There has been unusually heavy rain in Spain for an unusual length of time now, and reservoirs are well beyond average capacity.[1]
I wonder if this anomalous situation has anything to do with the blackout. Maybe the inability to continue pumping water led to the shutdown of the solar, maybe for economic reasons as is suggested in some other comment?
Readit News
This not useful. More data is required.
We haven't yet seen the sequence of breaker trip events. We haven't yet seen the graphs of power flows at tie points, and of frequency and voltage error.
The closest thing to a useful fact visible from Google seems to be: "The network lost 15 gigawatts of electricity generation in five seconds at around 1033 GMT, the Energy Ministry said on Monday evening, without explaining the reason for the loss." That's not a cause. Losing 15GW all at once, more power than any one plant generates, indicates some previous event had caused breaker trips somewhere. No indication of the previous event.
We do know this wasn't a supply shortage. There was plenty of generating capacity online.
Here's an analysis of the US Northeast blackout of 2003.[1] Until we start to see that level of detail, it's just blithering.
[1] https://en.wikipedia.org/wiki/Northeast_blackout_of_2003
Classic scifi might be a time traveller trying to get home, or someone opening an interdimensional gateway, or maybe an AI becoming conscious and needing the power to ascend.
This unusual situation points to a perfect storm of poor grid management and inadequate connections of solar facilities to the grid, as well as other unknown faults. In my opinion, there is a good chance ..."
We don't really know, I don't really know, but I'm going to write a long post about it.
I don't agree. You do not need a thorough root cause analysis to acknowledge the fact that a) some invariants in the system were violated, b) some of the people accountable are already making claims that fly in the face of reason.
Also, Spain has a regrettable track record of covering up the responsibilities of state institutions in major disasters.
> Current evidence therefore points to a problem in the synchronisation of the grid. All sources feeding power into the grid must be synchronised at the same frequency, 50 Hertz. To facilitate this synchronisation, stable base-load power is required, which is normally provided by nuclear and other large gas and hydroelectric facilities. These sources act as a natural buffer against disturbances
So:
- price is negative, so solar automatically disconnects not to pay for providing electricity
- nuclear is overloaded at unexpected time, it also disconnect due to safety.
Seems like a bug in accounting software for solar power plants. It disconnected too many power plants too quickly! I bet like 40% of solar plants are using the same software for managing connectivity.
Does HN know about their agenda's, biases, blind spots. Not necessarily to shoot them down, just to be aware of them.
https://en.wikipedia.org/wiki/The_Conversation_(website)
https://mediabiasfactcheck.com/the-conversation/
Dead Comment
This part didn't make sense to me.
The rotating machinery of hydroelectric, or nuclear's steam generators, produce the frequency of AC in synchronization with their mechanical rotation, but the inverters converting solar's DC into AC are making a purely synthesized waveform under control of high speed digital electronics.
I would think they should be the most able to modify the AC sine wave they are generating.
Maybe someone can comment and correct my logic...
> Grid-following inverters operate only if they can “see” an existing voltage and frequency on the grid that they can synchronize to. They rely on controls that sense the frequency of the voltage waveform and lock onto that signal, usually by means of a technology called a phase-locked loop. So if the grid goes down, these inverters will stop injecting power because there is no voltage to follow. A key point here is that grid-following inverters do not deliver any inertia.
For photovoltaics and wind, grid-forming inverters are not yet commercially available at the size and scale needed for large grids, but they are now being developed by GE Vernova, Enphase, and Solectria......
The beauty of it is the electronics can be connected and controlled much better with each other too, so more optimized control algorithms for supply and demand balancing is possible. Bad thing is, this will incentivize to solve problems by over the air updates. As we know software reliability has gotten so much worse, which makes it more prone to bugs being pushed. Obe should expect weird outages due to someone pushing something buggy.
I'll do some reading on "grid forming".
I'm no expert, but is there any storage system that is practical that can store the amount of energy we'd need to have most (all) electricity come from renewables?
In the meantime, load sharing is used to distribute power across a power grid for cases where one area has more power and another area needs power. For instance, wind may die out in one area but over larger areas there are usually still areas with wind. It’s harder with solar, of course.
It's quite possible -- over the previous 30 years we've never had an hour where there was no wind & sun anywhere in Europe. So we could theoretically solve it without storage, but that would require an insane amount of overbuild and interconnect. Imagine supplying all of Europe's electricity needs with just wind power from the south shore of Ireland.
Adding a few days worth of storage reduces the interconnect & overbuild needs dramatically, putting 99.99% carbon free within the realm of the possible.
Sure... but my expectation is that that 90% solution would include an unreasonable amount of storage (assuming you want most/all renewables). I guess was my point.
> It's quite possible -- over the previous 30 years we've never had an hour where there was no wind & sun anywhere in Europe.
But "some" wind or sun is not enough to support the whole grid.
It’s highly unlikely to be abnormally cloudy/calm over the whole of Europe, for example, so Europe’s large grid (https://en.wikipedia.org/wiki/Continental_Europe_Synchronous...) can be used to move electricity from where there’s excess energy from solar or wind to where there’s a deficit.
That requires large capacity connections, though that aren’t everywhere yet.I understand electricity was restored in the south of France much more rapidly last week than in Spain and Portugal because it’s much better connected to the rest of Europe.
This is a political problem, having Mitteleuropa depend on power from Putin is apparently OK, but on lazy southerners is verboten. Hopefully this blackout changes that (but I'm not holding my breath).
I have no idea if it’s feasible to fill longer-term gaps during extended cloudy/windless days and nights using that alone, though. Other than that, there are already large-scale battery plants deployed in some cities.
Another approach is to control the demand side: When air conditioning or heating with electricity, minutes usually don’t matter, and dropping/providing extra load at very short notice should be feasible in a smart grid.
https://en.wikipedia.org/wiki/Iron_redox_flow_battery
Its commercially available right now as well.
Its not as efficient as lithium or as energy dense, but that doesnt matter if you're putting them in wind/solar farms.
In theory, it seems like you could instrument a photovoltaic array to carry some "inertia" with the right control system.
If you need to feed power, you run some power point tracking algorithm, and if you need to consume power, you just overbias the cells and heat them up.
If some critical mass of PV micro-inverters exceeds the traditional generators, they'll push so hard that the grid itself will change phase, and blackouts are the result.
One possible solution might be to use a better oscillator in the micro-inverter and limit the rate of phase shift. Unfortunately, the grids of the world have been moving in the opposite direction and now allow more drift than in the past, so where do you draw the line?
https://medium.com/@brandonvar/power-grid-stability-issues-a...
That's a thundering herd problem: where all invertors have the same set-points and they all are synchonised to push in the same direction at once.
In networks, thundering herd problems are fixed by given each sender different random delays.
For power networks we could choose statistical methods to get individual solar generators to lead or lag so that the frequency becomes an aggregate vote.
Given that part of the blackout was due to large amounts of solar going offline at the same time, it's possible that all invertors with common software were tripped at the same set-point.
So trip conditions also need to be randomly fuzzy. E.g. if frequency drops below 49Hz +/- random spread of 0.5Hz.
Although it's difficult to match financial incentives against random variations (individual generators are incentivised to power outside of boundaries to keep getting paid, and a trip event can be expensive - due to restart costs).
Electricity market design is hard because the design needs to be resilient to perverse incentives.
same as inverter in electric car providing power to motors.
or inverter providing power to coils in your loudspeaker/ headphones.
inverter can adjust phase, voltage, frequency. it can means it is job of inverter to provide that in normal operation. that is why it is there in first place.
There has been unusually heavy rain in Spain for an unusual length of time now, and reservoirs are well beyond average capacity.[1]
I wonder if this anomalous situation has anything to do with the blackout. Maybe the inability to continue pumping water led to the shutdown of the solar, maybe for economic reasons as is suggested in some other comment?
[1] https://www.embalses.net/
Yippie Yah Hoo, Yippie Yah Yay!
Try https://www.youtube.com/watch?v=0TPOj4CTGKs if you don't get it ;-)