The root problem, as always, is that it has been normalized that devices in your house, accessed from your house, need cloud access to do this, or even to function all.
Metrics from an inverter, once upon a time, would have been a local web server in the device. Maybe with QR code printed on the device so the typical smartphone user could access it. Firmware updates ought to be physically "opt in" - like stick a USB stick or MicroSD card into the device and push a button.
Not some mysterious cloud that through legal issues, malice or sheer incompetence, can reach in and modify or delete functionality without warning.
My dishwasher has a little nag light to remind me I haven't connected it to my Wifi yet. I never will. It washes dishes just fine.
> Metrics from an inverter, once upon a time, would have been a local web server in the device.
Or just a regular serial port! For example, IEC 62056 [0] provides a fairly trivial standardized way to interact with an electricity meter using an IR reader head. Even easier, the DSMR standard outputs serial data via a 5V RJ12 connector [1]. You can connect that to a PC with a $5 USB-to-serial adapter, directly to a Raspberry Pi, or to one of a dozen $20 cloud dongle thingies.
Just mandate a serial interface, and the inverter itself doesn't need any kind of web interface whatsoever.
These kind of things will stop when they start getting treated as malicious attacks (similar to ransomware), i.e. the perpetrators become wanted people and if caught, see significant jail time.
This goes both for the malicious bricking of normal consumer devices, and attacks on critical infrastructure like this, except of course the punishment for the latter should be correspondingly more severe.
Assuming that one of these inverters is in North Carolina, they’re facing a $250,000 fine (hopefully per inverter), and second degree murder/40 years if anyone died:
Only if this actually gets treated as an attack though, which I haven't seen happen in similar cases in the past.
Sony BMG with the hidden DRM rootkit malware on their music CDs got some civil penalties but no criminal prosecution. Sony with the Playstation OtherOS removal had to pay a ridiculously low class action, no criminal prosecution. Lenovo got a slap on the wrist for putting an adware firmware bootkit into the machines, again civil only.
A lot of companies are still getting away with exfiltrating memory dumps by default as part of their error reporting, selling your location data, etc.
The only criminal prosecution (as in "butt in jail") for similar behavior that I'm aware of is Volkswagen's Dieselgate, and that was only prosecuted because it was seen as screwing over the US government, not consumers.
The biggest takeaway here should be that we need a domestic solar industry.
We can't hold Deye or Chinese companies culpable.
Moreover, this should serve as a warning shot for what could become a national security issue if we keep juggling international suppliers for critical infrastructure. They'll all have the capability of shutting down US electricity, which is unacceptable.
There's no reason we should be importing this stuff.
> The biggest takeaway here should be that we need a domestic solar industry. We can't hold Deye or Chinese companies culpable.
No, the takeaway is to not allow corps to have remote access to end-user owned devices in the first place.
This story of perfectly capable devices being bricked or having servers shut off has been told so many times with domestic (or friendly countries) companies it's laughable that the conclusion is 'do the same thing but onshore'.
I feel like stuff like this shouldn't be anywhere near the internet. Partly because of reasons like this where the manufacturer can just randomly decide to disable it, but also because its usually the software equivalent of Swiss cheese.
I’m not saying those things are safe but isn’t the attack surface pretty limited if you are behind NAT/a pretty basic firewall? The only connection to the internet should be the device reaching out to a server and asking for an update from time to time, it shouldn’t really be reachable from the outside. Unless the update server is compromised too, I don’t really see what can happen.
Only if that attack surface doesn't include employees, household members, contractors, shared spaces, etc. That is, a small business may be fairly safe if they're no cohabiting. A corporation probably isn't.
In this case the manufacturer was the one that triggered it. Even if it weren’t, how secure their servers are, or which foreign legislation they are subject to is a total unknown.
I have a solar edge inverter. I never connected it to the Internet out of concern that this was possible. While it is a different company, this vindicates my concern.
I have inverter of different brand and also had concern to allow it internet connection, so i ended with pi zero connected to it’s internal wifi with socat port gateway, a route on router to simulate it’s internal network and it’s app works thinking it is connected locally to device, even over vpn back to home.
I could use the device’s buttons and LCD to get some stuff, but I generally don’t bother. Maybe if I plugged it into the network and disallowed internet communication, I could poke around to see if there is a way, but I have not felt motivated to try.
People were buying Chinese inverters meant for the Chinese market off aliexpress on the gray market and shipping them to other countries. Deye decided to crack down on the behavior.
There’s nothing indicating this has anything to do with sol-Ark at this point other than them being the approved distributor of rebranded deye inverters in the US.
Sol-Ark’s markup is like 5x the list price just for the official rebadged version. Sol-Arks (“US veteran owned company”) still have the firmware made in China, and are susceptible to Chinese hackers, and had to be bought through a distributor. So naturally people went with off-listed Deye inverters because of the scheningans from Sol-Ark.
Now, people are without power and they have to go to Sol-Ark to get power restored, likely by paying through the nose.
That's one way to frame it. Another is Sol-ark incurs costs of developing, marketing and supporting their official devices and the contract manufacturer is able to sell their own version in the Chinese market. Greedy people who don't want to pay Sol-ark for all the costs they incurred bought grey market devices that Sol-ark has repeatedly warned are in contract violation in this market. The manufacturer, not Sol-ark, has now bricked those devices, and people are blaming Sol-ark anyway because they want to continue to justify their actions.
USA is a free market. Everyone is authorized all the time to sell every safe product. The terms "gray market" and "authorized reseller" are linguistic manipulations which benefit manufacturers at the expense of everyone else in society.
I think Daye broke US law when they destroyed law-fully purchased products inside USA. I hope the inverter owners bring a class-action lawsuit against Daye in the US. The court could block the sale of the company's products in USA until they restore the inverters and pay restitution.
That’s laughably wrong. Exclusive distribution rights are probably enforced more strictly in the U.S. than anywhere else in the world. They are governed by contract law. In addition, many product categories need to be demonstrated as safe to the right licensing agencies before being sold, not after.
Different countries have different laws and requirements around grid-connected inverters, mostly so people working on the grid don't get electrocuted when a stray inverter keeps feeding in power.
Usually you want some way of monitoring how much energy your panels are producing. This helps to realise you need to clean the panels or do some maintenance if panels start failing. Or it may be useful for scheduling home appliance usage.
But in practice this almost always means connecting to the internet, because the simplest interface is wifi and data collection/display at the producer's servers. So any extra features == internet connection.
Highly recommend using solarassistant for this, instead - local server software that install on a raspi, and you hook a usb on the raspi to the WiFi dongle port on your inverter with a serial cable. Don’t provide the inverter itself with any wifi credentials.
Solar assistant has the bonus of interfacing your inverter with homeassistant, and letting it control the inverter/get signals from it (so you can do things like, if grid voltage drops to zero, do xyz)
It's just a bad idea. I got caught up in a situation where one company sold me a solar installation, then a subcontractor installed and configured it. Apparently they got into a spat about money, because the subcontractor told me to pay the bill straight to them.
Otherwise they'd shut down the newly installed solar installation. I said, can you do that? Of course while talking, I changed the WiFi password.
Solar installations are expensive enough that some manufacturers can probably afford to integrate a cellular modem into the product (similar to how all new cars do it today). Good luck changing the Wi-Fi password on that!
I have a Axpert MAX E. It has a WiFi AP constantly advertised. The only way to configure/disable that is via a .cn app! The app also allows remote control and monitoring of the inverter, via some unknown cloud server.
I run everything local-only, so that is never going to happen.
> The only way to configure/disable that is via a .cn app!
What does it even mean for an app to be ".cn"? Apps typically aren't identified by DNS names. Did you have to download it from a .cn domain? Is it just a roundabout way of saying the app was Chinese?
It's not the solar inverters themselves that are usually internet connected, but rather the controller box (some kind of embedded system) that is internet connected to allow monitoring and control. Perhaps this manufacturer decided to economize and make both of them part of the same "box", with the result that an error condition in the controller would result in the non-operation of the inverter part.
Some systems like mine (Enphase) do a good job of letting the inverters operate independently of the monitoring/control software. But to do this, I believe they need to add data storage to the inverters themselves in order to log data during a controller "outage".
Mainly data collection (previous lead dev at solar forecasting startup). All the web UIs to view usage are also collecting useful information that can be used in forecasting models. One of the researches I worked with wrote some papers on using distributed home solar output measurements to assist with generating higher resolution irradiance forecasts and estimated actuals/observations. You have to do a lot of data cleaning to get this reliable though. Anyway, this data from memory was bought/sold for various research/commercial weather modeling.
Besides the reasons others have already mentioned, load management comes to mind:
Getting rid of excess energy in the grid can be just as hard a problem to solve as to deal with excess load, and being able to simply and very quickly remove some supply from the grid is very useful for that.
nice dashboards for information about generation. but most importantly remote troubleshooting/diagnostics. as example i have system made from multiple inverters, batteries, car charger and backup interface. after installation some stuff slightly misbehaved. manufacturer support were able to look at system logs and configuration and identify that system is slightly incorrectly wired/configured, after what installer was able to fix it.
same thing goes for malfunctioning parts of system. support can take a glance at it and issue rma on spot
Hi, idiot here. I badly wanted a US-made robot vacuum that uses LiDAR for mapping and a camera for object classification. This does not exist. Your only options are Chinese-owned-and-operated.
I could flash them with Valetudo and wire them up to Home Assistant, but doing so requires me to solder shit to the JTAG circuit and buy some niche hardware, which requires me to open up the vac and potentially brick it. I'm not risking that on a $1200 device.
That part is independent of internet connection. Especially since you can't rely on the internet connection in case of power delivery issues. It's a completely different network.
> U.S. Supreme Court Holds that Books Printed and Sold Abroad May Be Freely Resold in the U.S. Because the Copyrights Are Exhausted Under the First-Sale Doctrine.. The Kirtsaeng decision is significant to copyright owners, and it may also have important ramifications for patent owners who make and sell goods abroad that practice a U.S. patent.
> The first sale doctrine is a legal principle that limits the copyright owner's control over a particular copy of their work after it's been lawfully sold. This doctrine, in essence, acts to cut off the copyright owner's rights in the created work after the product is first sold (ie. when the copyright owner releases their work into the marketplace). Another way to describe it is that the copyright holder's right to control the distribution of their work goes away after the “first sale” of the work,(hence the name). In more straightforward and more practical terms, once you buy a book, CD, DVD, artwork or any other authorized copy of a copyrighted work, the copyright owner generally loses the right to control what you do with that specific copy. You can resell it, lend it, give it away, or even destroy it, without their permission.
Remote bricking requires software, which is sold under copyright law.
> nothing to do with exclusivity agreements arranged between companies
The 2013 U.S. Supreme Court case depended on a plaintiff that was making enough money on textbook arbitrage to fund a legal case all the way to the Supreme Court. It provided new clarity on book distribution and geographical "exclusivity".
If software enforcement of device distribution agreements affects a large enough flow of capital, then corner cases will accrue enough economic impact to be tested in courts. Manufacturers do not have carte blanche to manipulate hardware remotely, e.g. they cannot take actions that could injure humans. Where are the limits? For now, we have many opinions and few laws.
Readit News
Metrics from an inverter, once upon a time, would have been a local web server in the device. Maybe with QR code printed on the device so the typical smartphone user could access it. Firmware updates ought to be physically "opt in" - like stick a USB stick or MicroSD card into the device and push a button.
Not some mysterious cloud that through legal issues, malice or sheer incompetence, can reach in and modify or delete functionality without warning.
My dishwasher has a little nag light to remind me I haven't connected it to my Wifi yet. I never will. It washes dishes just fine.
Or just a regular serial port! For example, IEC 62056 [0] provides a fairly trivial standardized way to interact with an electricity meter using an IR reader head. Even easier, the DSMR standard outputs serial data via a 5V RJ12 connector [1]. You can connect that to a PC with a $5 USB-to-serial adapter, directly to a Raspberry Pi, or to one of a dozen $20 cloud dongle thingies.
Just mandate a serial interface, and the inverter itself doesn't need any kind of web interface whatsoever.
[0]: https://en.wikipedia.org/wiki/IEC_62056
[1]: https://jensd.be/1183/linux/read-data-from-the-belgian-digit...
Opt in for major functionality, that is fine.
None of the IoT devices we own have had an update that fixes a user facing bug, but most have had critical updates that break existing functionality.
This goes both for the malicious bricking of normal consumer devices, and attacks on critical infrastructure like this, except of course the punishment for the latter should be correspondingly more severe.
https://pemc.coop/bill-protecting-critical-infrastructure-si...
Sony BMG with the hidden DRM rootkit malware on their music CDs got some civil penalties but no criminal prosecution. Sony with the Playstation OtherOS removal had to pay a ridiculously low class action, no criminal prosecution. Lenovo got a slap on the wrist for putting an adware firmware bootkit into the machines, again civil only.
A lot of companies are still getting away with exfiltrating memory dumps by default as part of their error reporting, selling your location data, etc.
The only criminal prosecution (as in "butt in jail") for similar behavior that I'm aware of is Volkswagen's Dieselgate, and that was only prosecuted because it was seen as screwing over the US government, not consumers.
We can't hold Deye or Chinese companies culpable.
Moreover, this should serve as a warning shot for what could become a national security issue if we keep juggling international suppliers for critical infrastructure. They'll all have the capability of shutting down US electricity, which is unacceptable.
There's no reason we should be importing this stuff.
No, the takeaway is to not allow corps to have remote access to end-user owned devices in the first place.
This story of perfectly capable devices being bricked or having servers shut off has been told so many times with domestic (or friendly countries) companies it's laughable that the conclusion is 'do the same thing but onshore'.
Pretty competitive too.
(looks at election results)
Ok, tariffs. I guess tariffs are the new invisible hand.
And really, what we're talking here isn't domestic manufacturing. It's probably Mexican manufacturing.
Deye said something to the effect of "we have contractual obligations".
I think they're both at fault.
People were buying Chinese inverters meant for the Chinese market off aliexpress on the gray market and shipping them to other countries. Deye decided to crack down on the behavior.
There’s nothing indicating this has anything to do with sol-Ark at this point other than them being the approved distributor of rebranded deye inverters in the US.
Now, people are without power and they have to go to Sol-Ark to get power restored, likely by paying through the nose.
I think Daye broke US law when they destroyed law-fully purchased products inside USA. I hope the inverter owners bring a class-action lawsuit against Daye in the US. The court could block the sale of the company's products in USA until they restore the inverters and pay restitution.
Contempt of business model is legal, and vigilantism is not.
1. They're not properly licensed for other markets. Something equivalent to selling a radio transmitter in the US that's not registered with the FCC.
2. They price units outside of Asian markets much higher and don't want to allow/encourage arbitrage that they don't control.
This is definitely a case of "porqué no los dos" (or more).
But in practice this almost always means connecting to the internet, because the simplest interface is wifi and data collection/display at the producer's servers. So any extra features == internet connection.
Solar assistant has the bonus of interfacing your inverter with homeassistant, and letting it control the inverter/get signals from it (so you can do things like, if grid voltage drops to zero, do xyz)
Otherwise they'd shut down the newly installed solar installation. I said, can you do that? Of course while talking, I changed the WiFi password.
What does it even mean for an app to be ".cn"? Apps typically aren't identified by DNS names. Did you have to download it from a .cn domain? Is it just a roundabout way of saying the app was Chinese?
Some systems like mine (Enphase) do a good job of letting the inverters operate independently of the monitoring/control software. But to do this, I believe they need to add data storage to the inverters themselves in order to log data during a controller "outage".
Getting rid of excess energy in the grid can be just as hard a problem to solve as to deal with excess load, and being able to simply and very quickly remove some supply from the grid is very useful for that.
Deleted Comment
I could flash them with Valetudo and wire them up to Home Assistant, but doing so requires me to solder shit to the JTAG circuit and buy some niche hardware, which requires me to open up the vac and potentially brick it. I'm not risking that on a $1200 device.
Dead Comment
The short answer is: it's for load balancing, it can't be avoided.
> U.S. Supreme Court Holds that Books Printed and Sold Abroad May Be Freely Resold in the U.S. Because the Copyrights Are Exhausted Under the First-Sale Doctrine.. The Kirtsaeng decision is significant to copyright owners, and it may also have important ramifications for patent owners who make and sell goods abroad that practice a U.S. patent.
https://www.iveticlaw.com/owning-vs-controlling-understandin...
> The first sale doctrine is a legal principle that limits the copyright owner's control over a particular copy of their work after it's been lawfully sold. This doctrine, in essence, acts to cut off the copyright owner's rights in the created work after the product is first sold (ie. when the copyright owner releases their work into the marketplace). Another way to describe it is that the copyright holder's right to control the distribution of their work goes away after the “first sale” of the work,(hence the name). In more straightforward and more practical terms, once you buy a book, CD, DVD, artwork or any other authorized copy of a copyrighted work, the copyright owner generally loses the right to control what you do with that specific copy. You can resell it, lend it, give it away, or even destroy it, without their permission.
Remote bricking requires software, which is sold under copyright law.
This also has nothing to do with exclusivity agreements arranged between companies, as seems to be the case here.
The 2013 U.S. Supreme Court case depended on a plaintiff that was making enough money on textbook arbitrage to fund a legal case all the way to the Supreme Court. It provided new clarity on book distribution and geographical "exclusivity".
If software enforcement of device distribution agreements affects a large enough flow of capital, then corner cases will accrue enough economic impact to be tested in courts. Manufacturers do not have carte blanche to manipulate hardware remotely, e.g. they cannot take actions that could injure humans. Where are the limits? For now, we have many opinions and few laws.