There are a ton of gremlins in Tesla just waiting to go off like time bombs. I recently got a very expensive repair just outside of warranty where the car simply refused to charge. Paid nearly 3000 for the diagnosis and repair for the car to start charging again - we are talking about a car with no wear and tear parts for the most part costing nearly 3K because a chip somewhere between the charge port and battery failed. My whole reason for buying it was that this would be the ultimate reliable car without ICE or brake wear, leaving tires as the only consumable. Guess I was wrong.
Even in combustion engine cars if ECM buckles up it usually is expensive to repair(at least in my part of the world). I guess with a car like Tesla, rich with electronics & unavailability of parts to 3rd party service centers; repairs can be very expensive.
Is any car company making a simple electric car without fancy features, or at least one where they're optional? Even the Nissan Leaf now has all sorts of high-end features. I'd much rather have a simple, cheaper, reliable car with less points of failure.
ECUs made in the last 10-20 years don’t really ever fail. Bosch has really figured things out and produces reliable ECUs for most OEMs.
The repair on your Tesla will cost more than a motor or transmission replacement on a conventional vehicle.
Tesla owners should file a class action lawsuit to force Tesla to extend their warranty to cover this issue. Tesla will try to fight it and make you pay out of pocket.
That's kinda expected. Once a car becomes a pile of buggy software, someone needs to maintain it and fix bugs. That someone is called a SW engineer and happily charges 200-300/hour. Add on top of that the service dep fees that can be substantial (e.g. I pay ~150/hour for mechanical repairs, but the actual mechanic looks like a guy who works for a minimum wage).
The high hourly pay of a software engineer is not significant when that high cost is spread over hundreds of thousands of units sold.
A corrolary of this is that when it is not spread over a high number of units, in a niche product, it will be very expensive so we will see even less niche products as software takes over more functions.
And for many of us Tesla was the biggest intersection between DRM, @InternetOfShit , and #righttorepair . And even just the whiff of one of those was enough to be a hard no.
And for those of you whom know their history, this has all the hallmark signs of anti-trust.
Well now that you're out of warranty you no longer need to go to the dealer so to speak, you can go to aftermarket repair shops. Yes these do exist for Teslas.
Yup. The ICE engine reliability issue is more or less solved. I have built hotrodded engines in my garage that are still going strong at 275,000 miles. The real issue is that tesla reinvented the wheel with everything else too. Other solved problems like rustproofing, doorseals, and component access were overlooked for technology adoption. I admire the bold move, but it was not one destined for reliability. At the end of the day, the tesla is more complex, even in the drivetrain, than my honda civic, and costs quite a lot more, even under ideal conditions.
The earlier models were premium not economy pricepoints. Why is the ECU designed around low cost on-board, non-replaceable constructs like this? A premium product, would not nickel-and-time the SD card, it would have engineered it as a field replaceable unit, with an insert,format, wait (for download) and go mode.
Replacing the entire ECU feels like an admission like Apple and increasingly Lenovo (thinkpad), they have driven premium to the wrong place. "please give us soldered on memory and soldered on SSD" is not actually what we said, on our feedback forms to Apple, or IBM (X1 Carbon).
Nobody asked Tesla to shave $1 on the ECU assembly and the incremental cost of recall and fix, along with goodwill now outweighs the cost saving in any sense I can understand it. If the authorities demand he fix for free, it becomes another shit show of the TSLA share price. This wasn't a good choice against all the other choices.
Lenovo still makes Thinkpads that are extendable. My P51 has 3! m.2 (2 of which support NVMe,) 4 RAM slots, easily replaceable keyboard and more. But I think most people want thinner laptops, and the thinner ones have less of this. I don’t think this is a headphone jack type situation: people value portability a lot on laptops, and the difference between a rugadized, user expandable laptop and an “ultrabook” are striking.
My dell xps is super thin and it still has easily replaceable ram and battery. The massive use of glue, rivets, proprietary screws and cryptographic verification in macbooks is not for slimness but for planed obsolescence
Lighter maybe, not thinner. I There's no reason for laptop to be thinner than ethernet or VGA ports. Give me full size mechanical keyboard instead (minus numpad).
I don't want thinner, but I want the 13 inch form factor. I could change the memory, the HDD and the keyboard by removing a single screw on my old x220, and with zero risk of breaking little plastic legs.
> . Why is the ECU designed around low cost on-board, non-replaceable constructs like this?
What's not replaceable or too expensive about a $400 or so nVidia Tegra board in a $100k car?
> Nobody asked Tesla to shave $1 on the ECU assembly
Are you suggesting it would have cost Tesla $1 per car to go from an industry standard board used in many cars to a custom solution with all parts tested in-house?
Why doesn't any other manufacturer of cars have a similar problem? Did I miss the story where BMW or Mercedes demand $1200 to replace this part? Tesla knew it wanted a highly digital logging investment and they didn't design for use life. Doesn't feel like a sensible place to be.
But to your main point, no. The actual cost per board of getting a design done for replaceable parts would have been higher. Once they hit volume the amortization would kick in. But for the earliest, yes,expensive. Strangely in the world of eg routers, the first gen often has more hackabilty, more removable parts and it's the later ones which shrink down, gets the bill of materials down.
Seriously, they should comp all the replacements for the life of the car, just for p.r. value.
Tesla needs to have stock relative to replacement rate purchased before nVidia's last ship date. So if they used eMMC built in to a part like this then they should be raising problems for their parts logistics.
At the rate they were rushing to design and build that hardware. I'm absolutely not surprised they chose not to go through the extra steps of qualifying a connector/mount for a flash daughtercard.
It's very likely the hardware designers didn't expect the software living on that flash to iterate that many times.
In too many ways, Tesla was and still is a hardware startup.
Every write up I've seen indicates the amount of stuff being put into /var is inappropriate for flash, beyond that, those logs should be kept in RAM until shutdown then you commit them to disk.
I'd have no issue with this stuff bricking if Tesla replaced it free (even more galling that they won't on a 50k+ vehicle), you'll have a hard time convincing anyone that a ECU should last less than a decade under normal operation, Nor should you need to replace the ECU before you replace the battery pack on an electric car.
If storage space was alone an issue, then they shouldn't have extended the features of those firmware loads so much - as far as I know the owner has no choice on if you update or not.
If you buy an expensive car, produced in low volumes, with little competition in the independent repair shop and aftermarket parts industry, expect to pay through the nose for maintenance, repairs and body work.
That also applies to other expensive consumer durable goods.
I think normally for automotive components you are obligated to use electronic components that have already been stress tested and are rated for more violent environments (like next to an engine).
I wonder if Tesla maybe skirted this (like "oh this computer isn't in the car it's an attachable add-on).
Source: I interned at an electronics component company back in uni as a chip tester. Mostly same chips to everyone but ones rated for automotive or space tended to get a lot of extra testing (since failures on electronics are bimodal, running tests in extreme environments just for a bit catches a lot of the bad apples).
It's worse. Elon Musk used to be bragging that they used consumer grade electronics instead of automotive grade (which is higher than industrial grade).
>> on our feedback forms to Apple, or IBM (X1 Carbon).
Whoa Whoa Whoa ;-)
As somebody who has 9 ThinkPads around his house, including the controversial T25, don't forget Lenovo completely bought and owns all Thinkpad design production branding etc for the last decade.
Would IBM have maintained their priorities if they owned the product today? Not sure. But Applezation of the ThinkPads has happened strictly under Lenovo :-<
The P series is still pretty good; X1 was never about modularity but strict portability; it's the devolution of T series that bothers me personally the most - the T420s was barely thicker than modern T490, but had swappable battery and hard drive. Not "user replaceable", literally push a button and swap it out.
[Disclaimer: IBM employee, but in a completely unrelated part and a ThinkPad fan before my IBM employment; I work in software consulting on large ERP projects and haven't seen an IBM hardware person in my 15 years of service:]
There’s no excuse for this at all. It’s shitty short sighted engineering. Simple as that.
The main firmware should have been on a separate physical device if data was write heavy regardless of “enough space” being left. It would have cost a few dollars to make this issue go away.
The more I hear about the engineering the longer I will avoid buying an electric vehicle. Iterating big problems away on a tangible chunk of engineering rather than a piece of software is stupid.
A couple days ago I found this link [0] to a Reddit post on HN about the software engineering practices at Tesla. Don't know how accurate the story is, hard to tell, but it sure was worrisome.
> The more I hear about the engineering the longer I will avoid buying an electric vehicle.
You can look up the nVidia Tegra board on Wikipedia and see for yourself that the "shitty short sighted engineering" board is used in many gasoline cars.
Yes, but those ICE cars were designed by engineers who understood that you only get ~100ks of write endurance with this sort of nonvolatile memory.
Tesla's use case is unique because they stuff enormous firmware images into the on-board NVM and write to it constantly with very heavy logging. Of course your Flash/EMMC will wear out quickly if you constantly write to it while your device is on - that is the short-sighted part.
The board isn't necessarily a problem, but how Tesla chose to use it sounds like a very predictable problem.
There was once a guy that shared instructions on how to resolder that chip (Get dump of software, root it, solder new chip with more memory). It was instruction on how to root Tesla. But oh, he deleted his content and left some comments:
> As a result I've taken down my howtos. I'm sorry about this but there is some kind of political problem which I don't understand.
I'd be surprised if it's any different to modding your BIOS in a PC motherboard: Buy a SOIC-8 clamp, a USB interface and bob's your uncle. Unless the ECU is potted in epoxy.
Doubt it’s an SOIC. More likely a 40 or 48 pin TSOP or the like.
But if it was an SOIC... just lift the power pin, power the chip externally and find some other points to tap into the clock and data lines and dump away. Then solder the legs of the new one on one side and connect VCC on the other side.
Wear leveling isn’t the issue here as eMMC is wear leveled. The issue is probably just that the total volume of writes has literally just worn out too much of the flash.
Firmware doesn't update as often though so it'd have to be static wear-leveling. I'm not sure of the safety of doing this with firmware files though - could it actually decrease the life expectancy by moving critical blocks more often?
In any case, if 90% of the space is filled with static firmware files then wear-leveling becomes difficult. Probably best to turn it off and let the physical blocks with the logs fail while keeping the firmware blocks intact, or just stop writing logs after a certain point.
> could it actually decrease the life expectancy by moving critical blocks more often?
No. You do normal, dynamic wear leveling. When you notice that the block you want to write to is much more worn than the rest-- say 100 erases extra-- you move a firmware block to it instead and claim that for use. Then that block sits near-quiescent for a long time until it's time to put it back into the rotation of blocks that get actively written. Static wear leveling can just be a tiny, tiny share of write amplification-- a factor of 0.01 or even less-- while keeping all the blocks in play.
You only "lose" from static wear leveling if you end up moving something right before it becomes active again. Whatever your strategy, there's always a degenerate write pattern which will cause useless write amplification, but even relatively simple schemes are robust against basically everything except deliberately malicious write patterns, and you can bound this worst case.
Incidentally, one way flash fails is by dielectric breakdown increasing leakage causing worn-but-idle-blocks to slowly accumulate error. Moving / occasionally scrubbing static, important data actually helps.
Or write the logs to a separate and easily replaceable module/drive. Putting the firmware on the same chip as what you're regularly writing to is probably a bad idea.
This doesn't bode well for classic and antique car collectors of the future, who will want operable Tesla cars.
There are Ford Model T cars that still drive. They were made 92 to 111 years ago.
With undocumented and cryptographically signed interfaces between components, it will be impossible to keep a Tesla running that long. The best you could hope for is to graft the body panels of a Tesla onto the innards of a newer car, merely dressing up the newer car to cosmetically look like a Tesla. Such a hack would not be have vehicle title as a Tesla, would not drive like a Tesla, and would not have an interface like a Tesla.
This goes for any modern car, Tesla is no exception here. Cars contain large numbers of embedded computers, chances are that in every car there is at least some time-bomb like bug that the manufacturers aren't aware of that will kill the car in a few decades.
There is already some development of open source ECUs:
A problem like this could be significantly alleviated by not having the flash soldered onto the motherboard. This is one of my concerns with the current generation of Apple MacBook laptops. Considering the cost of a Tesla as a whole, surely it can't be ridiculously more expensive to put a m.2 socket and discrete flash module on a motherboard.
> surely it can't be ridiculously more expensive to put a m.2 socket and discrete flash module on a motherboard.
It would be more expensive for Apple because people would buy cheap models and later upgrade, instead of buying expensive models or buy new machines early.
I suspect that it may be worse if they didn't solder given that it is in a vibration prone environment. The underspeccing for wear is more of an issue than the soldering in my opinion since if you are dealing with chips anyway you are already pretty damn elbows deep into a specialized area.
Hackers are already ripped apart Teslas and building things from the parts. It's VERY common to use the battery packs (thus their high resale value). Motors are popular as well. Sure there's some crypto/DRM for things like the car's identity (and if it has free supercharging for life). But generally a large part of a Tesla would be usable even 100 years from now, even if Tesla (the company) is dead.
My estimate is that 50 years from now that encryption will be crackable.
Alternatively there may be holes that can be used--such as a 'boundary scan' of one of the chips yielding the key. I don't have a spare Tesla sitting around to test this theory, however (grin)
Might not be that long. In the early 2000s 256-bit RSA keys were common. These days manufacturers don't care about twenty-year-old cars an owners have factored the keys.
1. Let's go back to the good old HDDs. There are even automotive industry ones (I own one)
2.There's a difference (8-fold) between Gb (gigabit) and GB (gigabyte). Memory chip capacities (and data transfer rates) are often expressed in bits (per second), but memory device (RAM stick, storage,...) capacities and file sizes are always in bytes.
Readit News
Maybe "Made by Tesla" is the new "Made in China".
Edit: Thanks for the suggestions!
The repair on your Tesla will cost more than a motor or transmission replacement on a conventional vehicle.
Tesla owners should file a class action lawsuit to force Tesla to extend their warranty to cover this issue. Tesla will try to fight it and make you pay out of pocket.
A corrolary of this is that when it is not spread over a high number of units, in a niche product, it will be very expensive so we will see even less niche products as software takes over more functions.
The mechanical parts of a well maintained ICE car rarely die for no reason. Electronics on the other hand...
And for those of you whom know their history, this has all the hallmark signs of anti-trust.
Enjoy replacing literally every part of your car if you expect it to last close to a decade.
When I was considering a Model3 they harped on how reliable it would be. I still passed.
Only the gearbox (automatic) will cost you not less than 8000€.
ECU fails? Add 5000€
Deleted Comment
Replacing the entire ECU feels like an admission like Apple and increasingly Lenovo (thinkpad), they have driven premium to the wrong place. "please give us soldered on memory and soldered on SSD" is not actually what we said, on our feedback forms to Apple, or IBM (X1 Carbon).
Nobody asked Tesla to shave $1 on the ECU assembly and the incremental cost of recall and fix, along with goodwill now outweighs the cost saving in any sense I can understand it. If the authorities demand he fix for free, it becomes another shit show of the TSLA share price. This wasn't a good choice against all the other choices.
I don't know why you are lumping Lenovo ThinkPad in there. Their units are still highly repairable.
Even their thinnest model (X1 Carbon) still has the replaceable SSD. It accepts any standard NVME M.2 drive - https://www.windowscentral.com/how-upgrade-ssd-lenovo-thinkp...
What's not replaceable or too expensive about a $400 or so nVidia Tegra board in a $100k car?
> Nobody asked Tesla to shave $1 on the ECU assembly
Are you suggesting it would have cost Tesla $1 per car to go from an industry standard board used in many cars to a custom solution with all parts tested in-house?
But to your main point, no. The actual cost per board of getting a design done for replaceable parts would have been higher. Once they hit volume the amortization would kick in. But for the earliest, yes,expensive. Strangely in the world of eg routers, the first gen often has more hackabilty, more removable parts and it's the later ones which shrink down, gets the bill of materials down.
Seriously, they should comp all the replacements for the life of the car, just for p.r. value.
https://developer.nvidia.com/embedded/community/lifecycle
Tesla needs to have stock relative to replacement rate purchased before nVidia's last ship date. So if they used eMMC built in to a part like this then they should be raising problems for their parts logistics.
It's very likely the hardware designers didn't expect the software living on that flash to iterate that many times.
In too many ways, Tesla was and still is a hardware startup.
I'd have no issue with this stuff bricking if Tesla replaced it free (even more galling that they won't on a 50k+ vehicle), you'll have a hard time convincing anyone that a ECU should last less than a decade under normal operation, Nor should you need to replace the ECU before you replace the battery pack on an electric car.
If storage space was alone an issue, then they shouldn't have extended the features of those firmware loads so much - as far as I know the owner has no choice on if you update or not.
are there such connectors suitable for automotive environments ? you don't want your flash to fall off while driving
That also applies to other expensive consumer durable goods.
I wonder if Tesla maybe skirted this (like "oh this computer isn't in the car it's an attachable add-on).
Source: I interned at an electronics component company back in uni as a chip tester. Mostly same chips to everyone but ones rated for automotive or space tended to get a lot of extra testing (since failures on electronics are bimodal, running tests in extreme environments just for a bit catches a lot of the bad apples).
And it's already caused more issues: https://www.thedrive.com/tech/27989/teslas-screen-saga-shows...
This + the horror stories I read about getting replacement parts when you damaged the car is what keeps me from buying a model 3.
Whoa Whoa Whoa ;-)
As somebody who has 9 ThinkPads around his house, including the controversial T25, don't forget Lenovo completely bought and owns all Thinkpad design production branding etc for the last decade.
Would IBM have maintained their priorities if they owned the product today? Not sure. But Applezation of the ThinkPads has happened strictly under Lenovo :-<
The P series is still pretty good; X1 was never about modularity but strict portability; it's the devolution of T series that bothers me personally the most - the T420s was barely thicker than modern T490, but had swappable battery and hard drive. Not "user replaceable", literally push a button and swap it out.
[Disclaimer: IBM employee, but in a completely unrelated part and a ThinkPad fan before my IBM employment; I work in software consulting on large ERP projects and haven't seen an IBM hardware person in my 15 years of service:]
The main firmware should have been on a separate physical device if data was write heavy regardless of “enough space” being left. It would have cost a few dollars to make this issue go away.
The more I hear about the engineering the longer I will avoid buying an electric vehicle. Iterating big problems away on a tangible chunk of engineering rather than a piece of software is stupid.
[0] https://www.reddit.com/r/EnoughMuskSpam/comments/99sbwa/form...
You can look up the nVidia Tegra board on Wikipedia and see for yourself that the "shitty short sighted engineering" board is used in many gasoline cars.
Tesla's use case is unique because they stuff enormous firmware images into the on-board NVM and write to it constantly with very heavy logging. Of course your Flash/EMMC will wear out quickly if you constantly write to it while your device is on - that is the short-sighted part.
The board isn't necessarily a problem, but how Tesla chose to use it sounds like a very predictable problem.
> As a result I've taken down my howtos. I'm sorry about this but there is some kind of political problem which I don't understand.
https://www.diyelectriccar.com/forums/showthread.php/howto-t...
Pity the site is not available on waybackmachine.
But if it was an SOIC... just lift the power pin, power the chip externally and find some other points to tap into the clock and data lines and dump away. Then solder the legs of the new one on one side and connect VCC on the other side.
Its still possible to do this. You do wear level between the blocks containing the firmware and the logs.
In any case, if 90% of the space is filled with static firmware files then wear-leveling becomes difficult. Probably best to turn it off and let the physical blocks with the logs fail while keeping the firmware blocks intact, or just stop writing logs after a certain point.
https://en.wikipedia.org/wiki/Wear_leveling#Static_wear_leve...
No. You do normal, dynamic wear leveling. When you notice that the block you want to write to is much more worn than the rest-- say 100 erases extra-- you move a firmware block to it instead and claim that for use. Then that block sits near-quiescent for a long time until it's time to put it back into the rotation of blocks that get actively written. Static wear leveling can just be a tiny, tiny share of write amplification-- a factor of 0.01 or even less-- while keeping all the blocks in play.
You only "lose" from static wear leveling if you end up moving something right before it becomes active again. Whatever your strategy, there's always a degenerate write pattern which will cause useless write amplification, but even relatively simple schemes are robust against basically everything except deliberately malicious write patterns, and you can bound this worst case.
Incidentally, one way flash fails is by dielectric breakdown increasing leakage causing worn-but-idle-blocks to slowly accumulate error. Moving / occasionally scrubbing static, important data actually helps.
There are Ford Model T cars that still drive. They were made 92 to 111 years ago.
With undocumented and cryptographically signed interfaces between components, it will be impossible to keep a Tesla running that long. The best you could hope for is to graft the body panels of a Tesla onto the innards of a newer car, merely dressing up the newer car to cosmetically look like a Tesla. Such a hack would not be have vehicle title as a Tesla, would not drive like a Tesla, and would not have an interface like a Tesla.
There is already some development of open source ECUs:
https://www.google.com/search?q=open+soure+ECU
For ICE based cars, you can expect something similar to happen for electric cars.
It would be more expensive for Apple because people would buy cheap models and later upgrade, instead of buying expensive models or buy new machines early.
No, the best you can do is root the car and fix the software as you like.
https://teslamotorsclub.com/tmc/threads/let-the-hacking-begi...
Alternatively there may be holes that can be used--such as a 'boundary scan' of one of the chips yielding the key. I don't have a spare Tesla sitting around to test this theory, however (grin)
2.There's a difference (8-fold) between Gb (gigabit) and GB (gigabyte). Memory chip capacities (and data transfer rates) are often expressed in bits (per second), but memory device (RAM stick, storage,...) capacities and file sizes are always in bytes.