I have worked in card payment industry. We would be getting products from China with added boards to beam credit card information. This wasn't state-sponsored attack. Devices were modified while on production line (most likely by bribed employees) as once they were closed they would have anti-tampering mechanism activated so that later it would not be possible to open the device without setting the tamper flag.
Once this was noticed we started weighing the terminals because we could not open the devices (once opened they become useless).
They have learned of this so they started scraping non-essential plastic from inside the device to offset the weight of the added board.
We have ended up measuring angular momentum on a special fixture. There are very expensive laboratory tables to measure angular momentum. I have created a fixture where the device could be placed in two separate positions. The theory is that if the weight and all possible angular momentums match then the devices have to be identical. We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.
First, wow this is both incredible and crazy! Both the China-side hacks and your side's anti-hack. Mind. Blown.
Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?
This wasn't our device. There was a big, reputable company behind the device. We were ordering a number of those and they would be shipped to us directly from China.
Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device.
Fortunately this only lasted for few months until it was dealt with. It was quite new back then (a decade ago) and it was a surprise for everybody I guess.
> Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?
I'd like to know this too. Has the West completely lost the ability to mass produce microchips at even a reasonable cost for financial applications?
You are underestimating the FUN of playing anti-anti-^N-hacks. I have had the privilege to be paid to so anti-anti-^N-hacking on a firewall thingy in the past and it was a challenge and a joy!
When I worked in telecom (a while ago) the manufacturing was shifted from China to Thailand/Other SE Asia due to this. The Thai companies weren't as efficient, but were much more open and honest when problems would arise., plus they didn't blatantly steal tech
> as once they were closed they would have anti-tampering mechanism activated so that later it would not be possible to open the device without setting the tamper flag
You didn’t specify what type of anti-tamper was used, but I wanted to jump in and say usually that means nothing. The US government intercepted packages [0] and put in back doors (removing and replacing the seals), so I’m not sure why you were so quick to dismiss state sponsored attacks by something as simple as an anti-tamper seal. You can learn how to do it yourself at most medium to large hacker conferences too (DEFCON, BlackHat, HOPE, and CCC to name a few, but there’s more with it).
You can just buy counterfeit anti-tamper stickers but if there is a switch inside the unit that flips a bit in some sort of write-once memory, then that would require removal of an entire chip and replacing it with another that may not be 100% the same. You can have a chain of trust in the system where chips will only talk to each other if they all spit out the right hash. Bury the SPI/I2C lines you use for this trust check within the PCB so you can't access it without drilling the card and add a layer of anti-tamper traces that trigger another tamper event if disturbed. Now what was just a quick install has turned into a whole PCB rework job where you are having to swap all the chips with a virgin set, assuming you can't get your hardware in there prior to final assembly.
Use an AES256 key from the factory to hash the chip's burned-in serial number and the time from the RTC. Lock out JTAG interfaces so once the chips are burned, they are inside their own fortress. There are a ton of ways to really lock down the hardware other than a shiny sticker and weird screws. Those keep people from breaking their own hardware, anti-tamper tech in chips keep out bad guys.
Payment systems are typically better defended than by just a sticker.
It’s not surprising to see a ton of tamper switches, vibration/shock sensors, even light sensors. And they’re all powered by an internal batter and separate MCU that will brick the device upon open.
Wait a minute... So your company has a Chinese equipment supplier, finds out that the supplier is tampering with your purchased equipment, and your solution is to add criteria to the incoming inspection?
No wonder China keeps screwing with you guys. You aren't supposed to eat that cost! Write a PO with tons of fine print that says "We will disassembly units at random for compliance inspection. Non compliant products will be returned at the suppliers expense." And then add a clause that says ">3 non-compliance events in under # months will result in the entire PO (10 or 20 units) being returned and all contracts cancelled."
I cannot believe you are getting screwed by a company you choose to do business with and yet you eat cost to ensure they aren't screwing you. Just get a new supplier! Do on-site inspections at their facility. This is nuts.
There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company.
The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty (assuming the company was involved which they could not have been as they have been the ones damaged the most).
If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.
The way to work is then verify everything and politely point it out. If you notice they will correct apparent mistake.
he's saying that his company knowingly allows a percentage of fraud, identity, and financial theft to occur against his customers' customers because there are enough layers of indirection where they (he and his bosses) probably can't be held personally responsible. Enough efforts (fancy tables with big receipts, etc) have been provided for coverage in case of a court case or media blitz but actually using a reputable supplier is clearly too expensive.
I can't believe the parent comment is so brazen. It makes me physically sick.
This problem is not specific to China, it's a problem of going with the lowest priced supplier. Your proposed recourse presumes that the supplier is making enough money on the deal to engage in a contract with teeth... many are not. Also, even if they moved everything to the U.S. they'd likely have different flavors of the same types of issues (i.e. did the NSA embed someone in the company to implant backdoors etc.) Fraud, sabotage and espionage aren't confined to specific countries. When greed drives you to move operations to the absolute cheapest places you can find on Earth, that's not going to be without its own set of risks and problems.
Honestly, to anyone bashing GP, look into your pockets, laptops, watches, cars, TVs, routers, CCs, singing toys, bitcoin mining ASICs. All with sealed black-box chips. Sure we can x-ray a couple of randomly stripped chips, but each one?
Now show me alternatives when most consumer grade electronic parts are fully or partially made in China.
Maybe you and me are missing something here. It seems crazy that somebody would go to these lengths without switching suppliers, there must be an underlying reason that is assumed to be understood by the informed reader. (But isn't understood by me and you it seems.)
> We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.
You mean moment of inertia, not angular momentum.
You could measure all of them! Given the moments for the three principal axes at any point, you can use the parallel axis theorem to calculate all the rest. In general, there are 10 degrees of freedom: 3 for the position of the center of mass, 3 for the axes, three for the moments, and for the the total mass.
For a nicer way to count and to do the math, you have the inertia tensor at the CM (a 3-dimensional rank-2 symmetric tensor, 6 DoF) plus the location of the CM and the mass.
In any event, this is a cute tampering-detection trick, but I would have started with an X-ray or CT scan.
Another implication of the parallel axis theorem is that the attacker could perfectly mimic every moment of inertia by shaving plastic. They wouldn't have to know which two axes were being tested because there are only three real numbers worth of information in the system to begin with (once center of mass and total mass have been dealt with.) In the whole MOI tensor there are only six free numbers which sounds like a reasonable number of parameters to fix by adding and removing small amounts of material.
What was this company doing in hiring an untrustworthy manufacturer to handle secure devices? That's playing a game you've lost from the start. Not every problem is technical!
I enjoyed your comment. But you are looking at the difference in the inertia tensor of an adulterated board from a non-adulterated one. Not the inertia tensor itself. Signal to noise problems with the measurement.
Seriously, why are we still outsourcing chip manufacturing to other countries? Sure it's cheaper, but we sacrifice a lot to have a society of corporate slaves build our tech. Security, core domain knowledge, capability, corporate secrets, patent rewards and enforcement, etc... All of it you throw away the minute you ship your manufacturing out of the country. I've seen enough board printing machines out there to start working on our own. As a country, we need to close this gap, more automation and capability and there will be no need to outsource circuit board printing and manufacturing. We will be much better off.
Canadian steel is considered by this administration to be a national security risk. But Chinese made boards and chips installed in weapons systems and crucial data centers? No problem.
Counterpoint: even if we ignored the fact that you cannot possibly produce the volumes of chips necessary at the price necessary in your country rather than in "we don't have to acknowledge all the human rights violations" countries, why would you believe this problem goes away if chip manufacturing were done in your own country, rather than another?
The moment the option of taking control of a production line of something _this important_ becomes available, your local specialized organized crime outfits will start to figure out ways to insert themselves into those production lines, learning the ins and outs, and figuring out a way to get something, anything, in there that won't be noticed but will give them a hook into millions of systems.
The law does not prevent crime. It just puts a price on it. While that price is typically too high for individuals, for organizations that have no business registration to revoke, and no CEO to drag to court, it is an entirely trivial cost.
It's hard to compete domestically against low prices caused by China's completely different standards for wages and human rights for laborers. Plus, here we don't like the idea of manufacturing industries being subsidized by the government (except in the case of "defense" of course...), while China obviously has no qualms with doing so.
You're conflating a few things there. Whilst companies like to make noises occasionally about saving the planet, doing the right thing, making (your country goes here) great etc, it's just horseshit. They exist to make money. That's it.
Very interesting methods you used to detect the changes! Out of curiosity: was there a reason that taking an xray of the devices was not an option? Industrial/veterinary xray machines can often be had quite cheaply...
This feels cargo cultish. Products drop from the sky. One day they become poisonous. You have no idea how to reproduce them locally. So you come up with hacks to make then less dangerous.
We really need to get back into manufacturing if this is our brave new world.
Get an x-ray machine? They are surprisingly cheap pieces of hardware if you are willing to deal with a small area, low penetration image. Low penetration means no lead, which makes for something that's about as cumbersome as a large bar fridge.
It's mentioned in the article that X-Ray didn't help much: 'Gray or off-white in color, they looked more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment'
For the folks commenting below that we should bring the manufacturing back to the US, why wouldn't the bad guys just start bribing American workers to insert the attack hardware into devices made here?
It's not like Americans are somehow above being bribed.
The bad actors could be brought to trial in a United States court, which is a level of deterrent not included in offshore manufacturing. If a US manufacturer was found selling tampered chips then the company itself could be held liable. This would create a general pressure to increase onsight security.
There are other benefits to us based manufacturing, but you only brought up the crime aspect so I will leave the other benefits unsaid.
Americans can be bribed to do anything. However, you don't really need to bribe or ask someone to betray their country and fellow citizens when it's done on Chinese soil...
It's possible, but it's much, much, much easier for an American company to hold another American company accountable when something like this happens. Instead of having to go through all those hoops, you sue the pants off the manufacturer.
This reminds me of a story about, IIRC, Soviet intelligence personnel determining that a photocopy machine at their consulate in the USA was bugged by the CIA by measuring it's weight and comparing that value to the standard value published by Xerox.
is measuring the moment of inertia not standard? I think any mechanical engineer would suggest it for that kind of problem (disclaimer; I've suggested it before for the same problem)
Why don't you guys consider to expose this by suing the bad manufacturer? I believe this could help other truely honest manufacturers both in and outside China to beat the wrong doers.
The Chinese operation would simply shut down and reopen under a different name. And the credit card companies are always very worried about their brand image, so they are not interested in any negative publicity.
I do not for one minute believe, that china, whose production floors are spotless, watched like a hawk and every product tested, did not know this was being added. They would have been able to tell themselves by weight. Yes, product weight is used for shipping manifests too.
This is obfuscation of the fact, that china as a state actor has perpetrated this crime against our country period. full stop.
What good is angular momentum when the producer can have fluctuations in its supply chain? Yes you can see when devices are not the same, but what if that happens all the time, legitimately?
Gosh, I would love to read everything about this. I know there are some videos about anti-tampering card readers on youtube, but not on the feedback race between hackers and security.
Maybe a setup to measure inertia tensor and center of mass (in that setup's axes) will be easier and I think it's what your call "measuring all angular momentums".
Was the secret service contacted? Card skimmers are a big no no. Family friend works for Dept of weights and measures and finds skimmers all the time on gas pumps. Scary stuff.
Thanks for sharing this story, and I hope you aren't put off by the huge thread of people second-guessing your competence. We need more of your kind of story.
x-ray would have been easier, as others have said.
but you're talking about the addition of an entire board! probably on the order of 10% of the size of the main boards.
in this article, perhaps dumbed down or altered, they are talking about the addition of a single, tiny chip, too small to even be an MCU let alone have wireless capability (which BTW requires an antenna).
The main lesson I learned from my Dad’s employer (twice) was always have two vendors. You can play them off of each other. When I did contract work I saw how powerful getting out or putting for away the checkbook can be. Large vendors ignore you if the checkbook isn’t moving.
I say “twice” because they were also the biggest employer in town. I got out of there. Slim pickins for career opportunities that didn’t revolve around BigCo.
Sorry, but this doesn't sound as true or there is huge mistakes done in choosing supply chain for such sensitive matter.
How come company keep ordering devices from some unverified sources from China, and after hitting a wall keep doing same?
How do you accept shipment of such devices without randomly opening and inspecting sample(yes losing all data, but electronic inspection can be done).
How you didn't investigate that with Visa/Mastercard? Whoever does that, he will lose his payment terminal certification after such incident, because they will track them down by IC serials very quickly.
What if vendor changed power supply board or even components type on it, and your momentum or weight test will make false positive?
Unless... your employer or you buy single devices, on demand, from some shady aliexpress seller. But then, it is plain suicide.
While it may sound sensational this was more of an operational issue, really. We were told by Visa and Mastercard that it is not even a question if we are going to be targeted. If you work in payment card industry you are constantly being attacked and the only way is to make it part of the process to deal with those things. Our network was hacked but what was important was tight, almost mathematical processes around protecting very specific material like credit card data and PINs.
For example, PINs are only ever being in unencrypted form inside of Hardware Security Modules and only for the purpose of being encrypted with Visa/Mastercard exchange keys. The process was designed so that nobody has enough access to ever get enough cryptographic material to be able to decrypt anything, at least two or three people would have to collude to do anything.
It also happens that we put all our resources in investment in software for the platform locking ourselves in. It would be rash decision to change the platform and it would probably kill our company. Also we (correctly) gambled that it would be dealt with quickly.
Look, there are enough supply chain problems with counterfeits already, you don’t want to start thinking about malicious implants. Just google for it, it’s massive
Presumably the organisations responsible for hard coding backdoors in chip designs know how to test to confirm their presence.
Presumably some adversarial nation-states have moles inside these organisations > know how to remove them prior to fab.
Presumably these adversaries export genuine chips to their adversaries, thereby tricking them in to thinking the backdoors made it through the fab process, and only use chips that have the backdoors removed in their own critical infrastructure.
Presumably.
I’ve always had this fantasy of being a hextuple agent involved in this type of deep espionage.
Is there an article that describes a bit more in detail what the chips actually did (or were capable of doing)? They only say "the microchip altered the operating system’s core so it could accept modifications.", which I might interpret as circumventing signature checks to allow installing modified firmware on the systems? But how does the chip connect to the network and how does it receive commands?
That said, it's pretty scary that you can hide so much malicious functionality in such a small device, makes me wonder what might be hidden in my Lenovo. In any case it speaks highly of the auditing firm that they were able to locate this. I wonder if they performed an x-ray analysis of the board, as given the size of these chips it should be possible to embed such devices in one of the internal layers of the board as well, making them essentially invisible to optical inspection.
SuperMicro hardware has very extensive IPMI integration into the mothebroard, which amongst other things can take over and inject frames into the network interface, emulate a VGA device, talk to the CPUs serial lines directly, flash firmware, control the state of a number of physical devices- and this is what it supports just from the web interface it presents by default with the password "ADMIN:ADMIN". My money, based on experience attempting to harden their devices, is that any modification were injected into the IPMI hardware where most of this was already supported.
This stuff ends up being extremely difficult to disable. The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC invisibly to the OS and DHCPs on its own to open up an administration port, web interface, and some assorted call homes. You have to explicitly tell it to use the non-connected port, change credentials, and modify it so that it is not accessible within operating system as well. Hopefully while the machine is offline to prevented any automated scanning finding it within your network.
The number of times I'd end up nmaping our local networks and being able to remotely access production hardware with an interface that allowed me to reach this interface was maddening. The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely nu-authenticated avenues exposed. The course of action was always just to write off the hardware and bin it, because god knows what impact you could actually have using that interface.
> The organizations behind the new project each have already made substantial contributions to creating open source baseboard management controller (BMC) firmware. Now, working together, they will define the vision for a standard stack that can be used across systems and computing environments.
LinuxBoot and Open Compute OSF are working on open-source server firmware that can be measured on every boot and validated against hardware root of trust keys controlled by the server owner instead of the server OEM, https://www.platformsecuritysummit.com/2018/speaker/hudson/
The problem with an IPMI BMC is that if you have malware that roots the OS from the BMC that in turn roots the BMC to reinstall itself from the OS, then you can never actually get rid of it. I actually opened a bug with a major clustering vendor in about 2012 because of this. Their response was a docfix.
If people are interested in digging into how IPMI works every SuperMicro board I've ever seen uses some variety of ASPEED Baseboard Management Controller (BMC). Facebook uses the same chip in their open hardware projects.
> The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC
cool, thanks for that info.
> just to write off the hardware
maybe you could just standup the mgmt network but blackhole route it at each switch port. The mgmt NIC thinks it's working properly but it can't talk to anyone nor can anyone talk to it.
But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
To me, that makes it sound like they could download from a remote host and inject code and do literally anything.
If this is indeed the case, I'm surprised someone didn't catch something earlier when the device was calling "home" over their network. I'm wondering if China stole BlackRidge First Packet Authentication tech [1] to keep things dark. BlackRidge is... "involved" in IC and defense projects.
5. When a server was installed and
switched on, the microchip altered
the operating system’s core so it
could accept modifications. The chip
could also contact computers controlled
by the attackers in search of further
instructions and code.
So, in typical vulnerability/payload/exploit fashion, the board's bus is vulnerable by default, because the chip pierces all the usual lines of defense protecting against network and operator I/O. It carries a payload intended to target very common features used everywhere commodity servers are used, one that likely listens for DMA traffic on the bus, and alters the signal stream, by escaping upon the occurrence of a magic sequence, and inserting its own signal, before resuming the authentic stream in flight.
The payload could be pretty small, since the server boards are likely using OS packages that match the chipset. This limits the software to a small set of well known targets, Linux, Windows, Apple. Target their kernels, and you only have to snip out a small chunk of bytes, and splice your own pre-defined package in. Splice in a miniature runtime, that operates a turing complete set of operations, and open up a listener that waits for network access, and now, your payload can enable arbitrary code execution, irrespective of permissions.
Now, to exploit, the payload needs to time the opportunity to splice itself onto the disk correctly. If certain well-known chunks of code will always exist in each given operating system, then with every disk access event one just needs to wait for the inevitable moment those magic system-specific bytes travel over the bus, in order to replace the known bytes with the poisoned modification. Events might target when the bytes are originally installed with the OS, or every time the OS reads those known bytes back into live memory, from any source.
The total payload package could probably fit inside a couple of megabytes, pack on a few more for the "listen & splice" part of the attack to round out the entire mass, and all we know how much data an SD card can fit into say... five grains of rice?
It's not particularly magical, there's consumer chips around which are not a whole lot bigger (though obviously in a more standard package). You don't get a lot of resources, but you don't really need it if all the other frameworks are in place in other software. If this sort of thing is something you can buy on Mouser for a few cents, the espionage grade material is probably an order or magnitude more higher quality.
This sounds like speculation. I'm quite capable of coming up with my own unfounded speculation, but there is a real report out there with the actual details in that really needs to be made public, legally or otherwise. There ought to be a CVE about this. Where is it?
Hm, but DMA messages get distributed over a parallel bus and this chip seems to employ a serial interface, so I would assume that it's not directly connected to anything that requires high throughput (i.e. memory, disk and peripheral access).
They attacked the Base Management Controller. There's an article by Bruce Schneier from 2013 warning about exactly this attack. Quoting:
"Basically, it's a perfect spying platform. You can't control it. You can't patch it. It can completely control your computer's hardware and software. And its purpose is remote monitoring. At the very least, we need to be able to look into these devices and see what's running on them."
"Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."
To be fair, removing the last three words could make it sound like the Mormon church was beaming sermons to the adult film industry which would probably be even worse...
Interesting about how some of the trojan chips were hidden in PCB substrate layers to avoid optical detection. For a much stealthier approach again, it was shown in 2013 that slighting changing the dopant mask for a few gates on Intel Ivy Bridge chips could render RNGs insecure[1][2]. Mask inspection systems[3] are used to detect mask manufacturing defects, but the question then is, do those inspection systems use Super Micro motherboards?
We need to get the fuck out of China. It is becoming less credible to throw our hands up and say "China has all the silicon manufacturing, guess we have to put up with it!" - this is national security, both directly via hardware in the DoD and through our economic stability.
Saying "Well the Chinese companies are different" or "It's just rogue employees" or "We just have to accept it" is not good enough.
We need a little bit more economic nationalism and realize that this shit matters. How is it so shocking? How did a Canadian security firm find something before the CIA/DoD? How is this not a NatSec program to be inspecting this stuff through shell company orders and honeypots, like we have on the Internet?
> It is becoming less credible to throw our hands up and say "China has all the silicon manufacturing, guess we have to put up with it!"
The thing is, China doesn't. Most expensive chips are fabricated outside of China in nearby countries and shipped there for final assembly.
What goes on in China is usually human assembly. And that's what seems to have happened here -- in the process of putting chips on motherboards, someone added some bonus chips. It seems like it would be hard to do that without dedicated traces / solder points.
So the chip shown in the article looks like a typical SMD balun, it is a type of transformer used to adapt impedance between two transmission line. It’s designed to replace a series a lumped element (capacitor, inductors, resistors) normally used for impedance adaptation (in a T or Pi network). The most common used for the device is directly between an antenna an a RF front-end to serve as an antenna tuner.
Technically you could embed and power an RF front-end inside a “flavored” balun to intercept or alter any communication passing through that front-end or even use the antenna to communicate during the down time. So this literally would hack Wifi / Bluetooth at low level and inject code and at the same time create a mesh network of malicious devices to relay information. Welcome to IoT Cyberwarfare.
But this clever hack is probably not limited to RF and is likely to also be embedded in transformers used for isolating Ethernet lines. Common mode chokes (some SMD chokes also look like the chip they are showing) or even some integrated ESD protection solution would be an ideal target as they are inserted in series of the signal.
"But this clever hack is probably not limited to RF and is likely to also be embedded in transformers used for isolating Ethernet lines."
I was thinking along the same lines: a balun could harvest energy and use it for other purposes, but it should either store it for later use using maybe a tiny supercapacitor inside, or inject it into the data stream on the fly not unlike RFID dongles do. Or maybe just have enough power and memory to store a few hundred bytes and alter a few fields here and there, so that for example a network frame coming from a compromised machine can have its source field rewritten as it came from a trusted source.
In theory small pull up resistors also could be swapped with lookalike malicious parts capable of tampering with i2c traffic between boards peripherals so that devices can be activated/deactivated no matter what the CPU tells them.
We're going straight to the point when there won't be a single device in the world, from toasters to supercomputers through top brand network gear, that can claim and guarantee to be secure; inserting malicious hardware and firmware is getting just too easy for those with the necessary knowledge and resources. A technically interesting and scary scenario.
I don't know enough about this, but isn't the article saying that the appearance is deceptive:
The chips on Elemental servers were designed to be as inconspicuous as possible, according to one person who saw a detailed report prepared for Amazon by its third-party security contractor, as well as a second person who saw digital photos and X-ray images of the chips incorporated into a later report prepared by Amazon’s security team. Gray or off-white in color, they looked more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment.
If the photo in the article is real you wouldnt be able to identify this component as compromised "just" by visual (even xray augmented) inspection. TVS Diode Array looks the same from the outside, whats more its build in same way with silicon die embedded in tis structure. Other than signal analysis it would take decapping every single component of a motherboard to find this implant.
The article specifically says that the attack goes after the Baseboard Management Controller (BMC). In Supermicro equipment this takes the form of a separate SOC made by ASPEED that runs it's own operating system entirely separate from the main CPU but with access into the main system through various mechanisms. Facebook uses the same ASPEED chips in their open hardware projects.
Yes, TVS Diode Array (image google to see they look just like in the article) for ESD protection is the obvious candidate, inserted on serial communication lines - perfect spot for signal interception/injection.
It seems to me that embedding an RF front-end, back-end, and signal manipulation logic in a chip of the size shown in the article would be extraordinarily difficult. Assuming that feat was achievable, it would likely result in serious performance hits on that transmission line right? Signal latency would probably be a dead giveaway for something like EEPROM reads being manipulated.
you would be extremely hard pressed to fit that amount of logic (and code) into that small of a package. and to operate intelligently, as you suggest, it needs a CPU clock, not available to it from where it would sit. And it would need to do signal analysis in real time, with no power and no clock.
Readit News
https://news.ycombinator.com/item?id=18142277
https://news.ycombinator.com/item?id=18138990
https://news.ycombinator.com/item?id=18143569
Once this was noticed we started weighing the terminals because we could not open the devices (once opened they become useless).
They have learned of this so they started scraping non-essential plastic from inside the device to offset the weight of the added board.
We have ended up measuring angular momentum on a special fixture. There are very expensive laboratory tables to measure angular momentum. I have created a fixture where the device could be placed in two separate positions. The theory is that if the weight and all possible angular momentums match then the devices have to be identical. We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.
Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?
Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device.
Fortunately this only lasted for few months until it was dealt with. It was quite new back then (a decade ago) and it was a surprise for everybody I guess.
I'd like to know this too. Has the West completely lost the ability to mass produce microchips at even a reasonable cost for financial applications?
If the device is sealed with an anti-tampering system then the contents must be checked by a trusted entity before being sealed.
Trying to guess the contents of a box that you cannot open sounds a bit like madness.
You didn’t specify what type of anti-tamper was used, but I wanted to jump in and say usually that means nothing. The US government intercepted packages [0] and put in back doors (removing and replacing the seals), so I’m not sure why you were so quick to dismiss state sponsored attacks by something as simple as an anti-tamper seal. You can learn how to do it yourself at most medium to large hacker conferences too (DEFCON, BlackHat, HOPE, and CCC to name a few, but there’s more with it).
[0] https://www.techradar.com/news/networking/routers-storage/ph...
Use an AES256 key from the factory to hash the chip's burned-in serial number and the time from the RTC. Lock out JTAG interfaces so once the chips are burned, they are inside their own fortress. There are a ton of ways to really lock down the hardware other than a shiny sticker and weird screws. Those keep people from breaking their own hardware, anti-tamper tech in chips keep out bad guys.
It’s not surprising to see a ton of tamper switches, vibration/shock sensors, even light sensors. And they’re all powered by an internal batter and separate MCU that will brick the device upon open.
To echo, it's actually quite trivial to bypass anti-tamper stickers with acetone and a needle.
https://www.youtube.com/watch?v=SqkMIek8sqI
No wonder China keeps screwing with you guys. You aren't supposed to eat that cost! Write a PO with tons of fine print that says "We will disassembly units at random for compliance inspection. Non compliant products will be returned at the suppliers expense." And then add a clause that says ">3 non-compliance events in under # months will result in the entire PO (10 or 20 units) being returned and all contracts cancelled."
I cannot believe you are getting screwed by a company you choose to do business with and yet you eat cost to ensure they aren't screwing you. Just get a new supplier! Do on-site inspections at their facility. This is nuts.
The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty (assuming the company was involved which they could not have been as they have been the ones damaged the most).
If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.
The way to work is then verify everything and politely point it out. If you notice they will correct apparent mistake.
I can't believe the parent comment is so brazen. It makes me physically sick.
Now show me alternatives when most consumer grade electronic parts are fully or partially made in China.
Dead Comment
You mean moment of inertia, not angular momentum.
You could measure all of them! Given the moments for the three principal axes at any point, you can use the parallel axis theorem to calculate all the rest. In general, there are 10 degrees of freedom: 3 for the position of the center of mass, 3 for the axes, three for the moments, and for the the total mass.
For a nicer way to count and to do the math, you have the inertia tensor at the CM (a 3-dimensional rank-2 symmetric tensor, 6 DoF) plus the location of the CM and the mass.
In any event, this is a cute tampering-detection trick, but I would have started with an X-ray or CT scan.
What was this company doing in hiring an untrustworthy manufacturer to handle secure devices? That's playing a game you've lost from the start. Not every problem is technical!
Let that sink in for a moment.
https://www.wsj.com/articles/dont-trust-the-chinese-to-make-...
The moment the option of taking control of a production line of something _this important_ becomes available, your local specialized organized crime outfits will start to figure out ways to insert themselves into those production lines, learning the ins and outs, and figuring out a way to get something, anything, in there that won't be noticed but will give them a hook into millions of systems.
The law does not prevent crime. It just puts a price on it. While that price is typically too high for individuals, for organizations that have no business registration to revoke, and no CEO to drag to court, it is an entirely trivial cost.
I actually agree it might be better for the Americans to manufacture things in America - especially things used in critical government systems.
But this seems like a human problem - if all the factories were moved to America, couldn't those factory managers etc also be bribed?
American manufacturing in the 60s was rife with unions w/ ties to organized crime.
That's enough, full stop, say no more. The other costs are real yet they're either not marginal, are borne by others, or both.
We really need to get back into manufacturing if this is our brave new world.
You only need to measure three angular momentums, all other can be calculated. See https://en.wikipedia.org/wiki/Moment_of_inertia#Motion_in_sp...
"This shows that the inertia matrix can be used to calculate the moment of inertia of a body around any specified rotation axis in the body."
On the attacker side, they only need to make sure three angular momentums match.
It's not like Americans are somehow above being bribed.
There are other benefits to us based manufacturing, but you only brought up the crime aspect so I will leave the other benefits unsaid.
Dead Comment
Modern times, but same old methods of "debugging"
The angular momentum stuff is innovative though.
Angular Momentum of a body at rest is zero.
This is obfuscation of the fact, that china as a state actor has perpetrated this crime against our country period. full stop.
Can you elaborate on this concept a bit? I'm not familiar with this term and the sources I looked up were pretty physics-y and out of my depth.
This sounds like a very interesting and creative solution. Good lateral thinking on your part! :)
but you're talking about the addition of an entire board! probably on the order of 10% of the size of the main boards.
in this article, perhaps dumbed down or altered, they are talking about the addition of a single, tiny chip, too small to even be an MCU let alone have wireless capability (which BTW requires an antenna).
Deleted Comment
I say “twice” because they were also the biggest employer in town. I got out of there. Slim pickins for career opportunities that didn’t revolve around BigCo.
How come company keep ordering devices from some unverified sources from China, and after hitting a wall keep doing same?
How do you accept shipment of such devices without randomly opening and inspecting sample(yes losing all data, but electronic inspection can be done).
How you didn't investigate that with Visa/Mastercard? Whoever does that, he will lose his payment terminal certification after such incident, because they will track them down by IC serials very quickly.
What if vendor changed power supply board or even components type on it, and your momentum or weight test will make false positive?
Unless... your employer or you buy single devices, on demand, from some shady aliexpress seller. But then, it is plain suicide.
For example, PINs are only ever being in unencrypted form inside of Hardware Security Modules and only for the purpose of being encrypted with Visa/Mastercard exchange keys. The process was designed so that nobody has enough access to ever get enough cryptographic material to be able to decrypt anything, at least two or three people would have to collude to do anything.
It also happens that we put all our resources in investment in software for the platform locking ourselves in. It would be rash decision to change the platform and it would probably kill our company. Also we (correctly) gambled that it would be dealt with quickly.
Presumably the organisations responsible for hard coding backdoors in chip designs know how to test to confirm their presence.
Presumably some adversarial nation-states have moles inside these organisations > know how to remove them prior to fab. Presumably these adversaries export genuine chips to their adversaries, thereby tricking them in to thinking the backdoors made it through the fab process, and only use chips that have the backdoors removed in their own critical infrastructure.
Presumably.
I’ve always had this fantasy of being a hextuple agent involved in this type of deep espionage.
That said, it's pretty scary that you can hide so much malicious functionality in such a small device, makes me wonder what might be hidden in my Lenovo. In any case it speaks highly of the auditing firm that they were able to locate this. I wonder if they performed an x-ray analysis of the board, as given the size of these chips it should be possible to embed such devices in one of the internal layers of the board as well, making them essentially invisible to optical inspection.
This stuff ends up being extremely difficult to disable. The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC invisibly to the OS and DHCPs on its own to open up an administration port, web interface, and some assorted call homes. You have to explicitly tell it to use the non-connected port, change credentials, and modify it so that it is not accessible within operating system as well. Hopefully while the machine is offline to prevented any automated scanning finding it within your network.
The number of times I'd end up nmaping our local networks and being able to remotely access production hardware with an interface that allowed me to reach this interface was maddening. The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely nu-authenticated avenues exposed. The course of action was always just to write off the hardware and bin it, because god knows what impact you could actually have using that interface.
> The organizations behind the new project each have already made substantial contributions to creating open source baseboard management controller (BMC) firmware. Now, working together, they will define the vision for a standard stack that can be used across systems and computing environments.
LinuxBoot and Open Compute OSF are working on open-source server firmware that can be measured on every boot and validated against hardware root of trust keys controlled by the server owner instead of the server OEM, https://www.platformsecuritysummit.com/2018/speaker/hudson/
https://www.itworld.com/article/2708437/security/ipmi--the-m...
cool, thanks for that info.
> just to write off the hardware
maybe you could just standup the mgmt network but blackhole route it at each switch port. The mgmt NIC thinks it's working properly but it can't talk to anyone nor can anyone talk to it.
at the expense of a dedicated switch.
But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
To me, that makes it sound like they could download from a remote host and inject code and do literally anything.
[1]https://patents.google.com/patent/US8346951B2/en
https://freebeacon.com/national-security/military-warns-chin...
https://news.ycombinator.com/item?id=18144519
The payload could be pretty small, since the server boards are likely using OS packages that match the chipset. This limits the software to a small set of well known targets, Linux, Windows, Apple. Target their kernels, and you only have to snip out a small chunk of bytes, and splice your own pre-defined package in. Splice in a miniature runtime, that operates a turing complete set of operations, and open up a listener that waits for network access, and now, your payload can enable arbitrary code execution, irrespective of permissions.
Now, to exploit, the payload needs to time the opportunity to splice itself onto the disk correctly. If certain well-known chunks of code will always exist in each given operating system, then with every disk access event one just needs to wait for the inevitable moment those magic system-specific bytes travel over the bus, in order to replace the known bytes with the poisoned modification. Events might target when the bytes are originally installed with the OS, or every time the OS reads those known bytes back into live memory, from any source.
The total payload package could probably fit inside a couple of megabytes, pack on a few more for the "listen & splice" part of the attack to round out the entire mass, and all we know how much data an SD card can fit into say... five grains of rice?
https://www.microchip.com/wwwproducts/en/ATtiny4
For scale, this alone is about the size of a large SMD capacitor and would basically be lost in most designs today.
They probably found it out when they were repeatedly tried to reflash the BMC flash, and saw that checksums did not match.
Dead Comment
"Basically, it's a perfect spying platform. You can't control it. You can't patch it. It can completely control your computer's hardware and software. And its purpose is remote monitoring. At the very least, we need to be able to look into these devices and see what's running on them."
https://www.schneier.com/blog/archives/2013/01/the_eavesdrop...
Sure you can. OEMs regularly release patches for platform BMCs.
Do you know this, or are you speculating?
"The illicit chips could do all this because they were connected to the baseboard management controller..."
Well played, Bloomberg. Well played.
[1] Paper: http://www.emsec.rub.de/media/crypto/veroeffentlichungen/201...
[2] Presentation: https://www.iacr.org/workshops/ches/ches2013/presentations/C...
[3] Example system: https://www.lasertec.co.jp/en/products/semiconductor/mask_se...
Saying "Well the Chinese companies are different" or "It's just rogue employees" or "We just have to accept it" is not good enough.
We need a little bit more economic nationalism and realize that this shit matters. How is it so shocking? How did a Canadian security firm find something before the CIA/DoD? How is this not a NatSec program to be inspecting this stuff through shell company orders and honeypots, like we have on the Internet?
The thing is, China doesn't. Most expensive chips are fabricated outside of China in nearby countries and shipped there for final assembly.
What goes on in China is usually human assembly. And that's what seems to have happened here -- in the process of putting chips on motherboards, someone added some bonus chips. It seems like it would be hard to do that without dedicated traces / solder points.
Technically you could embed and power an RF front-end inside a “flavored” balun to intercept or alter any communication passing through that front-end or even use the antenna to communicate during the down time. So this literally would hack Wifi / Bluetooth at low level and inject code and at the same time create a mesh network of malicious devices to relay information. Welcome to IoT Cyberwarfare.
But this clever hack is probably not limited to RF and is likely to also be embedded in transformers used for isolating Ethernet lines. Common mode chokes (some SMD chokes also look like the chip they are showing) or even some integrated ESD protection solution would be an ideal target as they are inserted in series of the signal.
I was thinking along the same lines: a balun could harvest energy and use it for other purposes, but it should either store it for later use using maybe a tiny supercapacitor inside, or inject it into the data stream on the fly not unlike RFID dongles do. Or maybe just have enough power and memory to store a few hundred bytes and alter a few fields here and there, so that for example a network frame coming from a compromised machine can have its source field rewritten as it came from a trusted source. In theory small pull up resistors also could be swapped with lookalike malicious parts capable of tampering with i2c traffic between boards peripherals so that devices can be activated/deactivated no matter what the CPU tells them.
We're going straight to the point when there won't be a single device in the world, from toasters to supercomputers through top brand network gear, that can claim and guarantee to be secure; inserting malicious hardware and firmware is getting just too easy for those with the necessary knowledge and resources. A technically interesting and scary scenario.
The chips on Elemental servers were designed to be as inconspicuous as possible, according to one person who saw a detailed report prepared for Amazon by its third-party security contractor, as well as a second person who saw digital photos and X-ray images of the chips incorporated into a later report prepared by Amazon’s security team. Gray or off-white in color, they looked more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment.
Terrifying and neat; thanks for the explanation!
How easy is it for us (or rather the companies making the hardware) to detect and mitigate this?
it's not possible.