> Recommendation: Validate that the input of the function is not empty
> The Nomad team responded that "We consider it to be effectively impossible to find the preimage of the empty leaf".
> We believe the Nomad team has misunderstood the issue. It is not related to finding the pre-image of the empty bytes. Instead, it is about being able to prove that empty bytes are included in the tree (empty bytes are the default nodes of a sparse Merkle tree). Therefore, anyone can call the function with an empty leaf and update the status to be proven.
It seems like it was (at least arguably) impossible to exploit until they introduced a second issue:
>It turns out that during a routine upgrade, the Nomad team initialized the trusted root to be 0x00. To be clear, using zero values as initialization values is a common practice. Unfortunately, in this case it had a tiny side effect of auto-proving every message
EDIT:
Reading and noodling I'm 99% sure these are separate issues. The vulnerability talks about passing in an empty leaf to the prove function. But that's not what the exploit is.
The exploit is using an unproven message. So they are passing in an actual leaf to prove. The problem is that unproven messages have 0x00 as root and some jabroni set 0x00 as the trusted root. So every message was treated as proven by default when it should be the opposite.
Anyone who's studied NTSB reports knows this is almost always how they go; a single failure that is no problem becomes one because of some other issue that doesn't normally happen, or couldn't happen because normally a third thing is always done ...
I'm not completely sure the mechanics of this exploit, but I've recently adopted the personal guideline of "all enums and integer IDs start at 1", and 0 is simply an invalid value.
Not a hard and fast rule, and not something that will catch tons of problems, but now and again it does help me catch an uninitialized value.
If this legacy enum value had been handled later in the code, there would not have been a vulnerability.
(This isn't to say that the developers were bad. The person who wrote the code was extremely knowledgeable. It's just really hard to be perfect every time. )
That's common misinformation. The issue discussed in the review has no connection to what happened, and the narrative that it is the same issue, but it became exploitable after the update is incorrect too. It affects different part of smart-contract logic. Yes, it's pretty close at a glance—because you have zero-by-default problem somewhere close to Merkle tree.
>Messages popping up in public Discord servers of random people grabbing $3K-$20K from the Nomad bridge - all one had to do was copy the first hacker's transaction and change the address, then hit send through Etherscan. In true crypto fashion - the first decentralized robbery.
The old mantra of possession is 9/10ths of the law is and always has been false. If i have something i own it. That is the one fundamental truth. Now someone can come and try and take it back from me by force (Person, Court System, Rebels, Corporations) if they can exert more violence on 'me' than i can exert on 'Them'.
The problem with crypto is the 'keys' are what crypto is. No nationstate can come and take that away from me. They can kill/imprison/fine me, but then neither of us will have it. You would have to hack/fork the chain for that to happen (Which has happened) or find some social way around it (If i have it on a centralized exchange, if i have a hackable hard drive, found my keys on AWS, etc etc.) Additionally, with things like Monero, and tornado swap good luck trying to find them.
The international obfuscated c contest has taught me that programmers can make small mistakes on purpose and its almost impossible to identify legit mistakes from malfeasance.
If we have a situation where:
* Its hard to tell, after the fact, 'a mistake' was a bad actor.
* The programmers are, by and large, anonymous.
* The benefit of making 'a mistake' could be hundreds of millions of dollars that are not easily traced.
This situation seems rife for abuse and bad actors. Not saying it happened in this case. . . but how would you know?
> In Solidity, the order of evaluation of sub-expressions is unspecified. This means that in f(g(), h()), g() might get evaluated before h() or h() might get evaluated before g(). Practically, this order is predictable, but Solidity code shouldn’t depend on that behavior between compiler versions. In most circumstances g() is evaluated before h() (left-to-right order), which is also the behavior that most languages specify in their standards. However, in the case of emitting an event with indexed arguments, the arguments are evaluated right-to-left.
I feel that order-of-evaluation dependence is a special case of the general conflict between expression-oriented (functional-style) programming, and impure operations requiring sequential reasoning. Another case of this conflict is temporary values (expressions) with side-effectful destructors (sequential reasoning), for example https://fasterthanli.me/articles/a-rust-match-made-in-hell#w....
At this point, is it good practice to avoid using side-effectful procedure calls as parameters to other expressions (especially those with multiple inputs), but instead first assign to a temporary value to make order of operations explicit?
I've always enjoyed the underhanded C contest, but I don't think it's active anymore. Thanks for this. The 2022 entry that I saw was very much in the same spirit.
> hundreds of millions of dollars that are not easily traced
If they keep it in blockchains only, it's hard to connect to a real identity. But if they cross the line (which is everybody's goal eventually) to the real world, they can get caught as easy or even easier than in traditional financial system.
I don't how anyone would commit anything more than pocket change to a scheme where an insider could deliberately introduce a weakness and then exploit that weakness to walk off with all the funds committed.
Slipping an exploit into an npm package doesn't let you easily run away with tens/hundreds of millions of dollars in the same way web3 projects do.
That said, I personally doubt this happens much if at all, because if you want to scam on web3 you can just do a good old-fashioned pump&dump and nobody seems to be receiving any legal/criminal consequences as of yet.
‘Normies’ didn’t lose all of their money selling credit default options and swaps on mortgage-backed bonds. They wouldn’t have access to the markets for those instruments or the capital to do so, and if they had access or capital, they weren’t a normie.
Bad analogy. This explanation has actually been done pretty reasonably by a few people. Further than that - after such explanations, it's clear even to normal people who's to blame, whereas software and cryptography are much more esoteric in that department.
Of course there is no justice in either case, but at least normal people can see who is most appropriate to behead in the case of the traditional financial catastrophes, in the purely theoretical revolution.
If you got burned by mortgage backed derivatives and lost your life savings, it's ultimately because you were (knowingly or not) speculating on the value of real estate assets and making an assumption about future values of said assets.
In the case of Nomad, it's that you put yourself at risk by using their service you could've lost everything you put in.
>Challenge: explain to a normie that their life savings is gone forever because of a zero initialization vector.
You mistook a currency for an investment opportunity, and gambled your life savings on one thing. Currencies have always and will always fluctuate against each other. Diversify your investments.
I don't think that helps. Saying "5% of your money is gone because a developer fucked up, and you have no recourse." isn't going to go down well with anyone.
They’re not. The vast majority of early liquidity in this and most DeFi protocols is raised from institutions, VCs, and trading firms.
This is especially the case for protocols like Nomad that don’t yet have a native token. They’ll get liquidity commitments through over-the-counter SAFT agreements that give the VCs a percent of the future tokens.
The reason this happened is that Nomad's contract was "upgradable". This is a pattern where the source code of a contract is able to be replaced by a privileged developer account. This was not how Ethereum was intended to work and it actually needs some pretty convoluted stuff to make it work (see the UpgradeBeacon related code here: https://etherscan.io/address/0x88a69b4e698a4b090df6cf5bd7b2d...)
The reason developers make their contracts "upgradable" is simple greed- they want to be able to launch more quickly than other projects without needing to ensure their code will stand the test of time. This may be OK for a social networking app MVP, but it's not OK for a smart contract which a user ideally should be able to audit and understand (or at least rely on the audit of someone else). "Upgradable" smart contracts can always be changed after the fact, as happened here, which means that any audit is meaningless.
Top tier projects still do use simple un-upgradable smart contracts. Uniswap first wrote v1, then improved it and launched v2, then v3. The Uniswap v1 and v2 contracts are still running and usable, and will be for as long as Ethereum is around. Their security properties will always be the same as they were the day they launched.
"Upgradable" contracts mean that you are trusting your money to some anonymous fat fingered (or at worst, criminal) dev, and it could disappear at any minute. They defeat the entire purpose of even using a blockchain.
Yes - but not having them upgradeable means that if your contract is dealing with a lot of money and a small bug was discovered, you are unable to patch it after the fact, even if people are actively abusing that bug
It’s not really about greed. Deploying a program and having it unchangeable forever comes with risks, and more often when dealing with very complex applications, those aren’t worth it
People should NEVER touch any upgradable contract. It is literally centralized, and defeats the whole purpose of DeFi.
Yes, writing perfect code is very hard. But smart contracts are an example of code that must be extremely thoroughly tested, formally verified and so on.
But that doesn't go well with being first to market, move fast break things, etc.
Are you kidding me? They lost 150 million dollars and the only penalty is to write up the bug on twitter? These children are playing with peoples lives. There’s a body count to losing that much money
This is a really severe heist, but the latter part of your comment seems rather dramatic. Crypto is still generally not a medium of exchange, and most users are still speculative investors. Most of these investors have a hedge (or are using crypto as their hedge), except for the foolish.
There a monetary value you can attach to a human's life, as much as that seems to be taboo.
Depending on how you measure, that value is (in the US and Europe) typically in the order of 1..5 Mio USD.
So it's not outrageous to assume that losing 150m comes with a body count, even if the funds wouldn't have bee used to directly save or improve lives otherwise.
But when someone wisely observes that crypto is useless for anything except scams, some cryptoenthusiast answers, but no, it is useful as a medium of exchange for people in third world countries with difficult foreign exchange restrictions.
People are often in a prisoner’s dilemma, where they are relying on the developers/team to spearhead the investigation including the judicial investigation against the perpetrators with the chance of their being a financial remedy
and so therefore nobody is trying to kill or impair the developers/team
if you were referring to people committing suicide or being suicided by the people they borrowed money from, thats not everyone’s problem and people in those circumstances should re-evaluate to avoid that risk or accept that risk
It is important that users come to better understand the different risk profiles between:
1. Owning ETH with a non-custodial wallet.
2. Owning ETH on a CEX.
3. Depositing ETH into a smart contract to receive a wrapped asset. This includes rollups and L2s.
The majority of major crypto hacks[1] are in the 3rd group, and almost all of these hacks are related to protocol updates and governance. Either: the developers update their code, and accidentally push a bug, or one address or a group of addresses are allow-listed some privileged actions in the contract and that can become a weak point.
Proxying and governance isn't the only way to design contracts. Two examples counter to this that are more robust are WETH ($6B) [2] and ETH2 Deposit ($20B) [3] which cannot be attacked in this way. If users wanted a new feature from the WETH contract, they would have to manually migrate over to the new address. Eventually we might see this kind of design be applied to bridges and rollups.
Readit News
> QSP-19 Proving With An Empty Leaf
> Recommendation: Validate that the input of the function is not empty
> The Nomad team responded that "We consider it to be effectively impossible to find the preimage of the empty leaf".
> We believe the Nomad team has misunderstood the issue. It is not related to finding the pre-image of the empty bytes. Instead, it is about being able to prove that empty bytes are included in the tree (empty bytes are the default nodes of a sparse Merkle tree). Therefore, anyone can call the function with an empty leaf and update the status to be proven.
>It turns out that during a routine upgrade, the Nomad team initialized the trusted root to be 0x00. To be clear, using zero values as initialization values is a common practice. Unfortunately, in this case it had a tiny side effect of auto-proving every message
EDIT:
Reading and noodling I'm 99% sure these are separate issues. The vulnerability talks about passing in an empty leaf to the prove function. But that's not what the exploit is.
The exploit is using an unproven message. So they are passing in an actual leaf to prove. The problem is that unproven messages have 0x00 as root and some jabroni set 0x00 as the trusted root. So every message was treated as proven by default when it should be the opposite.
Not a hard and fast rule, and not something that will catch tons of problems, but now and again it does help me catch an uninitialized value.
yes, a routine upgrade. that's what it was..... (→_→)
The real issue was half-caught in a review on a pull request however. https://github.com/nomad-xyz/monorepo/pull/289/files
If this legacy enum value had been handled later in the code, there would not have been a vulnerability.
(This isn't to say that the developers were bad. The person who wrote the code was extremely knowledgeable. It's just really hard to be perfect every time. )
They are bad because they are not competent to write the decent code required by their profession and job environment.
In normal software writing trade, such engineers are called low performers and routinely managed out of any organization.
Sure, the mistake is not unusual from the perspective of general software engineering. But let's not forget what software they are working on.
I am totally fine with a bartender dropping a glass... I'll put a surgeon on trial if he cannot make his hands steady during a heart surgery...
Deleted Comment
>Messages popping up in public Discord servers of random people grabbing $3K-$20K from the Nomad bridge - all one had to do was copy the first hacker's transaction and change the address, then hit send through Etherscan. In true crypto fashion - the first decentralized robbery.
https://twitter.com/FatManTerra/status/1554258880380772352
I'm not a lawyer but I would be very surprised if courts in most countries would buy this argument.
Spelling or grammatical mistakes usually don't invalidate contracts in the real world and robbing a poorly secured vault is still illegal.
The thief was obviously trying to get other people's money without their consent.
The old mantra of possession is 9/10ths of the law is and always has been false. If i have something i own it. That is the one fundamental truth. Now someone can come and try and take it back from me by force (Person, Court System, Rebels, Corporations) if they can exert more violence on 'me' than i can exert on 'Them'.
The problem with crypto is the 'keys' are what crypto is. No nationstate can come and take that away from me. They can kill/imprison/fine me, but then neither of us will have it. You would have to hack/fork the chain for that to happen (Which has happened) or find some social way around it (If i have it on a centralized exchange, if i have a hackable hard drive, found my keys on AWS, etc etc.) Additionally, with things like Monero, and tornado swap good luck trying to find them.
Deleted Comment
thanks for the loud laugh
This is the funniest thing I've read all day
If we have a situation where:
* Its hard to tell, after the fact, 'a mistake' was a bad actor.
* The programmers are, by and large, anonymous.
* The benefit of making 'a mistake' could be hundreds of millions of dollars that are not easily traced.
This situation seems rife for abuse and bad actors. Not saying it happened in this case. . . but how would you know?
> In Solidity, the order of evaluation of sub-expressions is unspecified. This means that in f(g(), h()), g() might get evaluated before h() or h() might get evaluated before g(). Practically, this order is predictable, but Solidity code shouldn’t depend on that behavior between compiler versions. In most circumstances g() is evaluated before h() (left-to-right order), which is also the behavior that most languages specify in their standards. However, in the case of emitting an event with indexed arguments, the arguments are evaluated right-to-left.
I feel that order-of-evaluation dependence is a special case of the general conflict between expression-oriented (functional-style) programming, and impure operations requiring sequential reasoning. Another case of this conflict is temporary values (expressions) with side-effectful destructors (sequential reasoning), for example https://fasterthanli.me/articles/a-rust-match-made-in-hell#w....
At this point, is it good practice to avoid using side-effectful procedure calls as parameters to other expressions (especially those with multiple inputs), but instead first assign to a temporary value to make order of operations explicit?
Thank you man!
If they keep it in blockchains only, it's hard to connect to a real identity. But if they cross the line (which is everybody's goal eventually) to the real world, they can get caught as easy or even easier than in traditional financial system.
I don't how anyone would commit anything more than pocket change to a scheme where an insider could deliberately introduce a weakness and then exploit that weakness to walk off with all the funds committed.
That said, I personally doubt this happens much if at all, because if you want to scam on web3 you can just do a good old-fashioned pump&dump and nobody seems to be receiving any legal/criminal consequences as of yet.
Challenge: explain to a normie that their life savings is gone forever because of a zero initialization vector.
Somebody who directly invested in MBS is by definition not a normie.
Of course there is no justice in either case, but at least normal people can see who is most appropriate to behead in the case of the traditional financial catastrophes, in the purely theoretical revolution.
If you got burned by mortgage backed derivatives and lost your life savings, it's ultimately because you were (knowingly or not) speculating on the value of real estate assets and making an assumption about future values of said assets.
In the case of Nomad, it's that you put yourself at risk by using their service you could've lost everything you put in.
You mistook a currency for an investment opportunity, and gambled your life savings on one thing. Currencies have always and will always fluctuate against each other. Diversify your investments.
Usually just depositing money in a bank doesn't get it stolen so the assumption isn't unreasonable.
Even if it's only 0.25*life saving that's still devastating for most people.
This is especially the case for protocols like Nomad that don’t yet have a native token. They’ll get liquidity commitments through over-the-counter SAFT agreements that give the VCs a percent of the future tokens.
The reason developers make their contracts "upgradable" is simple greed- they want to be able to launch more quickly than other projects without needing to ensure their code will stand the test of time. This may be OK for a social networking app MVP, but it's not OK for a smart contract which a user ideally should be able to audit and understand (or at least rely on the audit of someone else). "Upgradable" smart contracts can always be changed after the fact, as happened here, which means that any audit is meaningless.
Top tier projects still do use simple un-upgradable smart contracts. Uniswap first wrote v1, then improved it and launched v2, then v3. The Uniswap v1 and v2 contracts are still running and usable, and will be for as long as Ethereum is around. Their security properties will always be the same as they were the day they launched.
"Upgradable" contracts mean that you are trusting your money to some anonymous fat fingered (or at worst, criminal) dev, and it could disappear at any minute. They defeat the entire purpose of even using a blockchain.
It’s not really about greed. Deploying a program and having it unchangeable forever comes with risks, and more often when dealing with very complex applications, those aren’t worth it
Yes, writing perfect code is very hard. But smart contracts are an example of code that must be extremely thoroughly tested, formally verified and so on.
But that doesn't go well with being first to market, move fast break things, etc.
* Initialization was done 42 days ago: https://etherscan.io/tx/0x53fd92771d2084a9bf39a6477015ef53b7... -- "Click to see More" and notice "Input Data" parameter [2] which sets _committedRoot to 0x00.
* Click through the To contract to get to the code (click on Contract tab): https://etherscan.io/address/0xb92336759618f55bd0f8313bd8436...
Just adding direct links to what samczsun and 0xfoobar are talking about in https://twitter.com/samczsun/status/1554260106107179010 and https://twitter.com/0xfoobar/status/1554269071214088193/phot...
Depending on how you measure, that value is (in the US and Europe) typically in the order of 1..5 Mio USD.
So it's not outrageous to assume that losing 150m comes with a body count, even if the funds wouldn't have bee used to directly save or improve lives otherwise.
and so therefore nobody is trying to kill or impair the developers/team
if you were referring to people committing suicide or being suicided by the people they borrowed money from, thats not everyone’s problem and people in those circumstances should re-evaluate to avoid that risk or accept that risk
1. Owning ETH with a non-custodial wallet.
2. Owning ETH on a CEX.
3. Depositing ETH into a smart contract to receive a wrapped asset. This includes rollups and L2s.
The majority of major crypto hacks[1] are in the 3rd group, and almost all of these hacks are related to protocol updates and governance. Either: the developers update their code, and accidentally push a bug, or one address or a group of addresses are allow-listed some privileged actions in the contract and that can become a weak point.
Proxying and governance isn't the only way to design contracts. Two examples counter to this that are more robust are WETH ($6B) [2] and ETH2 Deposit ($20B) [3] which cannot be attacked in this way. If users wanted a new feature from the WETH contract, they would have to manually migrate over to the new address. Eventually we might see this kind of design be applied to bridges and rollups.
[1] https://rekt.news/leaderboard/
[2] https://etherscan.io/address/0xc02aaa39b223fe8d0a0e5c4f27ead...
[3] https://etherscan.io/address/0x00000000219ab540356cbb839cbe0...