I once deleted like, maybe a few petabytes of data valuable enough to store on FusionIO cards with a bad config change. I think it destroyed $50-100MM once the dust settled in all the various ways that it cost money were added up.
I’m quite sure the hacker who pressed the button feels plenty bad enough already, and I hope the people around them were as kind as the people around me were.
In the comment benreesman only said that enough data was deleted to cause 50-100M$ in damages. They might very well have managed to save half the data, but the lost half was already quite expensive?
If MM were million-million, that would mean OP destroyed $50-100 trillion dollars. About the GDP of the entire world. So it probably doesn't mean million-million ;)
No, it was features for a P(click|impression) model, which are surprisingly mundane: it is basically impossible to care much either way, let alone be outraged, if you look at the actual features used in these kinds of things and it’s not explicitly your field. It’s not a microphone listening on your smartphone!
I've watched crypto for years with interest, I've worked in the field, speculated in it with my savings, used it for international transfers, spent it, etc. That is to say I'm not some armchair critic.
I find myself lately viewing crypto as mostly a bad idea, or a bad implementation of an idea with potential, or even outright as a Ponzi scheme.
These smart contacts are really a bad idea. We have contracts in life with lawyers, these tend to work out pretty well because we can use our human intelligence to arbitrate them. The idea of code as law is really dumb. I say that as a programmer who works on code all day. Computers are dumb like your calculator. They do exactly what you tell them to. Which sounds great until you realize how hard it is to describe what you want in a way the computer can understand. That's why I make a good salary and people think my job is difficult. The reality is computers often do what you tell them, but it's not what you wanted. Rather you have the instructions wrong or you made a mistake. That's why software is riddled with bugs. Avoiding that is very, very, very difficult. Nobody gets it right. These crypto finance bros will definately not get it right. And when they get it wrong, there's no human, no court to appeal to. How's that an improvement?
For all its flaws (and there are countless) the one thing about the show cryptocurrency space that stands out to me as a programmer is that programming errors can be suddenly very costly (granted, in this case it was more of a DevOps blunder).
Being able to very easily put a price tag on sloppy programming is intriguing to me.
Certain fields have always had a high cost of programming error, including a cost in human life. It's just that cryptocurrency combines this with a first-to-market rush that's somehow still going on, encouraging a rush to error.
I don't disagree with that there have always been niche areas where there is a high cost to programming errors (like space or medical).
But with crypto, the effect is much more direct. The programmer is handling money much more directly and if something goes wrong, they are much more directly affected and not being insulated from the effects.
First to market is only part of the problem. Another self inflicted problem is that execution of those "smart" "contracts" cost tokens (so essentially money). This incentivizes developers to write the shortest possible code, without any "fluff" like tests or additional checks or more verbose style. And the immutability problem also increases severity of the problem - you either deploy immutable "contract" signifying that it should be respected because it can't be changed later for malicious purpose, or you deploy modifiable "contract" and that is not good for the reputation of the company.
When the price tag is “everything you own”, how is this an improvement? Who wants to give over their financial life to a computer program?
This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
> When the price tag is “everything you own”, how is this an improvement? Who wants to give over their financial life to a computer program?
You can strike out "financial" and that's already the realty we're living in. We got lucky that Y2K was not an issue today. We wouldn't probably be able to fix enough code and nowadays much more is under direct computer control than back then.
Nobody "wants" that but it happens gradually. With crypto, it didn't happen organically but crypto blasted onto the scene from the side of complete digitization. Looking at that, it's easy to say that nobody would want that but while you're looking this way, traditional finance is creeping towards complete digitization as well, just behind your back.
> This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
I'm not saying that crypto is an improvement as it is now. Nor that it will ever be (it might but I have my doubts). But what you're missing is that crypto is complete wild west, like traditional finance was maybe in 1900. The whole history of finance is a sequence of fuckups and laws and regulations that were imposed to prevent similar fuckups to occur and we still got the 2007 financial crisis, after 150 years of improvements.
I can try to stay as far away from crypto as possible but as I said in my previous post, there are aspects that are interesting even if most of it is completely nuts.
The level of self-perceived vs. actual competence in the crypto space never ceases to amaze.
A mea culpa of “The one thing we purport to be good at we actually have literally no understanding of and when shit doesn’t work we just run it a few times with different arguments.” My god.
When I first found about cryptocurrencies it struck me as it was the first time in the history where data on my computer were directly worth something. Which suddenly made security worth something.
Commands like that really need a confirmation prompt and a command line switch to override like --do-as-i-say (long form only). Good example of developers being put in an end user's shoes, I hope they learn from that mistake and update their programs.
An example of such a command came up on Linus Tech Tips (LTT), from apt (IIRC) on Pop_OS!. Basically (paraphrasing) "this will destroy your system and render it useless; type 'I'm an idiot and wish to proceed'". Our eponymous youtuber went right on, typed the phrase and blitzed the system, and then moaned about how the system shouldn't have let him break it ... I suspect such prompts only work for cautious people, who possibly don't need the prompts as they'll actually read the scary warnings.
In his defense, it was literally the first package he tried to install, which was in the official repositories but horribly broken, so he had no real way to tell whether this was the normal flow and linux just made it really irritating to install packages.
That guy is a savvy YouTuber, man. He did that because the resulting content has high memetic replication properties.
I find the prompt for dangerous ops useful. GitHub will ask you to repeat the name of the repo before you destructive actions, Terraform will ask you to say yes to the prompt. These are all good things.
That's a technical solution to a psychological and organizational problem: when there should be no CLI in production, people still use it and get used to various errors and confirmations so much, that they ignore the signs of a catastrophe. Technical solutions never work in such cases.
And then there's a regulatory problem: investors trust their money to businesses which have not earned that trust, because of whatever magical thinking that exists on this market. At least the company seems to be able to return the money, but will it be sanctioned for this failure? There should be a regulatory incentive to do better next time.
I don't know about you folks but I can do stupid stuff also with confirmation. Maybe splitting the damaging actions in some substeps, so instead of a big bang you have a slow death?
You are right, something like closing a program shouldn't be valid immediately, the program should be frozen for a day with the ability to cancel the closing at any time until the day is over.
Also, for me when I make big mistakes like this I think I’m in a different environment or something. It’s not that I don’t realise what command I’m running, it’s that I miss the context.
So I would just confirm but not realise what I was confirming.
Command line switches like that won't, IMO and IME, make developers think "wow, I'm so glad this exists to guard against footguns". Rather they'll just set an alias to get around it. Likewise confirmation prompts albeit with a little more resitance. Once a repeat user is frustrated enough they'll bust out `expect` or similar, angry at the original command for mollycoddling them because they know what they're doing and don't make daft mistakes, damn it.
If you alias a —do-as-i-say switch to a command without wrapping into a script to make sanity checks of your own, IMO that’s negligence. The name of the flag should make you think thrice about invoking it without safety checks.
I have not understood most of that article (optifi, mainnet, solana, PDAs, deployer address, program, tvl, ...what?) but what I get from it is that people who run crypto-anything make costly mistakes.
Funny that people still ask me "hey, why do you trust those evil banks? you should do all in crypto instead!" and then laugh at me when I tell them I trust banks more than some random dev on the internet...
This is one of the biggest flaws in crypto. Small errors can erode hundreds of millions of value.
That's when most crypto companies right now are very small. Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
There's definite efficiency gains with crypto (a dex like Uniswap can do massive volume with very few developers for instance), but there needs to be a way to limit the cost of human errors.
Sure - but a lot of systems have a safety net of test environments and change control, and a safety net as they can revert changes, and a safety net as they can restore backups, and a safety net as they can ask counterparties nicely to help undo things, and a safety net through the legal system, and a safety net of insurance.
The normal banking industry is operating chainsaws very cautiously, with a lot of safety equipment and training. The cryptocurrency industry may be operating the same chainsaws, but they're trying to juggle them naked, on a floor slick with the blood of their peers.
This is a deeply stupid analogy, because as everyone is pointing out .. why would you do that? Why would you choose to build a system in which this is a risk, when it's not actually imposed by the laws of nature?
But we moved away from moving currency in sinkable boats hundreds of years ago. Error in plain old finance today can often be fixed. Do we need to regress back to antiquity?
> Yeah I can already imagine in other places, just ask Santander Bank. [0]
Per the article, they were very likely to recover most of that money (and were legally entitled to recover all of it, but were unwilling to do so for image reasons). So exactly the opposite of what happens with DeFi.
It's not exactly unique to crypto... in adtech, a configuration mistake on a web server can cost 6-7 figures as ads are being paid for, but take people nowhere.
Sure, but the company managing the ad network can choose to be gentle, and refund the money. With these sorts of crypto mistakes, the money is just gone, because humans are -- as a desired feature of the system -- unable to correct these sorts of mistakes.
Lessoned we learned harshly
EVERY DEPLOYMENT NEEDS A RIGOROUS PROCESS AND SINGLE POINT FAILURE CAN BE AVOIDED. PLEASE DON’T RUSH LIKE WHAT WE DID, ESPECIALLY FOR DEFI PROJECTS.
Best wishes to the company and individuals involved for recovery of assets and success going forward.
The money within the contract wasn't the problem, that was transferred out as part of closing the program.
However there were other assets on the chain that are locked using the program which are now inaccessible because its inoperable.
Consider it like an operation that closes and deletes your user-account but returns you an archive of your home directory when it's complete. However all files that weren't in your home directory that were encrypted with your local user key now can't be read.
It's likely infeasible to predicate program deletion on the program no longer being referenced in other programs/contracts.
Disclaimer: I'm not a Solana expert (or a crypto advocate for that matter)
Readit News
I’m quite sure the hacker who pressed the button feels plenty bad enough already, and I hope the people around them were as kind as the people around me were.
In the comment benreesman only said that enough data was deleted to cause 50-100M$ in damages. They might very well have managed to save half the data, but the lost half was already quite expensive?
Edit: thanks for replies! :)
See "How big is a billion?" from Numberphile:
* https://www.youtube.com/watch?v=C-52AI_ojyQ
* https://en.wikipedia.org/wiki/Long_and_short_scales
You can even read about it in this paper which publicly described the system contemporary to that event: http://www.quinonero.net/Publications/predicting-clicks-face...
If you read it and are still steamed up, well, then you hold that opinion honestly, but I suspect most would be like, “cool story bro”.
I find myself lately viewing crypto as mostly a bad idea, or a bad implementation of an idea with potential, or even outright as a Ponzi scheme.
These smart contacts are really a bad idea. We have contracts in life with lawyers, these tend to work out pretty well because we can use our human intelligence to arbitrate them. The idea of code as law is really dumb. I say that as a programmer who works on code all day. Computers are dumb like your calculator. They do exactly what you tell them to. Which sounds great until you realize how hard it is to describe what you want in a way the computer can understand. That's why I make a good salary and people think my job is difficult. The reality is computers often do what you tell them, but it's not what you wanted. Rather you have the instructions wrong or you made a mistake. That's why software is riddled with bugs. Avoiding that is very, very, very difficult. Nobody gets it right. These crypto finance bros will definately not get it right. And when they get it wrong, there's no human, no court to appeal to. How's that an improvement?
Being able to very easily put a price tag on sloppy programming is intriguing to me.
Certain fields have always had a high cost of programming error, including a cost in human life. It's just that cryptocurrency combines this with a first-to-market rush that's somehow still going on, encouraging a rush to error.
But with crypto, the effect is much more direct. The programmer is handling money much more directly and if something goes wrong, they are much more directly affected and not being insulated from the effects.
This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
Where do you think your income and bank balances are tracked and stored? On pieces of paper?
You can strike out "financial" and that's already the realty we're living in. We got lucky that Y2K was not an issue today. We wouldn't probably be able to fix enough code and nowadays much more is under direct computer control than back then.
Nobody "wants" that but it happens gradually. With crypto, it didn't happen organically but crypto blasted onto the scene from the side of complete digitization. Looking at that, it's easy to say that nobody would want that but while you're looking this way, traditional finance is creeping towards complete digitization as well, just behind your back.
> This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
I'm not saying that crypto is an improvement as it is now. Nor that it will ever be (it might but I have my doubts). But what you're missing is that crypto is complete wild west, like traditional finance was maybe in 1900. The whole history of finance is a sequence of fuckups and laws and regulations that were imposed to prevent similar fuckups to occur and we still got the 2007 financial crisis, after 150 years of improvements.
I can try to stay as far away from crypto as possible but as I said in my previous post, there are aspects that are interesting even if most of it is completely nuts.
A mea culpa of “The one thing we purport to be good at we actually have literally no understanding of and when shit doesn’t work we just run it a few times with different arguments.” My god.
You're welcome.
I find the prompt for dangerous ops useful. GitHub will ask you to repeat the name of the repo before you destructive actions, Terraform will ask you to say yes to the prompt. These are all good things.
And then there's a regulatory problem: investors trust their money to businesses which have not earned that trust, because of whatever magical thinking that exists on this market. At least the company seems to be able to return the money, but will it be sanctioned for this failure? There should be a regulatory incentive to do better next time.
It seems like this was in the "deploy to production" stage, and there has to be some mechanism for doing that.
(It also seems unnecessarily complicated; I'd appreciate a plaintext explainer of what actually went wrong)
So I would just confirm but not realise what I was confirming.
Funny that people still ask me "hey, why do you trust those evil banks? you should do all in crypto instead!" and then laugh at me when I tell them I trust banks more than some random dev on the internet...
That's when most crypto companies right now are very small. Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
There's definite efficiency gains with crypto (a dex like Uniswap can do massive volume with very few developers for instance), but there needs to be a way to limit the cost of human errors.
That's reality.
Same thing would have happened if they had put all that cash on a boat, and accidentally sunk it.
The normal banking industry is operating chainsaws very cautiously, with a lot of safety equipment and training. The cryptocurrency industry may be operating the same chainsaws, but they're trying to juggle them naked, on a floor slick with the blood of their peers.
Are you saying that it is impossible to get valuables out of a boat that sunk?
"Treasure trove of gold and jewels recovered from a 366-year-old shipwreck in the Bahamas"
https://www.livescience.com/bahamas-shipwreck-gold-jewels
But there were a few instances in WW2 where large amounts of value had to be transferred by boat, and in at least one case sunk by enemy action. HMS Edinburgh: https://www.warhistoryonline.com/instant-articles/hms-edinbu...
This is hardly exclusive to crypto.
> Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
Yeah I can already imagine in other places, just ask Santander Bank. [0]
[0] https://www.theguardian.com/business/2021/dec/30/santander-b...
Per the article, they were very likely to recover most of that money (and were legally entitled to recover all of it, but were unwilling to do so for image reasons). So exactly the opposite of what happens with DeFi.
see: Knight Capital
Dead Comment
Lessoned we learned harshly EVERY DEPLOYMENT NEEDS A RIGOROUS PROCESS AND SINGLE POINT FAILURE CAN BE AVOIDED. PLEASE DON’T RUSH LIKE WHAT WE DID, ESPECIALLY FOR DEFI PROJECTS.
Best wishes to the company and individuals involved for recovery of assets and success going forward.
However there were other assets on the chain that are locked using the program which are now inaccessible because its inoperable.
Consider it like an operation that closes and deletes your user-account but returns you an archive of your home directory when it's complete. However all files that weren't in your home directory that were encrypted with your local user key now can't be read.
It's likely infeasible to predicate program deletion on the program no longer being referenced in other programs/contracts.
Disclaimer: I'm not a Solana expert (or a crypto advocate for that matter)