Not very interesting. There are ways to solve all of these concerns--no fees are paid during the first X months, nodes periodically tested for bandwidth, etc. etc. Just because the author didn't bother to think it through doesn't mean it's not straightforward.
The whole idea of a blockchain is reputation CAN be built up over time, because every transaction is recorded and there might even be incentives to associate nodes to show X nines of pool reliability.
Is it a bad idea to use this as your only source of storage? Of course. Is it less useful than AWS if you do large local burst queries? Of course.
But none of the concerns mentioned in the post are relevant. This can be cheaper than AWS because they charge an enormous markup, and this automatically provides worldwide replication and accessibility.
Amazon already erases books from Kindles, Google scans your Docs for blasphemy against Fauci, and both have been known to throw businesses off their services. In a few years it will seem crazy not to have an extra copy of your data on here.
What do you base AWS storage having "an enormous markup" on? AWS has a type of storage to meet any access need / budget. S3 Glacier Deep Dive for example is perfect for storing archival items and costs $0.00099 per GB-month. So you can store a TB for about $1 per month.
There are even options like S3 Intelligent Tiering that enable you to store objects on S3 Standard and it will intelligently move them to lower-cost options based on the usage patterns of the data.
There are six different pricing options of S3 storage. There are EBS volumes, EFS volumes, FSx for Lustre and many other options for storage.
I used to mine BTC, currently mine Ethereum and looked into mining Filecoin and was left wondering if the people buying into it have any idea what cloud storage options are actually out there. There is so much competition and the prices are sooooo low that I don't see any viable use case for Filecoin outside of the one you mentioned about being concerned about being shut down by a single vendor.
It’s like the people who seriously touted Bitcoin as a cheaper payment mechanism in than credit cards 2013. Few of them explored how ~3% fees are a balance between customer rewards, regulatory priorities, and merchant convenience. In places like Australia regulators made other choices, largely eliminating rewards programs and capping interchange fees at around a tenth of what they are in the US.
There are good arguments for decentralization and censorship resistance, but from a first principles perspective scale and market pressures are good bets for delivering cost efficiencies.
>Amazon already erases books from Kindles, Google scans your Docs for blasphemy against Fauci, and both have been known to throw businesses off their services. In a few years it will seem crazy not to have an extra copy of your data on here.
Isn't this a non-issue because you can encrypt your data prior to uploading?
For backup scenarios, you can encrypt your data before sending it to Google/Amazon. Filecoin even accommodates (or at least plans to, from what I've read) for nodes that want to store only certain types of content based on legality or whatever.
I agree that all the concerns can be mitigated. But they're not just theoretical, they're also issues with the current implementations—some of which remain a year and a half after this was written.
One important property of decentralized storage is that it prevents data monopolies. For example, if you host your web app on Sia Skynet, then users own their data instead of being locked in by the app developer. It also enables a model where different apps can easily access the same data. The data is in the "cloud", but still owned by the user.
Do you know why Sia is overlooked so often? I see a lot of shills on forums and the founder seems to tweet bitterly about Filecoin, but folks seem optimistic about the tech. So what am I missing that makes Sia not get adoption vs. Filecoin?
The Sia team has let down customers for other tangential projects, and Sia has weaker guarantees than Filecoin. It’s been out a long time now and they have failed to address some core scaling and redundancy concerns. I wouldn’t say it’s bad, but it’s seeming more like maidsafe - never got enough traction and now the interest is on filecoin and arweave.
Likewise with ScPrime (https://scpri.me) which is like Sia but focused on B2B data storage and is changing to PoS instead of PoW to be more environmentally friendly.
I think that this paper is more about file storage, not web apps.
And for file storage, there are no data monopolies already. Take for example AWS S3 storage -- there are multiple providers (AWS, Backblaze or local Minio) and many, many clients in all sorts of languages.
You could implement the same Solid-style model using centralized storage or you could implement lock-in on top of decentralized storage. It's mostly orthogonal.
I've noticed the debate around (de)centralization always seems to focus on purely technical issues, or on economic issues (which are technicalities of another sort). In other words, it's always the following two points that are debated:
It seems to me that a crucial element is missing from these analyses: organizational policy. All other things being equal (which they may not be), it seems like decentralization appeals to those trying to restrict top-down control. Historically, this has been an ideological thing (e.g. bittorrent's copyleft slant), but more recently I've seen mainstream endorsement of concepts such as "flat hierarchy", "distributed workforce", "collaborative work", etc. Interestingly, I don't see this discourse being picked up by decentralization evangelists, and I'm not sure what to make of that.
Off the cuff, it seems like decentralized storage tech has a compelling story to tell about small distributed teams, working in a loose, peer-to-peer organizational structure. In this context, my mind immediately goes to the functional centralization of cloud storage. In general, the Cloud tries to concentrate control over computing resources in a single department (i.e. the devops team). But what happens when a bunch of freelancers collaborate on a project-by-project basis? From what I've seen, either (1) they use cloud providers, but the account is under the client's name or (2) they use some kind of SaaS solution with support for sharing or other forms of collaboration. With decentralized storage, it seems like a "bring your own cloud" approach is possible in principle, where these freelancers would pool resources (storage, compute, etc.). Are there any pain-points here? Is there perhaps a business problem that needs solving?
The question for me is whether these new patterns of collaboration are really becoming a thing, or whether it's a fad amongst business-school graduates.
At any rate, this comment is an invitation for y'all to opine on the subject. I feel like a good analysis of organizational policy could help us decide whether a decentralized web is economically viable, but I'm kinda stumped.
a classic essay discussing the real ground truth problems of decentralized anything is by Jo Freeman, called the tyranny of structurelessness [0].
though most of the punches land toward the end, it is a fascinating and short read, in which she dissects, with the sober disappointment of a former optimistic evangelist, exactly why structureless ("decentralized" before that term became vogue) movements are never what they claim to be, and indeed cannot be.
why? because structureless coordination of groups above a handful members doesn't scale beyond the achievement of the simplest kinds of goals, such as consciousness-raising (or, in crypto-promoter terms, spreading "adoption").
any group objective requiring specialization of labor, coordination of resources and responsibility inevitably succumbs to the insidious creep of informal, but officially disavaowed structural networks of insiders.
it is especially pernicious for newcomers, who have been heavily messaged about "decentralization" or "structurelessness" and engage themselves with the groups identity without any awareness of these cryptic back channel networks that actually govern its operation. only after they are committed do they slowly find out there really is a hierarchy and a command structure.
i highly recommend a quick read for anyone considering the merits of a given groups' claims of being decentralized.
Structurelessness and decentralization are only related in that structureless things are decentralized. Decentralized things are rarely structureless. Whether a networking protocol is decentralized or not, it's definitely not structureless; it's nothing but structure, it's a protocol.
I'm familiar with this essay, and agree with most of its conclusions.
However, I am not convinced it is exactly applicable to the situation I have in mind. Freeman's point pertains to extreme decentralization of organizational policy, i.e. the flat hierarchy. These are arguably not even organizations; the essence of organization is precisely hierarchy!
I'm more interested in organizations that do have some degree of centralization. My hypothetical freelancers are certainly not disorganized. What I'm trying to grasp is the conditions under which any group -- especially one with centralized policymaking -- may seek decentralized infrastructure.
In other words: when does decentralized infrastructure serve the goals of a well-organized group?
I think I could get excited about FileCoin if it allowed me to pool storage with friends and colleagues. I've been wanting to have a distributed hacker-garage for a while [0].
Importantly, I want it to be independent of any cloud provider because (1) it's hard to freely experiment/prototype when you have to keep track of billing and (2) functional centralization makes the AWS/GCE admin de facto responsible for everything running under the account. This is great for large enterprises that want to virtualize their IT department, but it really goes against the grain for hackers, freelancers, and -- IMHO -- startup founders.
Returning to Filecoin, I feel like all the *coin ventures make the same fundamental mistake of over-estimating how much we care about money. I really don't care about pimping out my hard-drive for a couple of cents. I'm motivated by gains on the order of several hundred to a couple of thousand dollars. The counter-argument I usually hear is that there will be more interest in developing nations, but then I don't want to store my data in e.g. Pakistan. I want data for pet projects in my own hacker-garage, and I want production data close to the end-user.
/rant
[0] Incidentally, I'm working on a project to do exactly this. It's a way of clustering a set of computers over the internet to produce a virtual cloud. It would be super cool to use FileCoin as a block storage layer in this context, but restricted to the hardware we own.
This speaks to the disingenuity of a lot of the decentralization talk in the crypto space. The Beaker browser and the hyperdrive protocol (formerly DAT) is an interesting case study. It has a level of maturity and delivers on many of the ideas that helped filecoin and other token-financed projects raise hundreds of millions of dollars, but it is relatively obscure and I don’t think I know anyone who has used it other than the people I’ve encouraged to check it out. If there were actual demand for dweb technology then it and other projects like it should be getting a lot more attention since it’s a free and relatively user friendly platform compared to services that require managing token exchange risk, accounting, and hassle.
I've been keeping my eyes on https://cobox.cloud/ - it's run by some very smart people who have had their brains in the decentralization (not blockchain) space for a while now.
The alpha already lets you do what you're talking about, but I haven't used it yet and so I can't speak to its current stability.
As someone who just set up Storj to backup my Nextcloud installation, I am very interested in this topic. Storj was a cheap alternative to AWS S3, and, at least out of the box, zero-knowledge encryption was easier, though I know there are ways to achieve zero knowledge encryption with any S3 provider. I hadn't considered Glacier, but I may look into it.
Edit: One thing I liked about Storj was that geographic redundancy is built in. I know Amazon has that too, but if three data centers caught fire on the same day, Amazon could lose your data. Super unlikely with Amazon, but virtually impossible with Storj, and most S3 providers have less geographic redundancy than Amazon.
> knowing your data is there does not mean that you can get it back when needed. This is the subject of the next blog post.
Presumably the argument is that a storage host might dutifully provide proofs of storage, but refuse to actually transfer your data to you later. This is true, but it's also true of centralized storage providers. The difference is that traditional providers are typically bound by SLAs -- something that a blockchain can't really offer.
Instead, decentralized storage gives you the ability to store your data with dozens of independent entities; as long as the behavior of these entities is sufficiently uncorrelated, then it is highly likely that you will always be able to retrieve your data from some subset of them.
or one entity operating dozens of nodes? droplets are cheap, pwned windows boxes even cheaper. they don't even have to be cheap by regular Joe standards, they could be cheap by nation state, or ransomware gang standards. the point is there's no way to prove they are decentralized. this is an intrinsic, structural property of the cryptocurrency model, thus it is not a mere technical defect that will get solved "in the future"
proving decentralization would require zero privacy, because you'd have to prove independent nodes are indeed independent and not part of a cartel, which means you'd have to somehow prove who operates each one.
nothing but faith-based decentralization here, as with all cryptocurrencies.
The argument (of the second post) is that their behavior will be correlated because they all have (more or less) the same incentives and cost structure.
> SLAs -- something that a blockchain can't really offer.
Hold on there, SLAs are actually something that blockchains are great at. Smart contracts provide cryptographic guarantees of service. In several of these types of systems, node runners must put up collateral as "stake" in exchange for collecting fees for their services. If a participant in the network identifies a misbehaving node (violation of SLA) then their stake can get slashed, or burned completely.
If a Filecoin node claims to be holding data for someone, but is unable to provide it in a timely manner when legitimately requested to do so, they're in for some serious penalties.
The great thing is that the consumer doesn't need to worry about some big company hiring expensive lawyers to weasel out of their contractual obligations. The rules are all readily transparent and enshrined as code that anyone can read and verify the validity of before participating.
In the case of Storj, I believe that the redundancy between storage nodes is sufficient that something like only 20 out of 80 nodes need to respond in order to restore your data.
S3 has no durability SLA, only an availability SLA. S3 also explicitly says it is not designed for 100% durability, but for a certian number of nines (99.999999999%).
Sure it can provide an SLA. The code is the SLA. And is available for all to examine. Via opensource. And governance allows the community to change terms/accept code... moving forward.
The next blog post it's already there, have a look. It's basically about the incentives that could make in certain scenarios more profitable to never give back data to the one who purchased the storage.
I looked into glacier pricing a while back it's..tricky. It looks super cheap up front, but gets extremely expensive if you ever need to get the data out of glacier.
Depending on how long ago you checked it out (relative to the ~9 years since Glacier was first announced) it might be worth another look.
It's still quite expensive if you ever need to retrieve data in a hurry, but it used to also have a complicated and unintuitive pricing model that made it extremely easy to accidentally spend orders of magnitude more money than you intended to. They redesigned the pricing structure in 2016 to make it a lot more straightforward.
Extremely expensive in terms of storage, but still much cheaper than S3.
I consider it as a mitigation against disaster and expect to never retrieve other than for testing the process. So it seems good for extremely low probability, high impact events. And I trust amazon to meet their SLA.
Why is Sia overlooked every single time this topic is brought up? It is shocking to me that Filecoin, an example of an ICO darling with no working product, is always at the top of the list.
I have used Sia and Skynet to build several useful decentralized media sharing applications with zero loss of data for over a year.
Sia currently costs 8$ TB/month, storj costs a fixed forever* 4$ TB/month.
* not counting usd inflation, because they don't lock storage price to coin value (with crazy volatility), they fix it against usd and covert it to their coin when paying their nodes.
Blockchain is not used for distributed storage itself. Existing concepts like IPFS are used for this. Blockchain is used to confirm that someone is storing certain data and to verify it's integrity through an additional protocol.
From ipfs.io about Blockchain
>With IPFS, you can address large amounts of data and put immutable, permanent links in transactions — timestamping and securing content without having to put the data itself on-chain.
You are corrent; I think the article is just clumsily worded, though. I think what he's saying is that IPFS gives you references to immutable data, which you are then free to commit to your favorite blockchain.
Interesting article. While reading about Filecoin the first time, I had to stop every paragraph and think "but what if the node just doesn't give you your data back when you finally ask for it?". And as pointed out by part 2 of the article, not only can they choose not to give it back, but they are incentivized not to.
edit: Wouldn't erasure codes just completely fix it (as described by the Storj guy)?
Readit News
The whole idea of a blockchain is reputation CAN be built up over time, because every transaction is recorded and there might even be incentives to associate nodes to show X nines of pool reliability.
Is it a bad idea to use this as your only source of storage? Of course. Is it less useful than AWS if you do large local burst queries? Of course.
But none of the concerns mentioned in the post are relevant. This can be cheaper than AWS because they charge an enormous markup, and this automatically provides worldwide replication and accessibility.
Amazon already erases books from Kindles, Google scans your Docs for blasphemy against Fauci, and both have been known to throw businesses off their services. In a few years it will seem crazy not to have an extra copy of your data on here.
There are even options like S3 Intelligent Tiering that enable you to store objects on S3 Standard and it will intelligently move them to lower-cost options based on the usage patterns of the data.
There are six different pricing options of S3 storage. There are EBS volumes, EFS volumes, FSx for Lustre and many other options for storage.
I used to mine BTC, currently mine Ethereum and looked into mining Filecoin and was left wondering if the people buying into it have any idea what cloud storage options are actually out there. There is so much competition and the prices are sooooo low that I don't see any viable use case for Filecoin outside of the one you mentioned about being concerned about being shut down by a single vendor.
There are good arguments for decentralization and censorship resistance, but from a first principles perspective scale and market pressures are good bets for delivering cost efficiencies.
Probably on the basis that backblaze costs an order of magnitude less for the same product.
Ps everything on aws is marked up by like 300%, it's easy to beat them on cost.
Deleted Comment
Isn't this a non-issue because you can encrypt your data prior to uploading?
What is this referencing? I'm just curious, this is something I haven't heard about.
Discussion: https://news.ycombinator.com/item?id=23275308
As usual, it may not be what you think.
Deleted Comment
And for file storage, there are no data monopolies already. Take for example AWS S3 storage -- there are multiple providers (AWS, Backblaze or local Minio) and many, many clients in all sorts of languages.
1. Economies of Scale
2. Network Properties (robustness, scalability, latency, etc.)
It seems to me that a crucial element is missing from these analyses: organizational policy. All other things being equal (which they may not be), it seems like decentralization appeals to those trying to restrict top-down control. Historically, this has been an ideological thing (e.g. bittorrent's copyleft slant), but more recently I've seen mainstream endorsement of concepts such as "flat hierarchy", "distributed workforce", "collaborative work", etc. Interestingly, I don't see this discourse being picked up by decentralization evangelists, and I'm not sure what to make of that.
Off the cuff, it seems like decentralized storage tech has a compelling story to tell about small distributed teams, working in a loose, peer-to-peer organizational structure. In this context, my mind immediately goes to the functional centralization of cloud storage. In general, the Cloud tries to concentrate control over computing resources in a single department (i.e. the devops team). But what happens when a bunch of freelancers collaborate on a project-by-project basis? From what I've seen, either (1) they use cloud providers, but the account is under the client's name or (2) they use some kind of SaaS solution with support for sharing or other forms of collaboration. With decentralized storage, it seems like a "bring your own cloud" approach is possible in principle, where these freelancers would pool resources (storage, compute, etc.). Are there any pain-points here? Is there perhaps a business problem that needs solving?
The question for me is whether these new patterns of collaboration are really becoming a thing, or whether it's a fad amongst business-school graduates.
At any rate, this comment is an invitation for y'all to opine on the subject. I feel like a good analysis of organizational policy could help us decide whether a decentralized web is economically viable, but I'm kinda stumped.
though most of the punches land toward the end, it is a fascinating and short read, in which she dissects, with the sober disappointment of a former optimistic evangelist, exactly why structureless ("decentralized" before that term became vogue) movements are never what they claim to be, and indeed cannot be.
why? because structureless coordination of groups above a handful members doesn't scale beyond the achievement of the simplest kinds of goals, such as consciousness-raising (or, in crypto-promoter terms, spreading "adoption").
any group objective requiring specialization of labor, coordination of resources and responsibility inevitably succumbs to the insidious creep of informal, but officially disavaowed structural networks of insiders.
it is especially pernicious for newcomers, who have been heavily messaged about "decentralization" or "structurelessness" and engage themselves with the groups identity without any awareness of these cryptic back channel networks that actually govern its operation. only after they are committed do they slowly find out there really is a hierarchy and a command structure.
i highly recommend a quick read for anyone considering the merits of a given groups' claims of being decentralized.
[0] https://www.jofreeman.com/joreen/tyranny.htm
However, I am not convinced it is exactly applicable to the situation I have in mind. Freeman's point pertains to extreme decentralization of organizational policy, i.e. the flat hierarchy. These are arguably not even organizations; the essence of organization is precisely hierarchy!
I'm more interested in organizations that do have some degree of centralization. My hypothetical freelancers are certainly not disorganized. What I'm trying to grasp is the conditions under which any group -- especially one with centralized policymaking -- may seek decentralized infrastructure.
In other words: when does decentralized infrastructure serve the goals of a well-organized group?
I think I could get excited about FileCoin if it allowed me to pool storage with friends and colleagues. I've been wanting to have a distributed hacker-garage for a while [0].
Importantly, I want it to be independent of any cloud provider because (1) it's hard to freely experiment/prototype when you have to keep track of billing and (2) functional centralization makes the AWS/GCE admin de facto responsible for everything running under the account. This is great for large enterprises that want to virtualize their IT department, but it really goes against the grain for hackers, freelancers, and -- IMHO -- startup founders.
Returning to Filecoin, I feel like all the *coin ventures make the same fundamental mistake of over-estimating how much we care about money. I really don't care about pimping out my hard-drive for a couple of cents. I'm motivated by gains on the order of several hundred to a couple of thousand dollars. The counter-argument I usually hear is that there will be more interest in developing nations, but then I don't want to store my data in e.g. Pakistan. I want data for pet projects in my own hacker-garage, and I want production data close to the end-user.
/rant
[0] Incidentally, I'm working on a project to do exactly this. It's a way of clustering a set of computers over the internet to produce a virtual cloud. It would be super cool to use FileCoin as a block storage layer in this context, but restricted to the hardware we own.
https://beakerbrowser.com/
The alpha already lets you do what you're talking about, but I haven't used it yet and so I can't speak to its current stability.
Edit: One thing I liked about Storj was that geographic redundancy is built in. I know Amazon has that too, but if three data centers caught fire on the same day, Amazon could lose your data. Super unlikely with Amazon, but virtually impossible with Storj, and most S3 providers have less geographic redundancy than Amazon.
> knowing your data is there does not mean that you can get it back when needed. This is the subject of the next blog post.
This is something I'd love to hear more about.
Instead, decentralized storage gives you the ability to store your data with dozens of independent entities; as long as the behavior of these entities is sufficiently uncorrelated, then it is highly likely that you will always be able to retrieve your data from some subset of them.
or one entity operating dozens of nodes? droplets are cheap, pwned windows boxes even cheaper. they don't even have to be cheap by regular Joe standards, they could be cheap by nation state, or ransomware gang standards. the point is there's no way to prove they are decentralized. this is an intrinsic, structural property of the cryptocurrency model, thus it is not a mere technical defect that will get solved "in the future"
proving decentralization would require zero privacy, because you'd have to prove independent nodes are indeed independent and not part of a cartel, which means you'd have to somehow prove who operates each one.
nothing but faith-based decentralization here, as with all cryptocurrencies.
Hold on there, SLAs are actually something that blockchains are great at. Smart contracts provide cryptographic guarantees of service. In several of these types of systems, node runners must put up collateral as "stake" in exchange for collecting fees for their services. If a participant in the network identifies a misbehaving node (violation of SLA) then their stake can get slashed, or burned completely.
If a Filecoin node claims to be holding data for someone, but is unable to provide it in a timely manner when legitimately requested to do so, they're in for some serious penalties.
The great thing is that the consumer doesn't need to worry about some big company hiring expensive lawyers to weasel out of their contractual obligations. The rules are all readily transparent and enshrined as code that anyone can read and verify the validity of before participating.
It's still quite expensive if you ever need to retrieve data in a hurry, but it used to also have a complicated and unintuitive pricing model that made it extremely easy to accidentally spend orders of magnitude more money than you intended to. They redesigned the pricing structure in 2016 to make it a lot more straightforward.
I consider it as a mitigation against disaster and expect to never retrieve other than for testing the process. So it seems good for extremely low probability, high impact events. And I trust amazon to meet their SLA.
Deleted Comment
* not counting usd inflation, because they don't lock storage price to coin value (with crazy volatility), they fix it against usd and covert it to their coin when paying their nodes.
Blockchain is not used for distributed storage itself. Existing concepts like IPFS are used for this. Blockchain is used to confirm that someone is storing certain data and to verify it's integrity through an additional protocol.
From ipfs.io about Blockchain
>With IPFS, you can address large amounts of data and put immutable, permanent links in transactions — timestamping and securing content without having to put the data itself on-chain.
Please correct me if I am wrong.
edit: Wouldn't erasure codes just completely fix it (as described by the Storj guy)?