I have to imagine that by making Storage Pods 1.0 through 6.0, maybe they "encouraged" Dell (and other manufacturers) that this particular 60+ hard drive server was a good idea.
And now that multiple "storage pod-like" systems exist in the marketplace (not just Dell, but also Supermicro) selling 60-bay or 90-bay 3.5" Hard drive storage servers in 4U rack form factors, there's not much reason to build their own?
At least, that's my assumption. After all, if the server chassis is a commodity now (and it absolutely is), no point making custom small runs to make a hypothetical Storage Pod 7. Economies of scale is too big a benefit (worst case scenario: its now Dell's or Supermicro's problem rather than Backblaze's).
EDIT: I admit that I don't really work in IT, I'm just a programmer. So I don't really know how popular 4U / ~60 HDD servers were before Backblaze Storage Pod 1.0
3U and 4U x86 whitebox servers designed for any standard 12"x13" motherboard, where the entire front panel was hotswap 3.5" HDD bays were already a thing many, many years before backblaze existed.
What wasn't really a thing was servers with hotswap HDD trays on both ends (like the supermicros) and things that were designed with vertical hard drives dropped down from a top-opening lid to achieve even higher density.
That's not the backblaze design. The backblaze design is that the drives are individually hot-swappable without a tray. 60 commodity SATA drives that can be removed and serviced individually while a 4U server continues to operate normally is pretty amazing.
You'd think they could build a ARM-powered, credit card sized controller for them with a disk breakout card and network IO. PC motherboard and full-sized cards seem like overkill.
They're running a fair bit of math (probably Reed Solomon matrix multiplications for error correction) over all the data accesses.
Given the bandwidth of 60+ hard drives (150MB/s per hard drive x 60 == 9GB/s in/out), I'm pretty sure you need a decent CPU just to handle the PCIe traffic. At least PCIe 3.0 x16, just for the hard drives. And then another x16 for network connections (multiple PHY for Fiber in/out that can handle that 9GB/s to a variety of switches).
We're looking at PCIe 3.0 x32 just for HDDs and Networking. Throw down a NVMe-cache or other stuff and I'm not seeing any kind of small system working out here.
---------
Then the math comes in: matrix multiplications over every bit of data to verify checksums and reed-solomon error correction starts to get expensive. Maybe if you had an FPGA or some kind of specialist DSP (lol GPUs maybe, since they're good at matrix multiplication), you can handle the bandwidth. But it seems nontrivial to me.
Server CPU seems to be a cheap and simple answer: get the large number of PCIe I/O lanes plus a beefy CPU to handle the calculations. Maybe a cheap CPU with many I/O lanes going to a GPU / FPGA / ASIC for the error checking math, but... specialized chips cost money. I don't think a cheap low-power CPU would be powerful enough to perform real-time error correction calculations over 9GBps of data.
--------
We can leave Backblaze's workload and think about typical SAN or NAS workloads too. More I/O is needed if you add NVMe storage to cache hard drive reads/writes, tons of RAM is needed if you plan to dedup.
With 60+ hard drives in a single enclosure, you want all of the processing power, PCIe lanes, and bandwidth you can get.
There are a small number of high end ARM server boards that could do it, but you’re not saving much money at that point. Might be more expensive due to lack of scale.
Amortize a server-grade CPU and motherboard across 60+ high capacity drives and it’s not really worth pinching pennies at the risk of lower performance.
It’s actually interesting to me that backblaze has actually reached a size where global logistics plays a bigger part in costs than the actual servers. (And the servers got cheaper).
Also, Dell and Supermicro have storage servers inspired by the BB Pods.
Glad to see this scrappy company hit this amount of scale; a long way from schucking Hard Drives
Any idea what Dell is actually selling them? The DVR's we buy (Avigilon) are white Dell 7x0's with a custom white bezel, but those only fit 18 3.5" drives.
Dell's densest server is the PowerEdge XE7100 [1] (100 3.5" drives in 5U) but the bezel cover picture looks like more like a standard 2U, maybe a R740xd2 (26 3.5" in 2U).
Based on the pictured bezel, it looks like they've got three rows 3.5" 14TB SATA drives up front in 14th generation carriers. Best guess would be something like an R740XD2 which has 26 total drive bays per 2U.
Not quite the same, but they do have something like the Pods, but a bit more modular:
It's their PowerEdge MX platform, which allows you to slot in different "sleds" for storage/compute etc. as needed. It can take 7 storage sleds for a total of 112 drivers per chasis.
I'm curious what is being used for the drives (and to a lesser extent, memory) - Dell or OEM and how does support work?
We sell a lot of Dell and for base models, it is very economical compared to self built.
The moment however we add a few high capacity hard drives or memory, all bets are off the table and it's usually 1.75-4x the price of a white box part.
I get not supporting the part itself, but, had them not support a raid card error (corrupt memory) after they saw we had a third party drive.... we only buy a handful of servers a month - I can imagine this possibly being a huge problem for Backblaze though...
That appeared to depend on whether the vendor imposed massive markups on the drives. However they also mentioned service etc.: If they struck a deal with Dell, then Dell might be perfectly happy to sell the servers at a very modest profit while making their money on the service agreement.
Especially flash storage goes through the roof at enterprise purchasing. I've bought the drive trays and used consumer SSDs in servers more than a few times with no real ill effects where SATA is acceptable. If you need SAS, you just need to accept the pain that is about to come when you order.
> That’s a trivial number of parts and vendors for a hardware company, but stating the obvious, Backblaze is a software company.
Stating the obvious: Backblaze wants investors to value them like a SaaS company. This blog post suggests they’re more of a logistics and product company— huge capex and depreciating assets on hand. As a customer, I like their product, but they’re no Dropbox. If they would allow personal NAS then I could see them being a software company.
For me the Big Deal is Backblaze B2. Especially when fronted by Cloudflare -- zero traffic costs. Storage is cheap as far as cloud storage provider goes and traffic is decidedly the cheapest possible.
It's easy to buy a bunch of hard drives and connect in a data center. Managing petabytes per user for thousands of users is the hard part, and it's a software problem.
BackBlaze is definitely a SaaS company... though the quality of their offering certainly lags behind Dropbox, both in terms of feature set and user experience. They're also in a very competitive industry. Storage/backup is basically a commodity nowadays.
In general, for larger (> 10TB) drives, most definitely. < 10TB drives have been mostly unaffected.
The direct impact on larger drives is entirely dependent on brand (Toshiba's Enterprise drives appear to be less in demand than Seagate's Exos, for instance), recording process (CMR versus SMR), and, to a lesser extent, power consumption. In virtually all instances though, the price has jumped significantly [0]. 16TB Exos more than doubled, for instance [1]. In large parts of Europe, large drives have been back-ordered since the beginning of April; my orders are scheduled for delivery in August, yet I would be surprised if I saw anything before September.
I've been watching drive prices, it's hard to say exactly why, but around mid April disk prices at Newegg and amazon jumped significantly. One drive that had been $300, jumped to $400, $500, and even spiked to $800 for a bit. By Jun 1st had dropped to $550, and only this week has dropped for $400. Still above the original $300, but at least a not terribly painful premium.
Chia mining-before-transactions-are-released -> Chia transactions released -> Chia price at a high price -> Chia price halves shortly after, turns out virtually impossible for US users to get on any of the exchanges handling XCH
Doesn't look like it, at least in the EU. You can check for yourself here[0]. Just find a drive such as this one[1] and check the chart on the upper-right corner for price history.
Not for hyperscalers like BackBlaze. They have contracts with specific purchase quotas and guaranteed price deltas. Chia has certainly affected prices on the secondary markets, there hasn't been a better time in the past decade to be a secondary server "junk" seller on eBay! NetApp 4246 JBOD's are going for $1000! Absolutely insane!
Dell me be very happy to have a high volume customer with very standardized & predictable needs, and so they're happy with modest markups & extra profit on the service agreements, which is a nice benefit for Backblaze since building their own pods doesn't give them any service guarantee/warranty.
Dell (and, really, all server providers apart from Supermicro) have crazy markups on storage.
It's where they make most of their margin.
And then, most of the time, the drives they sell come on custom sleds that they don't sell separately as a form of DRM/lock in.
Then you get a nice little trade on Chinese-made sleds that sort of work, but not for anything recent like hot swap NVMe drives.
I'm sure BB were able to negotiate down a lot (Dell usually come down 50% off the list price if you press them hard enough for one off projects), but... yeah. That's how it generally goes.
If you are buying in bulk the pricing can be a lot better than the list prices. But generally there isn't a problem buying a server without a drive, Dell will still support the server for non disk related warranty issues, you don't need special firmware or disks
May just be me, but I feel like it’s probably a bad idea to outsource one of your core competencies. Backblaze is quite literally built on their storage pods.
It’s like outsourcing the EU telecommunications, which we’ve seen a bunch of articles about recently.
By my reading of the article, making up chassis and building out servers was outsourced already and was a pita because it wasn’t a core competency.
At least that was my take.
Boxes full of disks used to be special, in a past time there was only the glorious Thumper. BB pods and hyperscaler servers built by Quanta and Supermicro have pushed "boxes full or disks" into race to the bottom territory.
Software is now the only thing that separates services like BB, Dropbox, et al.
Don't cry for BB pods, they did their job and now you can get a higher quality chassis at a similar cost basis from Dell as a result.
Readit News
And now that multiple "storage pod-like" systems exist in the marketplace (not just Dell, but also Supermicro) selling 60-bay or 90-bay 3.5" Hard drive storage servers in 4U rack form factors, there's not much reason to build their own?
At least, that's my assumption. After all, if the server chassis is a commodity now (and it absolutely is), no point making custom small runs to make a hypothetical Storage Pod 7. Economies of scale is too big a benefit (worst case scenario: its now Dell's or Supermicro's problem rather than Backblaze's).
EDIT: I admit that I don't really work in IT, I'm just a programmer. So I don't really know how popular 4U / ~60 HDD servers were before Backblaze Storage Pod 1.0
What wasn't really a thing was servers with hotswap HDD trays on both ends (like the supermicros) and things that were designed with vertical hard drives dropped down from a top-opening lid to achieve even higher density.
It had hot-swap SATA disks (up to 512GB disks initially!) and was actually pretty cool and forward thinking
https://web.archive.org/web/20061128164442/http://www.sun.co...
Given the bandwidth of 60+ hard drives (150MB/s per hard drive x 60 == 9GB/s in/out), I'm pretty sure you need a decent CPU just to handle the PCIe traffic. At least PCIe 3.0 x16, just for the hard drives. And then another x16 for network connections (multiple PHY for Fiber in/out that can handle that 9GB/s to a variety of switches).
We're looking at PCIe 3.0 x32 just for HDDs and Networking. Throw down a NVMe-cache or other stuff and I'm not seeing any kind of small system working out here.
---------
Then the math comes in: matrix multiplications over every bit of data to verify checksums and reed-solomon error correction starts to get expensive. Maybe if you had an FPGA or some kind of specialist DSP (lol GPUs maybe, since they're good at matrix multiplication), you can handle the bandwidth. But it seems nontrivial to me.
Server CPU seems to be a cheap and simple answer: get the large number of PCIe I/O lanes plus a beefy CPU to handle the calculations. Maybe a cheap CPU with many I/O lanes going to a GPU / FPGA / ASIC for the error checking math, but... specialized chips cost money. I don't think a cheap low-power CPU would be powerful enough to perform real-time error correction calculations over 9GBps of data.
--------
We can leave Backblaze's workload and think about typical SAN or NAS workloads too. More I/O is needed if you add NVMe storage to cache hard drive reads/writes, tons of RAM is needed if you plan to dedup.
There are a small number of high end ARM server boards that could do it, but you’re not saving much money at that point. Might be more expensive due to lack of scale.
Amortize a server-grade CPU and motherboard across 60+ high capacity drives and it’s not really worth pinching pennies at the risk of lower performance.
I don't think they actually sold any.
Deleted Comment
Also, Dell and Supermicro have storage servers inspired by the BB Pods.
Glad to see this scrappy company hit this amount of scale; a long way from schucking Hard Drives
It's apparently the "Dell PowerEdge R740xd2 rack server".
[1] https://www.delltechnologies.com/asset/en-ae/products/server...
https://www.servethehome.com/dell-emc-poweredge-xe7100-100-d...
https://i.dell.com/sites/doccontent/shared-content/data-shee...
So did they do something custom (unlikely at this volume) or did Backblaze change their hardware approach?
It's their PowerEdge MX platform, which allows you to slot in different "sleds" for storage/compute etc. as needed. It can take 7 storage sleds for a total of 112 drivers per chasis.
We sell a lot of Dell and for base models, it is very economical compared to self built.
The moment however we add a few high capacity hard drives or memory, all bets are off the table and it's usually 1.75-4x the price of a white box part.
I get not supporting the part itself, but, had them not support a raid card error (corrupt memory) after they saw we had a third party drive.... we only buy a handful of servers a month - I can imagine this possibly being a huge problem for Backblaze though...
Every year they post a summary of what models they're working with and how they perform, which is usually good reading. This is last year's: https://www.backblaze.com/blog/backblaze-hard-drive-stats-fo....
That appeared to depend on whether the vendor imposed massive markups on the drives. However they also mentioned service etc.: If they struck a deal with Dell, then Dell might be perfectly happy to sell the servers at a very modest profit while making their money on the service agreement.
Stating the obvious: Backblaze wants investors to value them like a SaaS company. This blog post suggests they’re more of a logistics and product company— huge capex and depreciating assets on hand. As a customer, I like their product, but they’re no Dropbox. If they would allow personal NAS then I could see them being a software company.
BackBlaze is definitely a SaaS company... though the quality of their offering certainly lags behind Dropbox, both in terms of feature set and user experience. They're also in a very competitive industry. Storage/backup is basically a commodity nowadays.
agreed, but I think s3 is the more realistic comparison than dropbox
The direct impact on larger drives is entirely dependent on brand (Toshiba's Enterprise drives appear to be less in demand than Seagate's Exos, for instance), recording process (CMR versus SMR), and, to a lesser extent, power consumption. In virtually all instances though, the price has jumped significantly [0]. 16TB Exos more than doubled, for instance [1]. In large parts of Europe, large drives have been back-ordered since the beginning of April; my orders are scheduled for delivery in August, yet I would be surprised if I saw anything before September.
[0] https://www.tomshardware.com/news/analysis-hdd-prices-skyroc...
[1] https://camelcamelcamel.com/product/B07SPFPKF4
[0] https://geizhals.eu/?cat=hde7s&xf=1080_SATA+1.5Gb%2Fs~1080_S...
[1] https://geizhals.eu/toshiba-enterprise-capacity-mg08aca-16tb...
There's a massive amount of premined chia controlled by the "chia strategic reserve".
It will take a decade for the amount of mined value to equal the pre-mined value.
Dead Comment
It's where they make most of their margin.
And then, most of the time, the drives they sell come on custom sleds that they don't sell separately as a form of DRM/lock in.
Then you get a nice little trade on Chinese-made sleds that sort of work, but not for anything recent like hot swap NVMe drives.
I'm sure BB were able to negotiate down a lot (Dell usually come down 50% off the list price if you press them hard enough for one off projects), but... yeah. That's how it generally goes.
It’s like outsourcing the EU telecommunications, which we’ve seen a bunch of articles about recently.
Software is now the only thing that separates services like BB, Dropbox, et al.
Don't cry for BB pods, they did their job and now you can get a higher quality chassis at a similar cost basis from Dell as a result.