Readit News logoReadit News
rsync · a month ago
It's rough out there and has become increasingly difficult to maintain our pace of storage deployment.

Further - and most concerning - is the pollution of the supply chain with refurbished/recertified stock being sold and marketed as "new".

One example:

https://kozubik.com/items/MaestroTechnology/

I strongly advise buyers to stick with trusted suppliers, avoid Amazon/ebay channels, and carefully vet your incoming stock with SMART tools to ensure you receive what you think you are ... especially for SSD parts.

estimator7292 · a month ago
DO NOT assume SMART is reliable. You can wipe SMART stats or write any values you want.

You have to actually examine the real bits on the drive. Resellers don't want to take the time to actually zero a drive, they usually just nuke the partition table.

You also need to physically examine the drive. Corroded fingerprints on the PCB, wear on the port contacts, scratches from mounting rails, etc.

That's how it found out that the last "new" drive I bought on Amazon was actually a used Backblaze drive. It contained terabytes of customer data, and a shit ton of cleartext files. SMART, of course, reported it was a brand new drive with zero hours. Cleartext logs on the drive showed many thousands of hours of runtime.

Physical examination is the only reliable method.

neilv · a month ago
> That's how it found out that the last "new" drive I bought on Amazon was actually a used Backblaze drive. It contained terabytes of customer data, and a shit ton of cleartext files. SMART, of course, reported it was a brand new drive with zero hours. Cleartext logs on the drive showed many thousands of hours of runtime.

This sounds like it could be a big problem for Backblaze customers, and consequently for Backblaze.

Can you alert the Backblaze CEO about their insufficiently-decommissioned drives leaking out like this?

Backblaze customers also need to know, but I would give Backblaze the first shot at figuring out how to notify, whom, of what.

atYevP · a month ago
Yev from Backblaze here -> That sounds very strange. I've asked our ops team and they said if you can provide the serial number of the drive you have we can track it. Would you be able to write in a support ticket w/ the serial number of the drive you bought so I can personally flag it and investigate? Once done you can write the ticket # here and I'll follow up with the team -> https://help.backblaze.com/hc/en-us/requests/new.
SoftTalker · a month ago
> drive I bought on Amazon was actually a used Backblaze drive

Assuming this is true, I find it weird/surprising that Backblaze doesn't at least zero their drives before disposing of them? I have to do that at my work, and at least by policy I could lose my job if I skipped doing it.

jacquesm · a month ago
Backblaze should get ironclad guarantees that decomissioned drives have been destroyed, besides that their storage system should not be storing anything in plaintext at all.
nerdsniper · a month ago
> actually a used Backblaze drive

Unless you're super, duper sure this is true ... you might want to post your evidence, rather than your conclusion. Any misinformation these days gets amplified even faster and bigger than it did just 1-3 years ago, especially via ChatGPT and Gemini.

gosub100 · a month ago
I don't understand any scenario where a NAS drive would have a normal filesystem with plainly readable files. Not because of security (as many would expect) but because drives used in arrays are striped and intended for parallel use. They do not contain complete individual files. I know this because when you lose more than 2 drives in, e.g. a RAID 5, you lose everything.

Equally unlikely is why a storage provider would use anything OTHER than an array for any of their drives. Again, tossing security aside, it is not a viable way to store data on one filesystem. That drive could die at any time and lose all the data. So it makes no sense to store it that way.

hnu0847 · a month ago
> You have to actually examine the real bits on the drive.

What is the best utility for doing this?

kkylin · a month ago
Question for all of you more knowledgeble than I: can SMART data be tampered with? When I get, say, a refurbished Mac from Apple, I'm trusting Apple won't stoop to that. But a SSD vendor I've never heard of?
rsync · a month ago
Yes. There are vendor-specific utilities that have escaped into the wild that allow bad actors to reset various SMART counters, etc.

A lot of abuse came to light during the launch and initial mining of the (ridiculous) Chiacoin[1] during which Chia miners would burn through SSDs to within a hair of their usable life, reset their SMART stats, and sell them as new on Amazon or ebay.

As can be seen in my above comment, larger distributors like "Maestro Technologies" have their stock polluted with parts like this and I find it very unlikely that they are not aware of the status of these parts they are selling as new.

[1] https://en.wikipedia.org/wiki/Chia_Network

fny · a month ago
Yes, it can be tampered with. Drives can even lie about the amount of storage they support. I once bought a 1TB pen drive that was only 32MB for $10. (Yes, I knew it was a scam beforehand.)
ikiris · a month ago
Trivially
PeterStuer · a month ago
I have given up on Amazon and similar "market makers" (bol.com is a regional one here) completely. Too much fraud. Instead I use specific vertical or store as sourced outlets.
catigula · a month ago
Nearly any product you can buy from Amazon, even when it says shipped from Amazon, is suspect.

I wouldn't shop there at all. It's a literal scam market. Allegedly.

tracker1 · a month ago
I try to stick to at least "sold by amazon" as much as possible, that or the mfg. Generally, even with comingled inventory, Amazon has been good about replacements/refunds.
bdcravens · a month ago
I've never considered myself very paranoid about Amazon, but recently I needed a cheap router, but couldn't shake the feeling that I shouldn't get it there (went with an in person Best Buy purchase instead)
hulitu · a month ago
> avoid Amazon/ebay channels,

But why ? They are the good ones. TEMU and Shein bad. /s

walterbell · a month ago
"OMEC" (Organization of Memory Exporting Countries) NAND production quotas?

https://x.com/jukanlosreve/status/1988505115339436423

  Samsung Electronics has lowered its target for NAND wafer output this year to around 4.72 million sheets, about 7% down from the previous year's 5.07 million. Kioxia also adjusted its output from 4.80 million last year to 4.69 million this year.. SK hynix and Micron are likewise keeping output conservatively constrained in a bid to benefit from higher prices. SK hynix's NAND output fell about 10%, from 2.01 million sheets last year to around 1.80 million this year. Micron's situation is similar: it is maintaining production at Fab 7 in Singapore—its largest NAND production base—in the low 300,000-sheet range, keeping a conservative supply posture.
Micron has a new US fab coming online in 2027, which should improve supply.

DRAM price fixing scandal: 1998-2002, 2016-2018, https://en.wikipedia.org/wiki/DRAM_price_fixing_scandal

acd10j · a month ago
It’s a cartel just like the OPEC oil cartel.

Deleted Comment

Eddy_Viscosity2 · a month ago
Is there anything that isn't functionally a cartel at this point?
zkmon · a month ago
This is getting out of control. Like a monster eating off all the things that are needed for normal people. Worse than nukes race, which at least did not affect common man. No one really gains anything. All gains are temporary competitive edge, that vanishes quickly while locking everyone into the vicious cycle.
joegibbs · a month ago
Worse than nukes race? I really much prefer being slightly inconvenienced when buying hard drives to risking being obliterated along with billions of other people and having all of civilisation wiped out with hundreds or thousands of years until recovery.
butlike · a month ago
At least nuclear holocaust is easily quantifiable
pinkmuffinere · a month ago
Maybe the real paperclip maximizer was the irrational exuberance we made along the way :')
WorldPeas · a month ago
> Worse than nukes race, which at least did not affect common man At least not visibly, we were deprived of a clean source of surgical steel and many in the areas were uncompensated for sicknesses they experienced as a result of testing
Dylan16807 · a month ago
Why would it matter for surgical steel?

I can find vague mentions of "medical" uses for low background steel, and one specific claim that it's used to shield some rooms or large devices for measuring how much radiation is coming off a person's body. That's pretty niche.

solarwinds · a month ago
Yeah, another hot take from some guy who has no clue what he's talking about.

To this day Kazakhstan pays out monthly monetary compensations to people whose health was negatively affected by the nuclear tests on the Semipalatinsk nuclear polygon. Several people I know personally receive these compensations.

https://egov.kz/cms/en/news/nuclear_tests

https://egov.kz/cms/en/services/family_support/pass035_mtszn

lisbbb · a month ago
I just find it so odd that the oh so environmentally conscious tech sector is now building out some of the most resource intensive facilities on the planet. It's like environmentalism was just a con.
nradov · a month ago
The tech sector "environmentalism" was obviously always just performative. They went along with the movement because it cost them virtually nothing, deflected pressure from the progressive political faction, and helped them get invited to fashionable parties. Now there are real costs so it's no longer convenient.
rr808 · a month ago
> No one really gains anything.

That's not true I've seen loads of amusing videos on Instagram.

WheatMillington · a month ago
>Worse than nukes race

That certainly is a take.

Dylan16807 · a month ago
I don't know if I agree, but it's not a very extreme take.

Pretend every country with lots of nukes instead had no more than 100 warheads at a time. Would the world be much different? Would the average person be better off in a noticeable way?

Would it make a big difference if we never had hydrogen bombs? Note that we already had them in the early 50s. Everything since then was about increasing the quantity and refining designs, no meaningful size increases.

fajitaforce5 · a month ago
Maybe when the bubble bursts there will be lots of good used hardware at massive discounts?
criemen · a month ago
But why? Training hardware maybe will appear, but everything that's running inference has real, paying customers on it, with current capabilities level. Even if the bubble bursts, why would that demand evaporate? People still would want to use chatgpt, claude code etc., even if they stop getting any better tomorrow.
outside1234 · a month ago
Don’t worry, OpenAI is about to implode.
mchannon · a month ago
During 1998-2000, AOL was ordering so many free trial CDs that it locked up world production, and music CDs faced 8-12 week delays. It was rumored that certain weeks there were no albums getting fabricated at all, worldwide.

I wonder if history isn’t repeating itself. AOL CDs had pretty much jumped the shark by 2000.

PeaceTed · a month ago
All that manufacturing demand led to enough free machinery to get those free Ubunutu CD's! ;)
cmckn · a month ago
Man, I loved those. A Kubuntu CD felt like such a treat to 10 year old me.
jazzyjackson · a month ago
That's somehow unsurprising given how many AOL CDs I saw at checkout of every grocery store
dredmorbius · a month ago
Stacks of them at a local post office back in the day. Covering every horizontal surface.
criemen · a month ago
So far, due to the AI boom, we're lacking:

* HDDs

* SSDs

* DRAM

* GPUs, obviously

* Power to hook up to our datacenters

Anything I'm missing? What a crazy world we're living in.

penguin_booze · a month ago
Intelligence. We're lacking intelligence.

Deleted Comment

Deleted Comment

magicmicah85 · a month ago
Water? Clean Air? ROI?
captainkrtek · a month ago
Nearly nothing left unexploited in the pursuit of profits.
Fabricio20 · a month ago
Water is one I really don't get, if anyone would be willing to explain it to me. Is it because of manufacturing? We already have these manufacturing plants, it's not like more are being started, that process takes years to decades. Hell intel couldn't even finish the few they were working on. Because of concrete for datacenters? I'm sure developing countries are using significantly more of that to build houses, so much so that it completely eclipses the one-to-two datacenters you might see built in the next 5 years in your area. Power generation? That water goes straight back to the atmosphere.. water cycle, even on the open loops, water becomes steam which becomes rain..
downrightmike · a month ago
Don't bring ROI into this, that's loser talk
1970-01-01 · a month ago
* True innovation

Every tech startup circa 2023 doesn't do anything without smearing AI all over their front page.

maerF0x0 · a month ago
Makes me wonder about the things that go into datacenters and power plants:

- Air conditioners

- generators

- Racks/Cages

- fire supression

- Concrete, Steel, power lines

etc

PeaceTed · a month ago
For all the issues with power use, I do grant it that a lot of materials and labor in needed at the same time. It isn't JUST burning up cash to make Sora videos.

But by the same measure, the Great Egyptian pyramids would have been a huge boom for work even if the final product didn't achieve much for most people.

wmf · a month ago
All those have multi-year backlogs.
QuantumGood · a month ago
A coherent regulatory framework? An honest conversation about whether we need all this?
falleng0d · a month ago
Profit too. We lack profit.
derefr · a month ago
I have a hypothesis that it was AI, not COVID/sanctions/etc, that was mainly responsible for the 2020-to-ongoing "chip shortage." Ignoring companies with their own fabs (Intel) and companies with pre-existing reserved-capacity contracts with fabs (Apple), everyone else is stuck waiting in line behind batch after batch of fab orders from Nvidia.

Downstream of that, AI is effectively also responsible for the current generation of game consoles never declining in price.

Because game consoles are fixed platforms that continue to be manufactured over 5+ years, normally the most expensive parts in the system (the CPU and GPU) would gradually get cheaper to manufacture [and in turn, cheaper to buy] over the course of the console's lifetime—which was often passed onto the consumer in the form of the console's MSRP gradually decreasing. Either the process node for the console's silicon design would stay fixed, and demand for this process node would gradually decrease as larger fab customers move on to newer nodes, decreasing the (effectively auction-based) pricing for fab time on the older node; or the console manufacturer['s silicon vendor] would put their silicon design through a process-shrink, and so, while still paying top dollar for use of the fab's newest node, would be getting more chips per wafer out of that, and again could charge less.

But instead, what we've seen since the start of the AI boom is that there's no longer any price-reduced timeslots to be sold to manufacture the low-BOM parts for these price-sensitive console-maker customers. Instead, both Nvidia and AMD are now getting such high value out of even the older nodes, that 1. the fabs know they can squeeze them, charging full price for those slots as well; and 2. both Nvidia and AMD, in their roles as silicon vendors to the console makers, haven't been able to justify using (very much of) the time they're paying so much for to fab low-BOM-cost parts to fulfill their pre-existing outstanding purchase orders, when they could instead be fulfilling much-higher-margin new POs from the hyperscalers.

Thus every console of the ninth generation (PS5, Xbox Series, Switch) still selling for their launch price (with no help from process shrinks); thus none of these consoles having been able to be produced in excess of demand to the point that supply-and-demand ever drove retail prices below MSRP; and thus the only tenth-gen console so far, the Switch 2, taking ~4 years longer than anticipated to release†.

---

† Nintendo were very likely waiting for Nvidia to run off enough of the Tegra T239 with a sufficiently low passed-on fab cost, for Nintendo to both 1. be able to build a backstock and non-run-dry pipeline of Switch 2s, and 2. be able to be positive-margin on charging the same price as the Switch 1 for them. They waited four years, and neither thing ever happened; so they eventually just gave up and priced the Switch 2 higher, and also built out an entirely novel D2C + online-marketplace-partner distribution pipeline so they could ration the tiny initial supply of units they had been able to build with the chips Nvidia had supplied them so far.

Though, that being said, Nintendo actually got a double whammy here. They were also waiting for fast NAND to come down in price, so that they could have physical game cards manufactured for a trivial BOM price while still enabling the "direct GPU disk-streamed assets" pipeline that games of the last generation had begun relying on. Obviously, as today's article points out, that hasn't been happening either! Thus game key cards; thus SD Express cards only beginning to trickle out, with no sizes above 128GiB available at the Switch 2's launch time; and thus those SD Express cards being ridiculously priced for their capacity compared to equivalent transfer-speed + die-size NAND (as seen in e.g. low-profile/flush-mount flash drives.)

magicalhippo · a month ago
> I have a hypothesis that it was AI, not COVID/sanctions/etc, that was mainly responsible for the 2020-to-ongoing "chip shortage."

Might be for higher-end chips like GPUs, but for smaller ones like microcontrollers it was very much COVID that threw a huge wrench in the gears.

Things are closer to normal now, but the toilet paper effect hit the electronics supply chain hard during COVID. People resorted to buying dev boards just to desolder the microcontroller on it to use in their commercial product, and similar desperate moves. With lead time measured in many months to years even when the factories are operational, sudden hoarding is not what you want.

keyringlight · a month ago
Something I think contributes is that TSMC seem to be so good at making chips now, it seems like the defect rate must be low enough and they can produce huge chips in volume that a decade ago would be burning money. If you can do that, why bother making less lucrative smaller chips? Also while there's still benefits to shrinking an existing chip with a revised design, I'd guess we're into diminishing returns for what it offers a console whether the chip itself gets cheaper or lets them reduce the bill of parts in other ways.

This is also the first generation where prices have gone up IIRC.

PeaceTed · a month ago
That sounds reasonable but I suspect it was a little bit of both. Initially Covid slow down followed by large demand. A lot of older Nodes were shutdown (300-400nm) with the Covid slow down and a lot of chips ended up having to be moved over to newer nodes once demand picked up again. It led to a big swing in the flow of production and would have roll on effects.

Combine that with a huge GPU boom and you have the setup for production issues.

lisbbb · a month ago
But the basic chips like the ones that go into cars are the ones that were the most scarce--the low margin stuff was not being made. You're saying it's because of Nvidia? I'm not totally tracking here.

The game console thing is also related to monetary inflation--the price of everything just keeps jumping.

domador · a month ago
* Time to wade through all the AI slop

* Trust that what we are watching or reading is human-sourced and real

* Time for real creators' unique voices to remain unique

* Hope for society's future

anigbrowl · a month ago
We should be making some effort to quantify the amount and cost of slop produced by both AI and simpler automated systems (spinners etc), it's a huge negative externality.
tintor · a month ago
Well, at least the paperclip maximizer is not running out of water and carbon.
outside1234 · a month ago
Definitely lacking profit. OpenAI is on track to lose $60B this year.
jacquesm · a month ago
Don't forget rising energy prices.
stuff4ben · a month ago
jobs...

Deleted Comment

IAmGraydon · a month ago
I built a computer December 2024 with a Ryzen 9900X, 64GB DDR5, 1TB + 4TB SSDs, 4060. If I were to buy the exact same parts now, nearly 1 year later, it would cost about $1,000 more for the exact same components. The memory alone went from $189 to $505. I'm not sure how much of that is from tariffs and how much is due to AI datacenter demand, but that is a massive increase.
LTL_FTC · a month ago
Yup. Walked into a microcenter a few weeks ago and was surprised and saddened by how expensive everything has gotten.
mschuster91 · a month ago
So, it's Chia all over again? [1]

Sometimes, I think government supply control isn't that much of a bad thing. That way, the government could force the availability of goods for the common market as well and not just for the really big dogs flush with ample VC money to burn who can pay any price.

[1] For the uninitiated: a cryptocurrency where the limiting factor wasn't CPU, RAM or GPU compute resource, but storage - in 2021, there was so much craze around it that HDD and SSD prices exploded, and after the bubble collapsed a lot of heavily abused drives flooded the markets.

LTL_FTC · a month ago
Chia was/is WORM (write once , read many) so not sure how these drives were heavily abused? As a storage nerd, I bought many 18tb drives for chia and they are still spinning just fine in my storage arrays. It’s been five years.
mschuster91 · a month ago
Honestly no idea, all I 'member is reporting from back in ye olde days [1]. Damn, time really flies.

[1] https://www.techpowerup.com/281979/chia-farming-already-caus...

evanjrowley · a month ago
The future of AI is everything we liked getting ruined by AI.