Love seeing these projects from him, but this is a rare miss in my opinion. This is the strange middle ground where it's closer to professional than "budget", by quite a bit. At some point, only a tiny tiny fraction of users need 40TB of space. I guess what I'm saying is, this isn't so much a NAS as a specialized youtuber video recording appliance. We have a number of home-hosting users on our platform that run entire racks filled to the brim - and while that's awesome - it's just not what "budget" or "NAS" implies, and it's an extremely limited audience.
At the end of the video, the config he settled on doesn't have 40TB of useable space. He only has 16TB of useable. Essentially, a RAID-10 with a hot swap. He's only getting the speed of striping 2 drives together. Sounds like he's playing it safe choosing a bit of redundancy rather than just straight speed.
In this case, the majority of my income depends on the edit server being online (thus the decision to go basically RAID10 + hot spare). I have a full backup plus an offsite copy, but they are over 2.5 Gbps on spinning disk, or take 6+ hours to restore from offsite, so they're emergency use only.
I'm also trying to get my homelab set up to be a bit more robust / automated / reproducible, and trying out a bunch of different ideas. I'll likely settle on something else in a year's time, but the main motivation for now was to get all the storage into my rack on my 10 Gbps network.
Ehhhh, he said "All-SSD NAS on a budget", not a "Budget SSD NAS".
Agree though that this was a bit of a miss; the cost of drives here dwarfs everything else, and it makes the choice of case, MB, etc. nearly irrelevant and needlessly cost optimized. Why not spend an extra $60 and get a 5.25->6x 2.5 hot swap cage, for example?
Yeah this was the big criticism on r/homelab, that it's not a budget build at all, it's very misleading. Most of that build was free because he is who he is.
The only other thing that annoyed me is calling TrueNAS Core (FreeBSD) a "Distro" in the same sentence as a Linux distro.
TBH, not sure if spending $3500 on 40TB storage v.s. ~$800 with rotating disks at the same storage capacity. You can put $200 on top with a 2TB NVMe SSD as cache.
The reason to question this is that 40TB seems small if you want to have a NAS for small video editing studios. And for personal use, you probably not going to need more than 2TB work set paged in at any given moment.
personally if I had to do this I would go with rotating disk for bulk storage in a NAS, and something like two 2TB to 4TB size NVME SSDs in a proper video editing workstation motherboard directly attached to pci-e 4.0 bus.
This will be considerably faster for working with "immediate" needs of video files rather than over a 10GbE network.
like, a difference of 900MBps over network vs 2500MBps with local sequential read/writes on NVME SSD on same motherboard.
Possibly for smaller projects, for anything remotely sizable, 2TB is likely not going to cut it. 5k prores is 1TB for every 30min of footage, which means you're only getting an hour out of a 2TB drive.
Storage needs for any pro video workflow get very large, very quick.
Well you're talking about 60fps ProRes 4444 (XQ?) footage, and the article is talking about 30fps ProRes 422 LT footage. That's 6.5x (10x?) less data for the same size project, and not much difference in how "pro" it is.
I'm not familiar with NAS file systems. Is it fairly straight forward to use hard drives with SSDs as transparent cache, and make it look like a single file share?
- ZFS caches reads into RAM. Gobs of RAM helps. You can also add a secondary read cache device (L2ARC).
- ZFS can caches synchronous writes to an SLOG device, but if you've disabled synchronous writes it won't make a difference. An SLOG device makes no difference for asynchronous writes.
I use ZFS for my Time Machine backups (among other things) and have synchronous writes disabled for the Time Machine datasets.
Before we had ZFS, the traditional way to speed up NAS was a raid controller with a battery-backed RAM cache.
I would be extremely cautious about using any consumer grade TLC or quad-level-cell SSD in a "NAS" for serious purposes because of well known write lifespan issues.
There's a reason that a big difference in price exists between a quad-level-cell 2TB SSD and an expensive enterprise grade one with a much higher TB-write-before-dead rating.
This might look cool but check back in a few years and see how much of the drives' cumulative write lifespan is worn out.
I also cannot even imagine spending $4000+ on a home file server/NAS with copper only 10GbE NIC and it not having at least one 10G SFP+ interface network card.
Okay, so he wants it to be tiny? But in a home environment the major problem is more power consumption and noise, so you can often go with a well ventilated 4U height rackmount case for full size ATX motherboard, which is roughly the size of a midtower PC case turned on its side.
This lets you use motherboards that will have enough PCI-E 3.0 x8 slots for at least one dual-port Intel SFP+ 10G NIC which are very, very cheap on ebay these days.
> I would be extremely cautious about using any consumer grade TLC or quad-level-cell SSD in a "NAS" for serious purposes because of well known write lifespan issues.
I don't know what you're using your NAS for, but the author is using it as scratch space for raw video files. It's not an OLTP DBMS or anything. It just needs really fast ingest of files beyond the capacity that a DRAM cache can provide.
> I also cannot even imagine spending $4000+ on a home file server/NAS with copper only 10GbE NIC and it not having at least one 10G SFP+ interface network card.
The author's editing rig doesn't necessarily have a 10G NIC, let alone being attached to a 10G switch with runs of CAT6a; and there's only one device talking to this NAS at a time (as the author cannot be in two places at once.) So what'd be the point?
If the author's editing rig doesn't have a 10G port, what is the point of spending money in $ per TB for the SSD on the other end of a network connection, because something like a 6 drive raid of spinning disks can very easily saturate approx. 100MBps on a 1000baseT link.
TrueNAS used to be designed to boot off of smaller SataDOMs that were used only for boot. They were effectively WORM. At least, it used to be a few years ago. Everything that was written for the server was either a RAM disk or spread out amongst the RAID drives (as a separate partition, which has its own issues, but still).
I had assumed this is what he was using the TLC SSD for. If that’s the case, so long as there isn’t much writing to it, it should be fine.
SMART will report expected remaining lifespan. Another thing you can do is write k TB per drive where k is 0, 1, ... N-1 where N is the number of drives. This staggers the endurance so they don't all fail simultaneously.
> I would be extremely cautious about using any consumer grade TLC or quad-level-cell SSD in a "NAS" for serious purposes because of well known write lifespan issues.
How many full-drive writes does your average video editing server need? I would expect a pretty small number. The average source file is sitting there for weeks or months.
Yeah, my bet is that the rarity of writes (some metadata in the video project files) will give these drives plenty of longevity.
For a use case like database transactions, log storage and frequent data dumps, the game changes quite a bit. I would definitely shy away from the QVO drives for that use case.
I've had these drives in service for about 8 months in my regular NAS, before transferring them to this new build, and they are all checking out okay still.
But this is also why I'm doing the striped mirror plus a hot spare. The only real challenge would be if the drives have a firmware issue, and they all die at the same moment after like 4 years due to a bug (like how HN's servers died...).
Jeff edits on a Mac with a 10 gigabit nic; he says as much in the article. Unless he's in an odd duck situation with an SFP device connected to his Mac I'm not sure what value going to SFP and back would add?
SFP is nice for the switch-to-device connection. It lets you choose between DAC (cheap and easy to route) if you're close enough, fiber, or copper (your choice). If I had an option, I'd use SFP+ everywhere.
But copper is usually a little simpler for consumer/prosumer devices. Someone does make a Thunderbolt to SFP+ adapter but that things like $300!
Personally I'd not be too worried since the last QVO devices I put in a NAS lasted three years, and had about 300TB of reads and 500TB of writes before they triggered the SMART endurance alerts and were replaced.
At 500x whole-drive rewrites I think I got my money's worth out of an $84 1TB drive.
Samsung QVO SSDs aren't very cheap to justify its lower TBW, especially if you write much like that. There are many TLC drives with great TBW rating at similar price, like Seagate Firecuda. Some are DRAM-less but HMB is good enough for vs QLC drives.
"I was able to get consistent performance using a striped RAIDZ mirror pool with four SSDs and a hot spare"
That doesn't make sense. The author is not using raidz. In ZFS terminology, he's mirroring two striped vdevs (like raid10), plus using a hotspare. And that is a bad choice, as he gets only 16 TB usable, and the mirrored stripes could fail with certain combinations of two drives failures.
Instead, he should have set up a raidz2 across 5 drives: 50% more usable space (24 TB), can tolerate any two drive failing, and it would give him higher performance on sequential I/O.
I have a raidz2 pool across 6 spinning rust 18TB hard drives, and my server can handle 900 MB/s sequential reads, and 700 MB/s sequential writes (benchmarked locally). It it was built on 5 SAMSUNG 870 QVO SATA III SSD drives like the author, it would certainly get to 1.5+ GB/s in both sequential reads and writes. In other words the bottleneck would shift to the 10 Gb/s network (1.25 GB/s). For comparison the author discloses his writes are limited to 700 MB/s (over the network, so his bottleneck is not the network link but local I/O contention).
That's right, a 6-HDD raidz2 matches the write performance of his 5-SSD raid10 setup. And that's because the write overhead of raid10 is 100%, while the write overhead of a 6-drive raidz2 is only 50%.
Edit: Oh and the strange performance issue that he noticed "disappeared" after a while is most likely due to cache recovery. The drive can sustain writes of about 490 MB/s for a little while then it drops to 170 MB/s, but after idling 5 minutes it can recover the initial speed. See https://www.tomshardware.com/reviews/samsung-870-qvo-sata-ss...: "As we noticed with the 1TB model, the 8TB model’s cache recovery mechanisms work similarly. After letting the drive rest at idle for 30 seconds, the 870 QVO gains back 6GB of its cache. It recovers fully with 5 minutes of idle time. "
Note that the 8TB model has about 50TB of cache (quite a bit more than the 1TB model—it doubles in size with each doubling of storage capacity).
One concern Wendell had with RAIDZ2 was the potential for the older Xeon to be a bottleneck for writes. It probably wouldn't be, but I didn't do too much testing with that layout. I might still, we'll see.
For reference, my server CPU is an AMD Ryzen 7 PRO 5750G (65W TDP, 3.8-4.6GHz, 8-core, 16-threads) and CPU usage is around 8% when writing at 700 MB/s on my raidz2. In other words, only a little more than 1 of the 16 hardware threads is utilized to handle this workload. Your older Xeon D-1521 should be able to handle that, but it's true that being an older processor, it is pushing it closer to its limits. Handling concurrently 10 Gb/s of network traffic would likely push it to more than 50% utilization. That's one reason I like to not cheap out too much on the CPU, especially for a 10GbE ZFS NAS.
In this case "on a budget" means $4,329. That's reasonable if it speeds up billable work, but sadly the cost puts it a bit out of reach for my home office.
That's a neat 2U case design, and will fit in some very shallow wall-mount network switch cabinets.
For installing outside of a machine room/closet/center, if you're using 2U of height, you might also fit a PSU with a larger and quieter fan, since all the Flex PSUs I've had come with noticeably loud fans. (I replace them with Noctuas, but it isn't a fun kind of soldering, IMHO.)
The components from the build would also fit in a Supermicro 1U short-depth chassis, especially if you can go a little deeper in your cabinet. (My new K8s server got a used Supermicro 1U chassis for ~$60 shipped, including a PSU. In the photo on https://www.neilvandyke.org/kubernetes/ , it's the 1U immediately below the 4U.)
Anyone know if any core level work on ZFS where there’s an effort to audit the code base for speed ups given the big differences in designs between spinning rust and SSDs?
I do know that if you have very fast NVMe SSDs (>6000MB/s or so) ZFS is not currently able to give you the whole performance, due to time spent memcpying to/from the ARC[0]. Direct IO support could eventually alleviate this[1].
SATA SSDs top out at about 550 MB/s. ZFS does ok with these. NMVe SSDs top out at somewhere between 3000 MB/s (typical enterprise PCIe Gen 3) and 14000 MB/s (mythical just released PCIe Gen 5 drives). ZFS (out of the box, anyway) can’t drive PCIe Gen 4 or newer drives very hard.
If your IO all goes over a 10 Gb link, that will be your bottleneck before ZFS is.
Since 1 MB record size was used, this may be a candidate for a bunch of mirrored HDDs with a few TB of nvme for l2arc and ZIL. But with a bunch of SSDs already on hand, why bother?
> Every minute of 4K ProRes LT footage (which is a very lightweight format, compared to RAW) is 3 GB of space
Do you really need to save all of that footage? I would think keeping the pro res footage for the current projects on the workstation and reencoded archive video on a NAS would be sufficient. I'm not a video professional, but I suspect it's easy to fall in the trap of thinking that you need to save everything in highest possible quality in case you need it later, but what are the realistic chances of that? If you end up needing some old footage again, AV1 coded 4K or even HEVC 1080p would probably be just fine. The final result are Youtube videos after all.
I know he mentions editing from it, but that's enough space for more than a week of pro res video.
I am a video professional, and you always keep your originals. However, the part being left out of the conversation is that as a storage pool for editing goes, this isn't deep storage. Content is typically only left on the edit storage while the project is active. Even the TFA mentions he goes all the way to cloud cold storage.
Edit storage has always been expensive, and maintaining capacity was always a juggling act. Just because terms like NAS are being used, one should not think of this type of storage as a dump it and forget it type of storage.
There are many levels of professional. On the high end, the footage from the camera is copied to multiple hard drives on set. These are the backups, and us old timers still use the terms camera originals. These are as sacrosanct as film negatives, only there's magically multiple copies. This data gets transferred to the editor's storage. Once the edit is completed, the edit session and other content used in the edit may be transferred to the camera original drives for archival. On the other end of professional, you take the SD card out of the camera and transfer directly to the edit storage. In these situations, woe be unto thee that doesn't make proper backups. At that point, it's really more pro-sumer than professional, but hey, if they're getting paid, they're professional enough.
So lets say you release a 10 minute Youtube video. You now have 60 minutes of raw footage with multiple takes, you sitting in front of the rolling camera going over the script, ums and ahs, and maybe a take where the phone rang in the middle of a sentence. What are you going to use that footage for in the future, and is this hypothetical use case absolutely dependent on having raw 4K video? I understand that your answer is from a general video production standpoint, but my question was more in the line of "is this really necessary for this kind of content creation?".
And yes, I did see the part about editing, but are you really editing nine days worth of footage at once? Would it make more sense to put less but faster storage directly in the workstation?
Love the write up and insight into the world of a professional.
Don’t take this the wrong way, I’m genuinely curious… what is a video professional doing on HN? Was it a past life? Is tech just a side interest? I see many non-tech professionals on here (doctors lawyers too) and I’m always curious what drew people in!
Personally I kind of fright from the "have you considered throwing away your best copies" as a sales strategy.
On the flip side, I think it would help a lot a lot a lot if there was more discourse out there about compressing high-dynamic-range content. AV1 supposedly has some capability to do a good-ish job with HDR. The idea of taking reels of raw video and spending a couple days squishing it to 1/10th the sizes but preserving the quality/flexibility very very well is a value proposition that I dont think is clearly attainable, even though it seems technically perhaps within reach.
Right now, I think the general feeling is, raw is raw & everything else bakes in a vast amount of assumptions & constrictions. Some advocacy that compressed video can be as flexible, as dynamic, as capable need to be more present, elaborated, & proven before anyone's going to be comfortable throwing away the bits. These are people's life's works & a couple hundred or thousand dollars a year more in storage isnt a real factor for such integral, near work.
I don't have a sales strategy, I'm not selling anything. I'm questioning the need of buying needless storage for data hoarding you're never going to use.
Have you seen Jeff's videos? They're mostly him talking to the camera in what I think is his basement, or closeups of various PCBs. If you're running a stock photo company I get wanting to keep footage at max quality, but once your Youtube video is published, do you really need to save all those alternate takes and cut out bits? I just don't see them being very useful in the future, and if you do end up needing some of it for B roll or whatever, would a couple of seconds of compressed video really make a drastic difference for the project as a whole? Especially since the final result ends up on Youtube and consumed on a phone screen or a TV two meters away.
Much of the content that's produced today is not trying to be timeless classics with endless rewatch value, it's ephemeral and only really relevant for a short time. If you can reupload your videos to other video hosts in the future that's probably good enough. Nobody is watching reviews of three year old Raspberry Pi add on boards for the low noise in dark parts of the image, or the accurate color reproduction, or even the 4K instead of 1080p resolution. Kill your darlings!
Not a video pro, but a counterpoint: "lossy encode, then process" is inferior to "process, then lossy encode". Reasoning: Some processing steps may amplify the noise introduced by lossy encoding. Other processing steps may be sensitive to SNR. Such steps chained together in a pipeline can make the otherwise small extra noise unacceptably big.
Note that I suggested reencoding footage for archiving, not processing. Although again, the final result ends up on Youtube and gets reencoded by them and viewed on all kind of devices.
Readit News
Love seeing these projects from him, but this is a rare miss in my opinion. This is the strange middle ground where it's closer to professional than "budget", by quite a bit. At some point, only a tiny tiny fraction of users need 40TB of space. I guess what I'm saying is, this isn't so much a NAS as a specialized youtuber video recording appliance. We have a number of home-hosting users on our platform that run entire racks filled to the brim - and while that's awesome - it's just not what "budget" or "NAS" implies, and it's an extremely limited audience.
I'm also trying to get my homelab set up to be a bit more robust / automated / reproducible, and trying out a bunch of different ideas. I'll likely settle on something else in a year's time, but the main motivation for now was to get all the storage into my rack on my 10 Gbps network.
Agree though that this was a bit of a miss; the cost of drives here dwarfs everything else, and it makes the choice of case, MB, etc. nearly irrelevant and needlessly cost optimized. Why not spend an extra $60 and get a 5.25->6x 2.5 hot swap cage, for example?
The only other thing that annoyed me is calling TrueNAS Core (FreeBSD) a "Distro" in the same sentence as a Linux distro.
The reason to question this is that 40TB seems small if you want to have a NAS for small video editing studios. And for personal use, you probably not going to need more than 2TB work set paged in at any given moment.
This will be considerably faster for working with "immediate" needs of video files rather than over a 10GbE network.
like, a difference of 900MBps over network vs 2500MBps with local sequential read/writes on NVME SSD on same motherboard.
Storage needs for any pro video workflow get very large, very quick.
https://www.truenas.com/docs/references/slog/
The short version is:
- ZFS caches reads into RAM. Gobs of RAM helps. You can also add a secondary read cache device (L2ARC).
- ZFS can caches synchronous writes to an SLOG device, but if you've disabled synchronous writes it won't make a difference. An SLOG device makes no difference for asynchronous writes.
I use ZFS for my Time Machine backups (among other things) and have synchronous writes disabled for the Time Machine datasets.
Before we had ZFS, the traditional way to speed up NAS was a raid controller with a battery-backed RAM cache.
[1] https://diskprices.com/?locale=us&condition=new&capacity=12-...
There's a reason that a big difference in price exists between a quad-level-cell 2TB SSD and an expensive enterprise grade one with a much higher TB-write-before-dead rating.
This might look cool but check back in a few years and see how much of the drives' cumulative write lifespan is worn out.
I also cannot even imagine spending $4000+ on a home file server/NAS with copper only 10GbE NIC and it not having at least one 10G SFP+ interface network card.
Okay, so he wants it to be tiny? But in a home environment the major problem is more power consumption and noise, so you can often go with a well ventilated 4U height rackmount case for full size ATX motherboard, which is roughly the size of a midtower PC case turned on its side.
This lets you use motherboards that will have enough PCI-E 3.0 x8 slots for at least one dual-port Intel SFP+ 10G NIC which are very, very cheap on ebay these days.
I don't know what you're using your NAS for, but the author is using it as scratch space for raw video files. It's not an OLTP DBMS or anything. It just needs really fast ingest of files beyond the capacity that a DRAM cache can provide.
> I also cannot even imagine spending $4000+ on a home file server/NAS with copper only 10GbE NIC and it not having at least one 10G SFP+ interface network card.
The author's editing rig doesn't necessarily have a 10G NIC, let alone being attached to a 10G switch with runs of CAT6a; and there's only one device talking to this NAS at a time (as the author cannot be in two places at once.) So what'd be the point?
I had assumed this is what he was using the TLC SSD for. If that’s the case, so long as there isn’t much writing to it, it should be fine.
Reports of SSDs write span issues have been greatly exaggerated ;)
At least nowadays even with QVOs thats not something consumers have to think of much anymore.
These SSDs have 8TB, so to exceed its write endurance Jeff would need to write 4TB to all of them each day for 3 full years.
How many full-drive writes does your average video editing server need? I would expect a pretty small number. The average source file is sitting there for weeks or months.
For a use case like database transactions, log storage and frequent data dumps, the game changes quite a bit. I would definitely shy away from the QVO drives for that use case.
I've had these drives in service for about 8 months in my regular NAS, before transferring them to this new build, and they are all checking out okay still.
But this is also why I'm doing the striped mirror plus a hot spare. The only real challenge would be if the drives have a firmware issue, and they all die at the same moment after like 4 years due to a bug (like how HN's servers died...).
But copper is usually a little simpler for consumer/prosumer devices. Someone does make a Thunderbolt to SFP+ adapter but that things like $300!
Deleted Comment
At 500x whole-drive rewrites I think I got my money's worth out of an $84 1TB drive.
That doesn't make sense. The author is not using raidz. In ZFS terminology, he's mirroring two striped vdevs (like raid10), plus using a hotspare. And that is a bad choice, as he gets only 16 TB usable, and the mirrored stripes could fail with certain combinations of two drives failures.
Instead, he should have set up a raidz2 across 5 drives: 50% more usable space (24 TB), can tolerate any two drive failing, and it would give him higher performance on sequential I/O.
I have a raidz2 pool across 6 spinning rust 18TB hard drives, and my server can handle 900 MB/s sequential reads, and 700 MB/s sequential writes (benchmarked locally). It it was built on 5 SAMSUNG 870 QVO SATA III SSD drives like the author, it would certainly get to 1.5+ GB/s in both sequential reads and writes. In other words the bottleneck would shift to the 10 Gb/s network (1.25 GB/s). For comparison the author discloses his writes are limited to 700 MB/s (over the network, so his bottleneck is not the network link but local I/O contention).
That's right, a 6-HDD raidz2 matches the write performance of his 5-SSD raid10 setup. And that's because the write overhead of raid10 is 100%, while the write overhead of a 6-drive raidz2 is only 50%.
Edit: Oh and the strange performance issue that he noticed "disappeared" after a while is most likely due to cache recovery. The drive can sustain writes of about 490 MB/s for a little while then it drops to 170 MB/s, but after idling 5 minutes it can recover the initial speed. See https://www.tomshardware.com/reviews/samsung-870-qvo-sata-ss...: "As we noticed with the 1TB model, the 8TB model’s cache recovery mechanisms work similarly. After letting the drive rest at idle for 30 seconds, the 870 QVO gains back 6GB of its cache. It recovers fully with 5 minutes of idle time. "
One concern Wendell had with RAIDZ2 was the potential for the older Xeon to be a bottleneck for writes. It probably wouldn't be, but I didn't do too much testing with that layout. I might still, we'll see.
For a video editor, at least one that's been around long enough to remember DAS and SAN solutions, $4300k for 40TB of edit capable storage is cheap.
Perspective is everything.
Or is it one of those things that would be cool to have but not really necessary?
For installing outside of a machine room/closet/center, if you're using 2U of height, you might also fit a PSU with a larger and quieter fan, since all the Flex PSUs I've had come with noticeably loud fans. (I replace them with Noctuas, but it isn't a fun kind of soldering, IMHO.)
The components from the build would also fit in a Supermicro 1U short-depth chassis, especially if you can go a little deeper in your cabinet. (My new K8s server got a used Supermicro 1U chassis for ~$60 shipped, including a PSU. In the photo on https://www.neilvandyke.org/kubernetes/ , it's the 1U immediately below the 4U.)
[0]: https://github.com/openzfs/zfs/issues/8381
[1]: https://github.com/openzfs/zfs/pull/10018
If your IO all goes over a 10 Gb link, that will be your bottleneck before ZFS is.
Since 1 MB record size was used, this may be a candidate for a bunch of mirrored HDDs with a few TB of nvme for l2arc and ZIL. But with a bunch of SSDs already on hand, why bother?
Do you really need to save all of that footage? I would think keeping the pro res footage for the current projects on the workstation and reencoded archive video on a NAS would be sufficient. I'm not a video professional, but I suspect it's easy to fall in the trap of thinking that you need to save everything in highest possible quality in case you need it later, but what are the realistic chances of that? If you end up needing some old footage again, AV1 coded 4K or even HEVC 1080p would probably be just fine. The final result are Youtube videos after all.
I know he mentions editing from it, but that's enough space for more than a week of pro res video.
I am a video professional, and you always keep your originals. However, the part being left out of the conversation is that as a storage pool for editing goes, this isn't deep storage. Content is typically only left on the edit storage while the project is active. Even the TFA mentions he goes all the way to cloud cold storage.
Edit storage has always been expensive, and maintaining capacity was always a juggling act. Just because terms like NAS are being used, one should not think of this type of storage as a dump it and forget it type of storage.
There are many levels of professional. On the high end, the footage from the camera is copied to multiple hard drives on set. These are the backups, and us old timers still use the terms camera originals. These are as sacrosanct as film negatives, only there's magically multiple copies. This data gets transferred to the editor's storage. Once the edit is completed, the edit session and other content used in the edit may be transferred to the camera original drives for archival. On the other end of professional, you take the SD card out of the camera and transfer directly to the edit storage. In these situations, woe be unto thee that doesn't make proper backups. At that point, it's really more pro-sumer than professional, but hey, if they're getting paid, they're professional enough.
And yes, I did see the part about editing, but are you really editing nine days worth of footage at once? Would it make more sense to put less but faster storage directly in the workstation?
Don’t take this the wrong way, I’m genuinely curious… what is a video professional doing on HN? Was it a past life? Is tech just a side interest? I see many non-tech professionals on here (doctors lawyers too) and I’m always curious what drew people in!
On the flip side, I think it would help a lot a lot a lot if there was more discourse out there about compressing high-dynamic-range content. AV1 supposedly has some capability to do a good-ish job with HDR. The idea of taking reels of raw video and spending a couple days squishing it to 1/10th the sizes but preserving the quality/flexibility very very well is a value proposition that I dont think is clearly attainable, even though it seems technically perhaps within reach.
Right now, I think the general feeling is, raw is raw & everything else bakes in a vast amount of assumptions & constrictions. Some advocacy that compressed video can be as flexible, as dynamic, as capable need to be more present, elaborated, & proven before anyone's going to be comfortable throwing away the bits. These are people's life's works & a couple hundred or thousand dollars a year more in storage isnt a real factor for such integral, near work.
Have you seen Jeff's videos? They're mostly him talking to the camera in what I think is his basement, or closeups of various PCBs. If you're running a stock photo company I get wanting to keep footage at max quality, but once your Youtube video is published, do you really need to save all those alternate takes and cut out bits? I just don't see them being very useful in the future, and if you do end up needing some of it for B roll or whatever, would a couple of seconds of compressed video really make a drastic difference for the project as a whole? Especially since the final result ends up on Youtube and consumed on a phone screen or a TV two meters away.
Much of the content that's produced today is not trying to be timeless classics with endless rewatch value, it's ephemeral and only really relevant for a short time. If you can reupload your videos to other video hosts in the future that's probably good enough. Nobody is watching reviews of three year old Raspberry Pi add on boards for the low noise in dark parts of the image, or the accurate color reproduction, or even the 4K instead of 1080p resolution. Kill your darlings!