https://github.com/chapmanjacobd/syncweb-py
https://github.com/chapmanjacobd/syncweb-ui
Unfortunately, mesh storage systems are very different conceptually so it is difficult for people to think about permissions and access. You can bolt on something familiar but then it really limits the usefulness of mesh storage and you may as well just be using HTTP servers.
unless tape, and the infrastructure to support it, is dramatically cheaper than disk,
This turns out to be the case, with the cost difference growing as the archive size scales. Once you hit petascale, it's not even close. However, most large-scale tape deployments also have disk involved, so it's usually not one or the other.That mail server used maildir, which...for those who are not familiar: With maildir, each email message is a separate file on the disk. Thus, there were a lot of folders that had many thousands of files in them. Plus hardlinks for daily/weekly/whatever versions of each of those files.
At the time there were those who were very vocal about their opinion of using maildir in this kind of capacity, likening it to abuse of the filesystem. And if that was stupid, then my use of hard links certainly multiplied that stupidity.
Perhaps I was simply not very smart at that time.
But it was actually fun to fit that together, and it was kind of amazing to watch rsync perform this job both automatically and without complaint between a pair of particularly not-fast (256kbps?) DOCSIS connections from Roadrunner.
It worked fine. Whenever I needed to go back in time for some reason, the information was reliably present at the other end with adequate granularity -- with just a couple of cron jobs, rsync, and maybe a little bit of bash script to automate it all.
If you ever need to do something like this again, it's often faster to parallelize rsync. One tool that provides this is fpsync: