Readit News logoReadit News
sandreas · 3 years ago
I don't trust Time Machine any more. Years ago I wrote some shell scripts that help me mostly automate my complete system setup (with brew and friends). From time to time, I wipe my entire system and restore it with these scripts. Sometimes I have to adjust them, but mostly, they work without changes.

For my data backups, I use restic. Big advantage is, that I can read my backups even when I don't have a macOS system present (e.g. my only macOS system had a hardware issue and my Time Machine Backup was pretty much useless until I got a new one).

I know, this solution is not for everybody, but Time Machine corrupted my backups more than 5 times now and it feels so slow compared to restic, that I don't even think about retrying it after a new macOS release any more - even if my solution is a bit more work to do.

microtonal · 3 years ago
People who want something similar may want to look at Arq [1]. Similar to restic, it provides incremental encrypted backups to most cloud providers (or a machine with SSH). But it is a Mac app, making it easy to configure and maintain. I never had issues with data corruption so far.

Disclaimer: I am not affiliated with them, just a happy user for 9 years.

[1] https://www.arqbackup.com

bartvk · 3 years ago
Seconding Arq.

It's incredibly freeing to be able to dump your remote backups wherever you have space. Right now, the best for me is to send them to a virtual Linux box where I have enough space left.

It supports Amazon Drive, AWS, Backblaze B2, Dropbox, Filebase, Google Cloud, Google Drive, OneDrive, SFTP, SharePoint, Storj, Wasabi, network share or some other random S3-compatible server.

It's a native, let me repeat that, NATIVE Mac app and behaves exactly like one. Incredibly satisfying. Unlike Time Machine, it'll tell you what went right and what went wrong. I have a lifetime license.

sandreas · 3 years ago
I'm more into Open Source, even if Paid solutions are easier and more reliable. But hanks for the suggestion.
dreamcompiler · 3 years ago
There was a time when Time Machine was pretty good. But Apple became increasingly parochial about TB backups such that you cannot cull them even with root access. Any culling must be done with the TM app from the original machine that owned the files. This makes it impossible to be the sysadmin for your family's backups.

Now I use Carbon Copy Cloner, Syncthing, and Arq. My family's backups are quicker, more seamless and much easier to administer as a result.

6ak74rfy · 3 years ago
I was trying out Restic (through auto-restic) this weekend. And I really wanted to make it work.

However, I have 2 user accounts on my macOS (mine and spouse’s) and I couldn’t get Restic to access the other account’s data. I am admin, I ran Restic as root, gave it “full disk access” - still couldn’t make it work.

Any tips?

sandreas · 3 years ago
Why not use 2 repos?
pixiemagic · 3 years ago
Would you be able to share those scripts? Or if not, an obfuscated/abstracted version of them?

I use brew but even with brew and brew cask, I find there's plenty of GUI apps that require manual installation, which will also have their own config spread out in various places, plus whatever CLI or background stuff I've installed myself (mostly hidden files/directories in my home folder so probably easier), system settings, etc. I don't know how I'd reasonably automate a restore of all of that stuff without having a Time Machine-style backup (essentially a disk image).

sandreas · 3 years ago
Unfortunatly, they contain too much sensitive information to provide them online and I don't have the time to make them nice and clean (it all started with a 50 lines bash script). But there are plenty of users providing macOS RICE / dotfiles...

Just a short search:

https://www.lotharschulz.info/2021/05/11/macos-setup-automat...

https://gist.github.com/bradp/bea76b16d3325f5c47d4

https://www.youtube.com/watch?v=eFlnqSqkTas

Tsiklon · 3 years ago
Is your Time Machine target HFS+ on CoreStorage, APFS, or SMB?

Have you experienced the same corruption on the APFS version

Deleted Comment

tailspin2019 · 3 years ago
What destination(s) do you backup to with Restic? I've been thinking about giving it a try.
CTDOCodebases · 3 years ago
I use Restic and backup to rsync.net. The only reason why I use them is because they use ZFS and have a admin/user account system that allows for immutable snapshots once everything is set up.

Honestly though it doesn’t matter. The backups are encrypted outside of the PC you are using Restic on. You just need a drive or a PC with storage space and SSH access that you can access.

0cf8612b2e1e · 3 years ago
Just start with an external usb drive. You can pickup 8tb for under $200. If you want to go the extra step, you can buy two or three and physically swap them every N weeks. Store the older one at a different location.

Far less sophisticated than a real NAS, but also fewer moving pieces and things to go wrong.

sandreas · 3 years ago
My home nas and via zfs send the off site nas from my parents.
mfashby · 3 years ago
Local NAS and backblaze. It works well.
gotschi_ · 3 years ago
Keep brew far away from your prod system. Everything you can‘t do in macos just use a vm. Don‘t touch the system and Timemachine will never corrupt, its that easy in my experience.
misnome · 3 years ago
But… brew doesn’t touch the system, and never did?

Even when it used /usr/local (before the root system became hard readonly) that was an unused folder path, and it explicitly refuses to install as root.

uselpa · 3 years ago
FUD. brew has nothing to do with Time Machine corruptions.
scorxn · 3 years ago
I have a Raspberry Pi running Pi-hole and Time Machine, making that little board probably the best value piece of tech I've ever owned. https://saschaeggi.medium.com/use-a-raspberry-pi-4-for-time-... is what I followed.
tverbeure · 3 years ago
RPis are great hobby electronics projects, but IMO they don’t have any benefit for this kind of use over a thin client?

They’re more expensive, you need an extra RPi enclosure and power supply, you can’t use M.2 SSD sticks without an additional USB3 enclosure, they have a history of corrupting flash cards, and their idle power is only a little bit lower.

squarefoot · 3 years ago
Can confirm about MicroSD flash cards, they're too unreliable for stuff that must be on 24/7, and get corrupt easily. Good for tinkering projects and media players, not for critical stuff where the downtime needed to pull the card and fsck/reinstall it might not be an option.
scorxn · 3 years ago
I feel ya. My RPi does indeed have a case, with a fan. The backups are offloaded to an external HDD.
satysin · 3 years ago
I use a Synology now but I used a Pi for a couple of years before that without issue.

I did give it to my daughter to use at university but I very much doubt she actually used it and I've not thought about asking until this post :^)

yjftsjthsd-h · 3 years ago
That seems pretty similar to this, just swapping out the thin client and pi (and the exact Linux distro, if that matters). Which makes it a trade in all the particulars:), depending on exact price, power use, etc.
climb_stealth · 3 years ago
My advice to retain your sanity: stop using Time Machine and use Carbon Copy Cloner [0] instead. It works. It keeps working. It has excellent documentation for any possible backup and restore cases. It is transparent about what it is doing.

Time Machine works fine until it doesn't. And it won't tell you that a backup is broken until you try to restore from it. The errors are going to be cryptic. There is going to be no support and the forums are not going to help. The broken backup is not going to be able to be repaired. Time Machine uses the "fuck you user" approach of not providing any information about what it does, or doesn't, or intends to do or whatever.

If your data is worth backing up, don't use Time Machine.

[0] https://bombich.com

crossroadsguy · 3 years ago
> Time Machine works fine until it doesn't. And it won't tell you that a backup is broken until you try to restore from it. The errors are going to be cryptic. There is going to be no support and the forums are not going to help.

and

> Time Machine uses the "fuck you user" approach

apply to every piece of software or service out there by Apple. Every single one!

iCloud? The only answer you’re ever going to get anywhere is - “No, your data is getting synced”. “Why don’t you enable and disable it”. Or “restart it”. My favourite is when Apple Support asks you to reinstall or reset the iOS or reinstall macOS entirely even if something as tiny as some sync problem occurs, with such confidence and matter of fact tone that you are forced to debate whether you are talking to some kafkasque and sadist bot.

This is the Apple playbook. Naming and shaming them on social media doesn't work. Emails don’t get answered. Your customer request gets stonewalled. Sometimes you feel maybe it’s Apple paying you to use their products and not the other way round.

An ex manager was thought to be crazy by us freshers when he would say that the moment he gets a Mac, for personal or work usage, the first thing he does is install Linux on that. We used to think he was some GNU/FOSS nut. He would smile and say - give it time, you’ll understand.

And I understand now how helplessly hostile the Apple ecosystem is. You literally feel like a hostage — you can use it with such stonewalling and infuriating limits and you start feeling this is the only way!

At this point I don’t think it’s anything other than self sabotage if anyone still says

- but I use Time Machine (or any XYZ Apple thing that involves data integrity and reliability) - it works, or is “good enough”

- or relies on something like iCloud for something like “data backup and integrity” in anyway.

Can’t wait for the day when file access is easier on iOS (of course aft explicit permissions - which is the case even today actually) etc, or Android sucks less - in this glorious duopoly. I meant when they’re forced to open up their platform, because they’re not going to do it themselves!

ksec · 3 years ago
I mean Apple was never perfect. But I could at least understand that, in the old days they were nearly bankrupt, they had comparatively speaking extremely limited resources. And they had to focus on certain things and neglect others. iPhone arguably took all their resources from 2005-2015. But somewhere along that line they had way more resources than they need or know how to spend on it. Some of the lacking attention and details no longer makes any sense. And not only have to not gone to fix it. Their quality has gone downhill.

It is sad. You would expect a bigger Apple to be a better Apple. But almost like everything in life they are counter intuitive.

climb_stealth · 3 years ago
Hah, I think I hit a nerve there. I hear you. Overall I'm actually super happy with my Macbook. Luckily for the most part no one is forced to use the Apple eco system.

CCC for system backups. Tresorit for backing individual directories with a bit more privacy and flexibility than Dropbox.

I don't think it's self-sabotage to keep using apple software, but more a case of people not having been in the situation where things go wrong and there is nothing you can do :)

vGPU · 3 years ago
Agreed. Have a 14 MBP. Nice hardware. MacOS makes me want to scream and tear my hair out.
uselpa · 3 years ago
I agree on iCloud, which behaves in a very erratic and intransparent way. Don’t rely on it. I use Nextcloud instead and it works much better. One software that syncs extremely well with iCloud is Obsidian, for reasons unknown (to me).
smcleod · 3 years ago
CCC costs $77.50 AUD per major version just to use the app - it might be OK but that's a lot of money!

TimeMachine is free and "good enough" for local / local network backups for most people, for remote backup BorgBase (although the "Vorta" borg GUI app is dreadful) and Backblaze are affordable-ish options.

Also it doesn't look like CCC has write-only backups (immutable from the client side once created) like Borg has, there's no mention of encryption or deduplication in their feature list either - https://bombich.com/features

alwillis · 3 years ago
> TimeMachine is free and "good enough" for local / local network backups for most people

If you backup to an APFS-formatted network share, Time Machine is quite good and much more reliable than it was when backing up to HFS+ volumes. There was a lot of hackery going on to make HFS+ act like a modern file system emulating snapshots, which it didn't have. You could end up with millions of hard links; turns out that was kinda fragile…

APFS supports snapshots and a bunch of other features that makes Time Machine faster and more reliable.

climb_stealth · 3 years ago
Yes, it is pricey. But you get what you pay for. The attention to detail in the documentation and patch release notes from Bombich is on point.

Zero regrets from my end about the money spent for it and it is one of the few software products where the price tag seems justified.

FWIW I first bought a license in 05/2019. The only major version update since then has been in 07/2021. And the license for the upgrade cost $30.47 AUD. They don't seem to be exploiting version updates as a form of revenue generation.

And yes, I'm sure there are backup solutions that work better in different scenarios. My main point is that Time Machine is rubbish for full system backups. CCC is polished, powerful, robust and power user friendly and one option that I have been super happy with.

earthnail · 3 years ago
A lot of money… until you need to recover data and would pay SO MUCH more just to get your data back.

I think it’s a fantastic price.

rcthompson · 3 years ago
Can I ask what you find inadequate about Vorta? I've recently started using it and it seems perfectly functional to me (although it's probably not an app I'd recommend to someone who doesn't already understand borg).
prmoustache · 3 years ago
I don't think backup is a place where you want to go cheap really. Also 80€ every couple of years should be nothing when you can afford a Mac.
rhaway84773 · 3 years ago
It’s the cost of a Mac.

You’re probably spending upwards of $1000 USD to buy your Mac. You’d might as well factor in the one time (or maybe two time if you need to upgrade the major version again during the lifetime of the mac) purchase of a backup solution into the cost of your Mac.

Because relying on Time Machine basically means your computer is a ticking time bomb just waiting to lose your data.

js2 · 3 years ago
Over the years, I've used both CCC and SuperDuper in order to have bootable backups. I've always used these in addition to TimeMachine though.

I currently use TM to both my TrueNAS server and to a local disk (when I remember to plug it in). I use Arq to B2 for my home directory.

For my family, I have them use Backblaze. It's the only thing I've found that's really set and forget. They never remember to plug in a local TM drive. And setting up TM with a network drive, it flakes out every few months, then they just ignore the messages from TM that their drive isn't being backed up.

Backblaze pretty much Just Works. And it sends me an email report once a week.

A full restore via Backblaze would be a bit painful. But it's better than no backup at all.

torstenvl · 3 years ago
Most Time Machine weirdness comes down to metadata mismatches after migrating from machine to machine.

I made some fairly simple shell scripts to help fix these issues.

https://github.com/torstenvl/tmutils

They're pretty beta so if you use them please be careful. There are confirmations for major metadata changes, and the dirdedupe script runs in test mode by default (you have to use the --execute flag if you want it to actually do anything).

Eric_WVGG · 3 years ago
I also use CCC — in addition to Time Machine, running on a tiny little SD card sitting flush inside the laptop. https://www.bhphotovideo.com/c/product/1687325-REG/transcend...

CCC is great, but it doesn’t back up in near-real time and doesn’t do versioning. It wouldn’t have saved my butt while I was on the road last Friday like “built-in” Time Machine in my SD slot did.

yurishimo · 3 years ago
How reliable do you find it? I suppose if it's just for catastrophic emergencies, it will probably work through needing to use it. I'm guessing it will fail to write before it fails to read.
msephton · 3 years ago
How does this work with mounting/unmounting, power cycling, etc.
krackers · 3 years ago
Carbon Copy Cloner has a mode to periodically check backup integrity, and will alert you on mismatch. Given that unlike ZFS, APFS doesn't actually checksum user data, it's worth it for that feature alone. If you use ZFS for your backup destination, then this effectively ends up giving you verification of your non-ZFS source as well. And even if you don't backup to ZFS, then you still get notified that corruption was detected, allowing you to triage and take action.

I think it's CCC5 or CCC6 that actually exposed the list of file mismatches to the user UI, but the diffs were greppable in the diagnostic logs all the way back to CCC3.

firecall · 3 years ago
My advice is to use TimeMachine plus other backups.

I use:

1. TimeMachine on a 20TB TrueNAS ZFS setup

2. BackBlaze

3. ...not a versioned backup, but iCloud Synching, which is a useful disaster recovery.

Brajeshwar · 3 years ago
I've found that the ideal size of Timemachine drive is just about twice of the target drive. It last for months if not years -- enough to go back. Beyond that, there must be an archival/backup system in place if you still want to get back files deleted years ago.

Deleted Comment

dddddaviddddd · 3 years ago
> Time Machine works fine until it doesn't.

I experienced this a couple years ago when I somehow had a directory name with a UTF-8 byte order mark, causing mysterious Time Machine failures: https://www.davidschlachter.com/misc/timemachine-encoding-no...

Brajeshwar · 3 years ago
I use Carbon Copy Cloner to have a replica of my drive. However, I still use Time Machine as a convenience to go back in time for my mistakes. My daughter thinks I'm one of those nerds that married a regular girl from college and can do magic with software because I can make her computer go back in time and bring back versions of her files.
908B64B197 · 3 years ago
> My daughter thinks I'm one of those nerds that married a regular girl from college and can do magic with software because I can make her computer go back in time and bring back versions of her files.

Wait until she discovers git!

As an aside, I wish there was an alternative like Time Machine on Windows.

telcy · 3 years ago
> Time Machine works fine until it doesn’t

I have been backing up to Time Machine for years without issues. Recently I started using pnpm (Nodejs package manager) and the backup process hangs on Mui material icons files (@mui/icons-material). Using "sudo fs_usage -f filesys backupd" I see that it is looping over the same files over and over.

nabeards · 3 years ago
I have found that Asimov alleviates this issue.

https://github.com/stevegrunwell/asimov

I wrote a tiny blurb about it here:

https://nabeards.com/time-machine-backups-ignoring-npm-modul...

tverbeure · 3 years ago
In addition to Time Machine, I use iCloud for generic Mac apps, Backblaze for real disasters, and GitHub for all my personal projects. So my backup needs were already mostly covered without TM.

Though the one time I needed a full SSD restore, Time Machine was the fastest way to do it, and it worked like a charm, after literally 8 years of not looking at it.

But if I want to switch to CCC, you run their client app on your Mac, but how do you set up the server? Is it just a Linux NAS running Samba?

ChildOfChaos · 3 years ago
It's not really an issue though.

You just restore the OS without the time machine backup and then you browse the time machine backup manually and copy the files you want. It's usually just a random file it can't read correctly in it's index so it breaks the whole thing, it's not like the whole backup is useless, you can manually browse all the files and you are still good, not sure how that is the 'fuck you user' approach.

Shawnj2 · 3 years ago
CCC is great although it’s not quite as useful as it is on Intel Macs on AS ones because the whole paradigm of “clone the drive onto a USB and boot it” doesn’t work.
pazimzadeh · 3 years ago
what about on apple silicon macs?
e40 · 3 years ago
I had a weird issue with ccc and the dev and I couldn’t figure it out, so I switched to TM. I also use backblaze and local rsync’s.
s3p · 3 years ago
Can I use CCC to backup to something like an AWS bucket of Backblaze B2 if I don't have a NAS in my house?
blacksmith_tb · 3 years ago
Generally I think the use case is "duplicate the boot volume of my machine onto an attached usb drive" though I think these days not all Macs will boot from an external drive? I have SuperDuper doing scheduled backups like this, along with Backblaze for offsite (but not so bootable) backups.
climb_stealth · 3 years ago
Maybe if the location can be mounted as a network drive. But I'd say it would be making it do something it is not intended to do. I'd say it is really meant for backing up to external storage or drives on the local network.

There are probably better tools for this.

Jedd · 3 years ago
I note that $25 got the author 16GB of storage - another $70 was spent on boosting that to 2TB.

For GNU/Linux users - but will obviously work just fine for Microsoft and Apple clients - a good combination is SyncThing on local + remote mini-server, with BorgBackup running on just the server side.

SyncThing gives you near-instant synchronisation, and BorgBackup gives you periodic archives (at your periodicity & retention preferences).

For isolation purposes, I went for a small-footprint VM for each family member, and instructed that anything they really care about needs to go into ~/work/ or ~/private/ to be looked after.

puzzlingcaptcha · 3 years ago
You can often get those for as little as $10 these days, I paid $25 in 2020.

Interestingly the APU supports AES-NI and has a little crypto accelerator for SHA (unlike RPi which didn't splurge the extra dollar on Armv8 Cryptography Extensions).

I'd recommend going for t620 over the t520 for double the cores at the same idle power draw (6.5W).

If you are fine with just two cores Fujitsu Futro s520 is also great.

https://heap.ovh/tag/thin-client.html

st3fan · 3 years ago
I’m not sure if AFP (via Netatalk) is really the right thing to use. I’m pretty sure that “native” Time Machine now prefera CIFS (Samba) over the network.
hoherd · 3 years ago
Correct.

> If given the choice between SMB and AFP, use SMB to back up to your external backup disk.

https://support.apple.com/guide/mac-help/types-of-disks-you-...

mustache_kimono · 3 years ago
> I’m not sure if AFP (via Netatalk) is really the right thing to use.

For years, Netatalk was still the more dependable solution, despite the deprecation warnings. I've had very good luck re: my current set up (ZFS and Samba 4.15.13 on Ubuntu 22.04) but Netatalk I believe still works very well.

donatj · 3 years ago
Agreed, I still generally have much better luck with Netatalk which is why I have kept using it all these years.
tverbeure · 3 years ago
Thanks! I've added a warning at the top of the blog post, and will revisit next weekend.
pcurve · 3 years ago
"Don’t worry, I also use BackBlaze for automated off-site backups. And all my projects are stored on GitHub and different PCs as well"

That's smart. I've had 7 catastrophic drive failure in 20 years, 4 when I was using homegrown backup.

Due to my being sloppy / lazy, I still managed to lose a lot of data in 2 of the instances.

Homegrown backup solution isn't for me. :)

saagarjha · 3 years ago
Ok so like this seems to have been written today but I think it’s notable that the Mac they showed in the post is running a five year old version of macOS: I don’t actually think this is a good idea anymore, and I say this as someone who ran a Raspberry Pi in a similar configuration for a while.

Today, if you’re using Time Machine, you should back up to an APFS volume with snapshots. They are far faster and more reliable than HFS+ disks. I get the feeling that macOS might at some point stop supporting new HFS+ backups altogether. So, I wouldn’t recommend making them anymore.

Unfortunately, APFS drivers for Linux kinda sorta don’t exist. Not to the point where you’d want to rely on them for backups, anyways. So the move here is really to just get a Mac to handle this. My network has an old MacBook Pro (one of the last pre-Touch Bar ones) wired into the backbone for this. It’s pricey but believe me, this is the setup you want.

(By the way, if you’re using Time Machine to a network share, I recommend keeping another “worst case” backup that you refresh periodically. The network disk is convenient which is good for having periodic backups and quick access but from time to time this setup is liable to Time Machine corrupting itself in a way it doesn’t know how to fix.)

newaccount74 · 3 years ago
When Time Machine backups over the network, it creates a disk image and writes to that. So I don't understand why you would need APFS drivers on Linux?
tverbeure · 3 years ago
Yeah, I wrote this on a 2012 MacBook Pro running Mojave. My replacement MacBook Air 15” will be ready for pickup in a week. :-)
bombcar · 3 years ago
You can still do this setup (I have it on Gentoo) but I also have a local SSD for when I’m docked. It alternates between them.

Nowadays it’s done with some special extensions to samba instead, but it works the same.

fastaguy88 · 3 years ago
Another cheap networked Time Machine solution is a used Intel Mac Mini (~$120 for 2.5 Ghz Intel i5, 8 GB Ram, 256 GB SSD), and set it up to export your time machine disks over the network. Not only is it "real" Time Machine, it allows you to add to an existing Time Machine disk that used to be plugged into your machine. No need to learn anything new.