It is interesting how often I see companies that refuse to make back-ups of their cloud hosted data on the assumption that this is now someone else's problem. I also have a - recent - case of a very large manufacturer of storage solutions that managed to fuck up a restore of a raid array to the point that a whole pile of companies lost their data.
Backups are so simple, and yet the only times people seem to realize their true value is when they don't have any.
I am not arguing the importance of backups, but tbh, simple is not a fair statement.
At the very least, if you have backups, you might have the ability to rebuild/redeploy, but for any non-trivial system, without a comprehensive DR plan that is drilled regularly, even having backups won't save the business if the tooling to get those backups into a production ready state.
It gets more interesting when you're trying to backup a distributed database. When your durability strategy is to spread your risk across a lot of nodes in a cluster, and you're federating your cluster across multiple sites, traditional backups may not be a practical option. However, as I write this, I'm realizing that there's nothing simple about distributed anything, and saying "it's hard to..." may not be all that interesting.
Disagree. A high-confidence backup plan that includes continuous restoration is of middling difficulty to design. Many get backups wrong and only find out when their archives turn out to be incomplete.
It's also a major flaw of the cloud and SaaS providers. They don't encourage customers to maintain local (or alternate-cloud) backups, because they refuse to admit the appearance of weakness.
If a Cloud/SaaS provider is going to claim to be your hassle-free one-stop shop, shouldn't they provide, as a matter of course, a feature to dump your data daily/weekly/monthly to a designated off-cloud site of your choosing? But they don't, because they want to pretend that they are impenetrable and 100% available.
Backups are NOT simple, especially when you have things like GDPR in play. I'd imagine companies can't afford to commission backup that is compliant with the regulations. If you spend on a backup system, your company goes down in debt, you don't - there is a chance you go down anyway.
QuickBooks Online does not have a native backup or restore, at all, and it's pretty ridiculous. QBO told me they also can't do on-request restorations with their backups.
There's a third-party or two that can use the API to do a backup/restore, but also, still, not all of the QBO data has an API. Eg, IIRC, recurring transaction tasks.
Just a matter of time for ransomware to replace data via APIs.
> Just a matter of time for ransomware to replace data via APIs.
I asked about this in a cybersecurity meeting at an investment bank a few years ago - how much security is there around Bloomberg's data feed? All of the algorithms that make million & billion dollar decisions rely on that data so I asked if someone stepped in the middle and fed the algorithms bad data or even slightly delayed data would we even know it had happened? They looked at me like I was a crazy person. I'm actually kind of surprised it hasn't happened yet. Or maybe no one has noticed yet.
This is a question I've thought alot about, so many some Sys Admins can give a good idea about how to approach it:
How do you create a backup server that is reachable by production servers (so that they can back up to it) without then being vulnerable to the same kind of ransomware attacks that infect the production servers? You can't exactly make them read-only, or else they can't accept the "legitimate" writes that might occur during the normal backup process.
One absolutely diabolical mechanism that was used (at least 5 years ago when this scourge of ransomware started to rear its ugly head) goes something like this:
1. Gain access to change the code on the front-end web servers (usually PHP)
2. Change the database access layer to transparently encrypt data being written to the database, and decrypt data being read from the database. The key would be loaded into memory by curl'ing an attacker-controlled website at startup.
3. Wait 30 days
4. Notify the company that they're compromised, turn off the attacker-controlled key service, and restart the web front end
Now step (3) ensures that most data in the database has been re-written, and if your backups are dumps of the production database, you now have a month of encrypted backups that you can't read... If you're lucky, you may have a month-old backup to restore from; if you're unlucky, you rotate every 30 days.
It's nice to have a restore step in there too, so one has both a validation that the backup is usable, and gives a "playground" where one can have a safe day old environment for testing/training/whatnot against production data.
Also, backups must include a physical process of moving a copy of backups to an airgapped secondary system (a human ejecting disks/CDs/tapes and carrying them to another storage container), so that it's impossible for an attacker to compromise the backups via the same software exploit that corrupts the primary data.
Two separate S3 buckets with different credentials. The server pushes its own backups to the "quarantine" bucket and later a process moves the backup to the long term storage bucket.
I’m surprised this answer is so far down. Production machines write to an S3 bucket with a lambda trigger. As soon as the upload is finished, the lambda either lifts the file to another location or changes the acl so the production machines can no longer access it.
Two separate S3 buckets with different credentials
This gives me the heebie-jeebies.
It seems better to me to have one on Amazon, and one on another cloud provider entirely. Plus regular sneakernet archives, if your data is of a size that would permit that sort of thing.
I'm fortunate that this kind of thing is handled by other people where I work. It's my understanding that we have production data on one service on the east coast, a mirror with a different company in the south, and weekly backup archives stored offline on the west coast.
I use a single bucket with versioning enabled. Production server has access to write as it pleases, but isn't allowed to touch past versions of a file.
ZFS Snapshots are really handy - it takes root access to remove them and you can't delete them with the normal UNIX delete. IF you're sharing with Samba or NFS, there's no way for any remote user to be able to delete.
That is what I do. The backup server can access the production server, but production can't access the backup. But my setup requires the backup to go to disk on the production server and not stream out directly, so that aspect is annoying.
The easiest solution to this is out-of-band backups happening at the storage layer. The details depend heavily upon your platform and storage hardware, but most SAN and virtualization solutions today allow one to backup hosted systems without being accessible by those systems (and no agents installed into the guest OS).
You pull from the production servers rather than push to the backup servers. That way even if the production servers are compromised your backups are still safe.
Unless you have persistent mount to network FSs like NFS or SMB how do you think the ransomware would spread? You sure don't need network mounts for backups.
Cronjob to an (S)FTP server and an upload script trigger to chown/chmod all incoming files making the whole thing WORM (Write Once Read Many).
Once its submitted the same user account can't alter it. Even if the malware is clever and scans for .netrc and .id_rsa and manages to create its own connection to the backup server it doesn't have access to anything anyway.
I'm not a sysadmin, and I'm pretty sure that <big company> does it different, but here is how I do it at home:
- I have a UnRaid machine, and a backup machine. The backup machine is a small itx board, and has a single HDD attached.
- A NodeRED instance has a so-called "Flow" on the UnRaid machine that is waking up the backup machine every 7 days.
- Thanks to anachron, with a 10 minute delay, rsnapshot connects to the UnRaid machine, pulls the data, and then issues a shutdown to the backup machine.
Drawbridge, you configure a firewall to allow access only at certain times. Also you can configure a NAS so that deleted items are preserved without a admin removing them.
I use FreeNAS (FreeBSD NAS with ZFS). The ZFS snapshotting feature effectively gives me protections from this type of issue.
My home computers push backups regularly throughout the day, and every day I create snapshots of each volume (how long to keep the snapshots is another question). This snapshot can only be accessed or managed on the NAS itself.
This effectively creates an append-only backup NAS thanks to the periodic snapshots.
In the past I had the backup server ssh via public key into the production server and pull the data via file system snapshots. It’d be a matter of restoring the most recent snapshot prior to the attack.
You can write backups to an NFS share, and create regular snapshots on the file system server that are not writable from the outside (you have to log into the file system server, which hopefully is extra secured / limited access, and has a different OS than what you use elsewhere). It's not perfect, but at least a single vulnerable client can't encrypt or delete your backups (in the snapshots) that way.
As usually with security, the principles of least privs and segregating as much as possible are important.
Make S3 bucket on a different account, grant it write privileges from your main one. Enable versioning and setup lifecycle to purge old versions in 90 cays. That’s it - no matter what your main account does, you will bave 3 months to undo it
Backup could (should?) be on tape, which once written is physically removed from the system and put in cold storage for some period of time (years?). That's how we did it at my first job, at least.
Of course IANAL so idk how this jives with various EU laws.
Also how do you prove integrity of your backups if you don’t have certainty of initial perhaps temporarily dormant breach? I mean your backups to tape could still have bad data.
I had no idea this was how QuickBooks even worked in the cloud. Could Intuit be in any way liable (like is this a sort of franchise type of thing)? Or is it closer to Word Press where different companies can install "QuickBooks Cloud" and then offer it?
Think “Microsoft Access in the cloud”. How would you do that without Microsoft’s cooperation? Virtual machines running Windows. Full service hosting, the works. This is classic ISP stuff, back when ISPs were service providers and not just Internet connections and other off the shelf no-touch products.
I’m not familiar with the branch of synesthesia that detects similarity as electrical impulses, but you’re welcome to email the mods and ask them to delete my post and give you the karma. I don’t know if they’ll do so or not but I certainly have no investment in the integer (as my most recent post shows).
It's not immediately obvious, but iNSYNQ provides hosting for QuickBooks Desktop as a virtual desktop service (think VNC or RDP). This is distinct from Intuit's own QuickBooks Online cloud service, which I believe is unaffected by this breach.
I was concerned because one of my clients' customers rely heavily on QuickBooks Online and her app integrates heavily with it.
I can confirm QBO is up (I am in Canada). Holy Batman, the chaos if that data would be gone... I did a backup, quickly. I need to ask my accountant whether they back this data up regularly, if not then I need to. I am a very small company but my invoices are international and while I have my invoices at hand, if I would need to reconstruct the exchange rates for an audit a few years back... shudders
Was this service really less expensive than just using actual QB on an RDP available cloud server from Azure or any number of other services?
Also, what kind of hacky backup system takes this much time to sort through to identify issues. They should have a clean image, and a clean way to backup/restore data for the application being hosted as a pull from production/active deployments.
In the end, this will or maybe even should kill the company in question. Beyond this, it is an opportunity for others. For that matter, really surprised Intuit doesn't have this as a cloud service at this point.
It's not impacting all of Insynq's services. I work with an Insynq customer. Their Insynq services are still running, by some generous stroke of fate.
The only outage we noticed was in the middle of the day on the 16th. For about an hour, users weren't able to access the service. I called support, and was diverted to a recorded message saying that they were doing normal maintenance, would be finished shortly, and were aware of and sorry for the disruption.
It's been my outspoken opinion that this was an inevitable outcome for as long as I've been familiar with their product.
Wonder if they obtained any people's financial data or social security numbers. Probably mostly self employed people and small businesses. Pretty scary how people use their SSN for everything. Seems so insecure to have a number you just openly pass around... Need to get a ID or license, credit card, bank account, your doctor, dentist asks for them, your phone company, cable company, of course when getting paid, and even police officers ask for them sometimes and write down in a notepad if your name happens to match someone else's name who has a warrant. Many other uses probably too I didn't think of off the top of my head.
I was randomly one day looking at dentist new patient forms and one even wanted to know your relationship status, not sure how that's relevant if a single or married guy gets a cleaning... I know home alone when the internet went out, so called the local cable company to see if an outage and the lady wanted the social security number on the account before continuing, which I didn't know. Just insane how many things use the same number, it's like single sign on for real life.
Same issue with bank account numbers. To pay someone with direct deposit, they can use the same number to withdraw from your account. I'm surprised banks haven't figured out a way to offer deposit only option... Just create a new account number but linked to another account, where deposits to account 4321 goes to account 1234 instead, but can't ever withdraw from 4321.
I got a feeling Facebook's account system is probably more secure than my local bank. Pretty sad when someone's hobby blockchain project has more technology in it than banks with billions of dollars of assets under management.
> I was randomly one day looking at dentist new patient forms and one even wanted to know your relationship status, not sure how that's relevant if a single or married guy gets a cleaning...
This is for spousal rights - i.e. if your spouse is allowed to request access your data.
> I'm surprised banks haven't figured out a way to offer deposit only option
They have, some German banks assign an IBAN also for "Sparbücher" (saving plans). These cannot be withdrawn from.
For withdrawal security, under SEPA rules you have 8 weeks to (instantly!) reverse a transaction. If you misuse this, you can get your account closed and criminal proceedings filed so that is a relatively effective fraud prevention.
Interesting, don't know much about the spousal rights thing as never been married but felt like they are asking too much info. I figured it's to try and collect if you don't pay since also employment info was a question.
I know I heard in Europe there's some law called PSD2 that banks would provide standard APIs too, but haven't been following that space since not in Europe. I know there's budgeting and other apps but they login to your bank account and scrape the data. I was using a app that categorize your spending for a little while but got sick of it making me relog my accounts over and over. I think one of my credit cards was thinking their servers were trying to hack my account. So a actual official API sounds like the move in the right direction.
Readit News
There is discussion on twitter that the company said the backups were on the same network as the data. Hopefully there is an offsite backup available.
https://twitter.com/ConleyU/status/1151862278909825024
https://twitter.com/MRasconCPA/status/1151894366291734533
https://twitter.com/hockeygirlPDX/status/1151945932935585792
Ouch. This is the sort of stuff that can kill a company.
Does Quickbooks with the cloud option offer local backups?
Backups are so simple, and yet the only times people seem to realize their true value is when they don't have any.
At the very least, if you have backups, you might have the ability to rebuild/redeploy, but for any non-trivial system, without a comprehensive DR plan that is drilled regularly, even having backups won't save the business if the tooling to get those backups into a production ready state.
It gets more interesting when you're trying to backup a distributed database. When your durability strategy is to spread your risk across a lot of nodes in a cluster, and you're federating your cluster across multiple sites, traditional backups may not be a practical option. However, as I write this, I'm realizing that there's nothing simple about distributed anything, and saying "it's hard to..." may not be all that interesting.
Disagree. A high-confidence backup plan that includes continuous restoration is of middling difficulty to design. Many get backups wrong and only find out when their archives turn out to be incomplete.
If a Cloud/SaaS provider is going to claim to be your hassle-free one-stop shop, shouldn't they provide, as a matter of course, a feature to dump your data daily/weekly/monthly to a designated off-cloud site of your choosing? But they don't, because they want to pretend that they are impenetrable and 100% available.
(Which works reasonably well as long as the damages are quantifiable and the other company is actually able to pay them....)
There's a third-party or two that can use the API to do a backup/restore, but also, still, not all of the QBO data has an API. Eg, IIRC, recurring transaction tasks.
Just a matter of time for ransomware to replace data via APIs.
I asked about this in a cybersecurity meeting at an investment bank a few years ago - how much security is there around Bloomberg's data feed? All of the algorithms that make million & billion dollar decisions rely on that data so I asked if someone stepped in the middle and fed the algorithms bad data or even slightly delayed data would we even know it had happened? They looked at me like I was a crazy person. I'm actually kind of surprised it hasn't happened yet. Or maybe no one has noticed yet.
How do you create a backup server that is reachable by production servers (so that they can back up to it) without then being vulnerable to the same kind of ransomware attacks that infect the production servers? You can't exactly make them read-only, or else they can't accept the "legitimate" writes that might occur during the normal backup process.
Production has no access to backup.
Backup has read only access to production.
Backup writes are append and not overwrites.
Deletes/archival are governed by a retention process.
1. Gain access to change the code on the front-end web servers (usually PHP)
2. Change the database access layer to transparently encrypt data being written to the database, and decrypt data being read from the database. The key would be loaded into memory by curl'ing an attacker-controlled website at startup.
3. Wait 30 days
4. Notify the company that they're compromised, turn off the attacker-controlled key service, and restart the web front end
Now step (3) ensures that most data in the database has been re-written, and if your backups are dumps of the production database, you now have a month of encrypted backups that you can't read... If you're lucky, you may have a month-old backup to restore from; if you're unlucky, you rotate every 30 days.
It's nice to have a restore step in there too, so one has both a validation that the backup is usable, and gives a "playground" where one can have a safe day old environment for testing/training/whatnot against production data.
The network is segregated to limit impact if they do get hacked.
To keep everyone familiar, Data Recovery processes are practiced regularly.
...
This gives me the heebie-jeebies.
It seems better to me to have one on Amazon, and one on another cloud provider entirely. Plus regular sneakernet archives, if your data is of a size that would permit that sort of thing.
I'm fortunate that this kind of thing is handled by other people where I work. It's my understanding that we have production data on one service on the east coast, a mirror with a different company in the south, and weekly backup archives stored offline on the west coast.
You just go back to the last good version.
Deleted Comment
Cronjob to an (S)FTP server and an upload script trigger to chown/chmod all incoming files making the whole thing WORM (Write Once Read Many).
Once its submitted the same user account can't alter it. Even if the malware is clever and scans for .netrc and .id_rsa and manages to create its own connection to the backup server it doesn't have access to anything anyway.
- I have a UnRaid machine, and a backup machine. The backup machine is a small itx board, and has a single HDD attached.
- A NodeRED instance has a so-called "Flow" on the UnRaid machine that is waking up the backup machine every 7 days.
- Thanks to anachron, with a 10 minute delay, rsnapshot connects to the UnRaid machine, pulls the data, and then issues a shutdown to the backup machine.
This setup let me sleep pretty well.
My home computers push backups regularly throughout the day, and every day I create snapshots of each volume (how long to keep the snapshots is another question). This snapshot can only be accessed or managed on the NAS itself.
This effectively creates an append-only backup NAS thanks to the periodic snapshots.
As usually with security, the principles of least privs and segregating as much as possible are important.
Of course IANAL so idk how this jives with various EU laws.
EC2 -> S3 bucket with only write access and versioning enabled. EC2 -> EFS and it's a rotating set of 7 with 7 different security groups that rotate.
Deleted Comment
This does not affect non-iNSYNQ QuickBooks instances, such as those operated by Intuit (the creator of QuickBooks).
I was concerned because one of my clients' customers rely heavily on QuickBooks Online and her app integrates heavily with it.
Also, what kind of hacky backup system takes this much time to sort through to identify issues. They should have a clean image, and a clean way to backup/restore data for the application being hosted as a pull from production/active deployments.
In the end, this will or maybe even should kill the company in question. Beyond this, it is an opportunity for others. For that matter, really surprised Intuit doesn't have this as a cloud service at this point.
They do have a cloud offering, QuickBooks Online:
https://quickbooks.intuit.com/online/
But it does not have all the same features as the Desktop version, giving rise to a number of third party offerings, like Right Networks' "QuickBooks Desktop Cloud": https://www.rightnetworks.com/cloud-solutions/accounting-sol...
It's been my outspoken opinion that this was an inevitable outcome for as long as I've been familiar with their product.
I was randomly one day looking at dentist new patient forms and one even wanted to know your relationship status, not sure how that's relevant if a single or married guy gets a cleaning... I know home alone when the internet went out, so called the local cable company to see if an outage and the lady wanted the social security number on the account before continuing, which I didn't know. Just insane how many things use the same number, it's like single sign on for real life.
Same issue with bank account numbers. To pay someone with direct deposit, they can use the same number to withdraw from your account. I'm surprised banks haven't figured out a way to offer deposit only option... Just create a new account number but linked to another account, where deposits to account 4321 goes to account 1234 instead, but can't ever withdraw from 4321.
I got a feeling Facebook's account system is probably more secure than my local bank. Pretty sad when someone's hobby blockchain project has more technology in it than banks with billions of dollars of assets under management.
This is for spousal rights - i.e. if your spouse is allowed to request access your data.
> I'm surprised banks haven't figured out a way to offer deposit only option
They have, some German banks assign an IBAN also for "Sparbücher" (saving plans). These cannot be withdrawn from.
For withdrawal security, under SEPA rules you have 8 weeks to (instantly!) reverse a transaction. If you misuse this, you can get your account closed and criminal proceedings filed so that is a relatively effective fraud prevention.
I know I heard in Europe there's some law called PSD2 that banks would provide standard APIs too, but haven't been following that space since not in Europe. I know there's budgeting and other apps but they login to your bank account and scrape the data. I was using a app that categorize your spending for a little while but got sick of it making me relog my accounts over and over. I think one of my credit cards was thinking their servers were trying to hack my account. So a actual official API sounds like the move in the right direction.