Sorry, but "Untrusted remote storage" and "No server-side processing" are exactly the opposite of what I need.
If the original box is ever compromised, I don't want the attacker to gain any access to the backup. If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box, and anyone who gains control of the original box can destroy your S3 bucket as well. Ditto for any SSH-based backup scheme that requires keys to be stored on the original box. A compromised box could also lie about checksums, silently corrupting your backups.
Backups should be pulled from the backup box, not pushed from the original box. Pushing backups is only acceptable for consumer devices, and even then, only because we don't have a reliable way to pull data from them (due to frequently changing IP addresses, NAT, etc).
The backup box needs to be even more trustworthy than the original box, not the other way around. I'm willing to live with a significant amount of overhead, both in storage and in bandwidth, in order not to violate this principle.
The backup box, of course, could push encrypted data to untrusted storage, such as S3. But only after it has pulled from the original box. In both cases, the connection is initiated from the backup box, not the other way around. The backup box never accepts any incoming connection.
Does Attic support this kind of use case? The documentation doesn't seem to have anything to say about backing up remote files to local repositories. I don't see any reason why it won't be supported (since rsync does), but "nominally supported" is different from "optimized for that use case", and I suspect that many of the latest generation of backup tools are optimized for the opposite use case.
I really try to restrain myself when a backup article pops up on HN, but there are two things you raise here that I'd like to address ... first:
"Ditto for any SSH-based backup scheme that requires keys to be stored on the original box. A compromised box could also lie about checksums, silently corrupting your backups."
This is a good thought - you should indeed be thinking about an attacker compromising your system and then using the SSH keys they find and wiping out the offsite backup. All they need to do is look into cron and find the jobs that point to the servers that ...
So how do we[1] solve this ? All of our accounts have ZFS snapshots enabled by default. You may not be aware of it, but ZFS snapshots are immutable. Completely. Even root can't write or delete in a snapshot. The snapshot has to be deliberately destroyed with ZFS commands run by root - which of course, the attacker would not have access to. It's a nice safety net - even if your current copy is wiped out, you have your ZFS snapshots in place.[2]
"Backups should be pulled from the backup box, not pushed from the original box."
This was the tipping point - I had to comment. Since day one, we have, free of charge, set up "pull jobs" for any customer that asks for it. Works just like you'd like it to on whatever schedule they can cram into a cron format. It's a value add we've always been happy to provide.
[1] You know who we are.
[2] Yes, if you don't notice for 7 days and 4 weeks that the attacker has wiped you out, at that point your snapshots will all rotate into nothingness as well. Nothing's perfect.
Is it possible to be notified if a certain percent of the backup is changed? Something that would let me tell if something like 50% of the bytes or 50% of the files are different between snapshots? Just a simple 'zfs diff | wc -l | mail' in cron?
By pulling backups, you're giving the backup box full control over your computer, meaning that yes it must be more trustworthy. Push backups can indeed allow the initiator to wreck the remote state.
But in the modern commercial market, who are you going to trust as your backup box provider? A USA company subject to NSLs? Run your own in a rack somewhere? Having an untrusted server greatly decreases cost and increases the chance that you can actually produce a successful backup infrastructure at all.
It is possible to do it safely.
Since you say you're "willing to live with a significant amount of overhead", i would suggest a two-tier push/pull configuration. Desktop pushes to site A; then site B pulls from site A. This also increases redundancy and spreads out the attack surface.
Append-only is another good solution - i don't believe attic formally supports this today but it should be as simple as patching `attic serve` to ignore delete requests. Good first patch.
(Also if you really trust your backup server, then you don't need encryption anyway and can just run rdiff-backup over ssh.)
> By pulling backups, you're giving the backup box full control over your computer
Not really. On my production boxes, I usually set up a "backup" account and give it read-only access to paths that need to be backed up. Nothing fancy, just standard POSIX filesystem permissions. The backup box uses this account to ssh in, so it can only read what it needs to read, and it can never write anything to the production box. I wouldn't call that "full control".
> Desktop pushes to site A; then site B pulls from site A.
What you described is similar to my own two-tier configuration, except I pull first and then push to untrusted storage like S3 (using encryption, of course). The first step uses rsync over ssh. The second step is just tar/gzip/gpg at the moment, but if I want deduplication I can easily switch to something like tarsnap.
I guess it depends on your security model. With one single pull backup,
- if your backup box is on another network then it can be coerced into malicious reads (leaking private information, trade secrets, your competitive advantage etc).
- if it's on the same network then it's subject to your same failure patterns.
Push backup has some disadvantages, but there's a lot of peace-of-mind in never (intentionally) granting additional users access to the unencrypted data.
Two-tier is one approach. There's another comment in this thread about snapshotting filesystems (ZFS, or i suppose LVM snapshots might be easier) which would be another method of addressing concerns about the client tampering with the backed up data.
You can have a self-controled intermediary machine that pulls backups, encrypts, and then pushes them to the untrusted cloud.
When I had no resources for this (eg: low income student), I had a server at my mom's place that did this for me. Low-cost offsite, trustable backup server for personal usage.
If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box
I believe this isn't strictly necessary if you use asymmetric cryptography (e.g. curve25519). For a file, generate a temporary key pair, use it and the backup's public key to encrypt the file, then throw out the private key and send the encrypted file + public key to the server.
Apple uses this technique to move files to the "Accessible while unlocked" state without having the key for that state (i.e. while the device is locked).
Just for the record, asymmetric cryptography is not efficient for encrypting content. What you should do is:
- Generate a temporary key
- Symmetrically encrypt with that key
- Encrypt that key with your long-term assymetric private key, and send the encrypted version along your backups.
And before you hack around your own version, I'd like to point out this is exactly what PGP (and really, any crypto scheme that involves asymmetric keys) does. So, basically, just GPG your backups.
I use duplicity + rsync.net and have wondered about this attack vector. My solution (which admittedly only protects against remote backups being deleted by an attacker, not read) is:
1. Use sub accounts on rsync.net so backups from different parts of the system are isolated from each other.
2. Use a different GPG keypair and passphrase for each host being backed up.
3. Have an isolated machine out on the internet somewhere (that, importantly, isn't referenced by anything in the main system including documentation / internal wikis i.e. so the attackers don't know it exists) that does a daily copy of the latest and previous full backup plus any current incrementals directly from rsync.net's storage. This way I'm still covered (and can restore relatively quickly) if an attacker gets in to the system and deletes the rsync.net hosted backups for lulz.
If you're truly paranoid or need to protect backups going back over months you could also introduce a final routine that duplicates the data from the ghost machine to Amazon glacier (and then optionally pay for an HDD to be shipped periodically to your offices).
Your rsync.net account has ZFS snapshots enabled - at the very least, the smallest default is 7 daily snapshots.
The ZFS snapshots are immutable. Completely. Even root can't write or delete in a snapshot. The snapshot has to be deliberately destroyed with ZFS commands run by root - which of course, the attacker would not have access to.
Ha! Well that simplifies things for me :) Honestly, your service is so solid I can't remember the last time I actually logged in to check something. It really is backup as a utility (you should consider re-branding as "BaaU" lol)
Better, but it might still expose to the attacker more data than he would otherwise have access to. For example: production box only contains data from the last 3 days, but the backup contains data from the last 12 months.
Even stricter access controls (write once, no read) might help with that. Not sure if you can do that with S3 though.
>production box only contains data from the last 3 days, but the backup contains data from the last 12 months
Even if the source can only perform new backups, it's a timing attack with a deduplicating system. The attacker can attempt to back up chosen data to infer properties of the existing backups.
You can remove this only by removing deduplication (or by crippling deduplication to work only at the server-side, and incur wasteful network requests)
Which is why tarsnap says about your keys: STORE THIS FILE SOMEWHERE SAFE! Copy it to a different system, put it onto a USB disk, give it to a friend, print it out (it is printable text) and store it in a bank vault — there are lots of ways to keep it safe, but pick one and do it.
Indeed, that's exactly what I do (with a small script that wraps rsync): the backup server PULLS data from my machine, and saves incremental backups (uses hardlinks, see --link-dest). In case my machine is compromised, the backup server is still inaccesible.
If the original box is ever compromised, I don't want the attacker to gain any access to the backup. If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box, and anyone who gains control of the original box can destroy your S3 bucket as well. Ditto for any SSH-based backup scheme that requires keys to be stored on the original box. A compromised box could also lie about checksums, silently corrupting your backups.
Backups should be pulled from the backup box, not pushed from the original box. Pushing backups is only acceptable for consumer devices, and even then, only because we don't have a reliable way to pull data from them (due to frequently changing IP addresses, NAT, etc).
The backup box needs to be even more trustworthy than the original box, not the other way around. I'm willing to live with a significant amount of overhead, both in storage and in bandwidth, in order not to violate this principle.
The backup box, of course, could push encrypted data to untrusted storage, such as S3. But only after it has pulled from the original box. In both cases, the connection is initiated from the backup box, not the other way around. The backup box never accepts any incoming connection.
Does Attic support this kind of use case? The documentation doesn't seem to have anything to say about backing up remote files to local repositories. I don't see any reason why it won't be supported (since rsync does), but "nominally supported" is different from "optimized for that use case", and I suspect that many of the latest generation of backup tools are optimized for the opposite use case.