Want to Read Your Parents' / Boss' / Coworkers' Files? Mozy Client Circumvents Windows File Access Permissions

Last fall I wrote about Mozy, a cloud-backup company (now owned by EMC), which had a severe bug: it didn't back up all the files it was scheduled to. What was most disturbing was not that they hadn't caught the bug ... it's that when I tried to work with them to resolve it, they blew me off. Apparently I wasn't the only one to have this problem either, as people from all over found my blog post and emailed me, saying they saw the same behavior and asking if I had a fix.

I still think that not backing up files is the biggest possible bug for a backup program.

But now there's competition: their latest client allows a non-privileged user on a Windows XP system (haven't tried it on Vista yet) to restore private files belonging to any other user ... including Administrators, and place the "restored" files into the non-privileged user's folders, with full access, and fully decrypted.

How does this work? Say Jim, the Admin on the box, installs Mozy in a default configuration. Mozy is backing up Jim's "My Documents" folder and subfolders, among various other things. Jim's son Lenny, who is a non-privileged XP user and who is not supposed to have access to Jim's private files logs on.

Lenny notices that "My Computer" contains a virtual drive corresponding to the Mozy backup set. In that virtual drive is ... a "C:" drive ... and a "Documents and Settings" folder ... and a "Jim" folder. Now in XP, "C:\Documents and Settings\Jim" is another name for Jim's "My Documents" folder. This particular instance though is not the actual "C:\Documents and Settings\Jim", which Lenny doesn't have any access rights to, but the backup image of the folder.

So Lenny browses through, finds something interesting, right clicks and chooses "Restore To," which lets him "restore" his dad's file to somewhere of his choosing. He browses to his Desktop, clicks ok, waits a few seconds, and now he has the file.

(Even if Jim has chosen to manage his own crypto key -- one of Mozy's coolest features -- the Mozy client keeps that key accessible so that it can perform automated backups. Unfortunately, it also uses the key for restore operations no matter who performs the restore ... so files restored in this way are decrypted.)

Ok, so there are no secrets on the family computer. Not the biggest surprise, since if junior really wants the data, there are tons of other ways to get it, from booting a Live CD and mounting the hard disk, to yanking the hard drive right out of the box.

But here's where it gets more interesting: In many (most? ... all that I've worked at anyway) corporate, Active Directory- / Domain Controller- managed XP Pro deployments, folks can log on to each others' workstations at will, provided they use their own credentials in the Domain. Their Domain profile updates to the local machine as necessary, and they can then work there. They may even be able to log in remotely via Remote Desktop.

So at work, I walk up to my co-worker's machine (or maybe RDP in) and log in as myself. I open the Mozy tray icon, and proceed to restore their files from the backup set to my own directory, or to an unprotected area like C:\Temp. From there I can open/read/copy these files however I like. Incidentally, if my boss' files weren't already scheduled to back up, I can add them to the backup set. Next time the backup runs, they'll show up in my view of the "Virtual/Restore Drive."

I haven't tested the Mozy "Pro" business client, but since the docs [PDF] look identical to the Home client (apart from a color accent) I suspect it behaves exactly the same way (see in particular sections 7.3 "Using the MozyPro Virtual Drive" and 7.4 "Right-Click Restores") Not to mention that if Mozy fixed this in the Pro edition I can't think of any reason they'd intentionally keep a broken code fork for the Home edition.

I think there are some other fun tricks one could play with this client too, but it all boils down to two things:

1. The underlying process is a privileged process, but it takes orders from a client run by any user.(I'm no security guru, but this sounds like a Confused Deputy problem to me.)
2. The full fidelity of the local file (including a way to associate ownership and permissions) is not being preserved through the backup round trip.

I'd have reported it to Mozy before writing about it here, but they made their lack of interest clear last time around.

Update: I forgot to mention, MozyPro offers "Network share and mapped drive support" ... combined with the bug described here, that's some serious potential risk to add to the mix.

MozyHome Backup Fails to Backup Designated Files

I have been testing out Mozy's MozyHome remote backup product, and have found that it sometimes ignores new or changed files in the backup set marked to be backed up. Sometimes it "discovers" these files (or changes) days or weeks later and backs them up; other times, if I modify a file "nearby" (relative to the way my backup set is constructed), it will suddenly discover all the other changed files and back them up. Still other times, no poking or prodding seems to make it back up these files.

This is a serious problem. After all, the raison d'être of the product is backup. Imagine that a home user or a business installs this solution, marks files to be backed up, observes that the backup process is running successfully and then -- after a data disaster -- learns that, well, some of the files are backed up and some simply are not.

If Mozy were a fly-by-night outfit, one might say that better due diligence is required in choosing a backup provider. But with its recent acquisition by storage giant EMC and its global contract with GE, Mozy appears to be a solid company.

The software, though not perfect, basically works. The backup and restore are straightforward. You can encrypt locally with your own key so that no one but you could ever decrypt your data in the case of a breach (although interestingly, the file names are not encrypted, so don't count on hiding the existence of my_illegal_off_balance_sheet_transactions.xls, or gifts_to_my_mistresses.doc).

But we're talking about the number one, sine qua non, only -- really -- important use case for backup. It has to back files up. Or at least, if it doesn't, it needs to tell you what failed, when, and why.

When I first discovered this problem, I realized that publicizing it could have a negative impact on Mozy's business. So instead of blogging, I contacted them directly to learn more. Unfortunately, after a few back-and-forths, including my running their diagnostic tools and sending them the reports and explaining that I was not going to give their support personnel full remote access to my box without more information, they have gone radio silent.

As a software engineer, I am fully aware that this problem could be a strange corner case. Perhaps it is so narrow that it never affects anyone besides me. But I doubt it. And, in any case, this problem is severe enough that it warrants a little investigation to determine its breadth.

Why do I doubt it is an unusual corner-case failure? Simply because my configuration of the service is so "typical." I'm running XP SP2, NTFS on a well-maintained, modern machine, on a secure home network behind NAT, with no strange services or applications of any kind running (e.g., the kind that might hook and hack kernel file system operations, or leverage alternate data streams). The affected files are not under unusually named file paths, or have any funky attributes set on them.

The only things I am doing that are not defaults are using my own encryption key, and adding a few files to my backup set that aren't in the "My Documents" tree. My conclusion is that it is likely that whatever glitch is causing the software to miss files on my file system, is also missing files on other people's file systems. And they have no idea.

Perhaps this problem doesn't affect other users -- but without trying to verify the bug, how can Mozy know? In my last email to them, I specifically asked if their QA team had even attempted to replicate the bug. Had they tried and failed to reproduce it? Fair enough, maybe I could offer some help. But if they haven't tried, it makes you wonder what bug report could possibly be a higher priority? Maybe if their app runs off the rails and reformats your drive, that's a higher priority. But barring active destruction of your data, or a major security bug that could compromise you to a third party, I can't think of anything.

The ultimate problem here is not even an engineering problem. Yes, there's a bug in the software, but there are bugs in almost all software. Rather, it's a process problem. How does the QA process work? How does customer support work? What steps do you take if someone reports the Really Big Bug? Is the right thing to assume they're a crackpot? Can you afford to do that and not even look into it? (Hint: No.)

</end of regular post> <free:bonus>

Since I'm really not out to get these guys, but ideally want to help, I'm gonna offer a free first step: it's really easy to tell after the fact if this bug is manifesting, since the set of files actually backed up simply doesn't match the local description of the backup sets. So an easy diagnostic is to write a list of these two sets of files and diff them. If there are deltas, you've got a problem.

So... you push out an update to the client that creates a list of the files in the active backup sets and sends it over to QA as the last step in the online backup process. Then QA just has to generate a matching file list (the match is on the account id [email address] and either the date/time or the id number of the backup) from the Mozy meta-data store and compare.