Hard drive woes - happy ending

So, a couple of weeks ago, the 6 TB LaCie drive that I had my Plex library and my digitized cassettes of all of my school recordings died. I contacted LaCie/Seagate, and it turns out that the drive had three weeks left on its warranty. I had them send it to Bowling Green where we are spending the New Year with Jade's mother. I took my giant Time Machine backup drive, a 4x3 TB RAID 1 array from Other World Computing, with us to restore.

Once I got the new drive, it was tricky to figure out how to restore the data. It's not a boot drive, and all of the instructions for Time Machine say that you have to restore the system. Turns out the Time Machine backup is not actually restoreable as a system, for whatever reason. Does not show up in Migration Assistant. :(

I did find out that you can boot off of a Time Machine backup as a recovery partition, which is really quite handy if you have to install to a brand new drive in your machine. But I was still unable to restore a complete system.

I tried installing a new version of MacOS on the new drive, and then using Migration Assistant to restore from; same problem.

Was wondering if I was going to have to order a drive from BackBlaze, my iCloud backup. That would mean several more days lost, and given that we are traveling back to Texas soon, I was getting really frustrated.

I then looked at the file system of the Time Machine backup again. At the top level of the drive is a folder called "Backups.backupdb".

If you open this folder, you will see another folder for each computer that has backed up to this Time Machine backup:

If you open the correct machine folder, you get a list of backups.

The key is that last entry, "Latest". When you open that, you see your complete file system, organized by volume/drive.

Notice that the title of the folder is the date and time of the last backup, and indeed, matches the last entry before "Latest" in the list. "Latest" is what is called a "symbolic link". Basically, on the hard drive, it's a special file that points to another file. It is a symbolic link, because the file system sees it as a file at its lowest level. If you remove the symbolic link, the file/folder it points to sticks around.

The contents of the folder, however, are a combinations of files/folders, and what is called "hard links".

Each version of every file you own and have ever backed up to Time Machine is uniquely stored in your Time Machine backup the first time Time Machine sees it. Every time Time Machine backs up your drive, however, if it sees that your file has not changed, it creates a "hard link" to it. This kind of link creates multiple references to the same file. If you remove the file or a hard link to it, they all go away.

The upshot is that every single folder in that list represents the full contents of your file system as of the time the backup was taken.

OK, so once I remembered that, I did the following:

• Reformatted the new drive. Don't need a bootable system on it.
• Opened Terminal.
• Typed the following UNIX commands:

mkdir -p /Volumes/Bari/Users/
rsync --progress --partial -av /Volumes/Beethoven/Backups.backupdb/Syd\'s\ iMac/Latest/Bari/Users/jazzman /Volumes/Bari/Users/


Let's break that down.

"mkdir" is the Unix command for making folders (which in Unix are called "directories"). Adding "-p" to that command tells Unix to make all of the folders containing the final folder. "/Volumes/Bari" is the path to the drive I just formatted. This command then made "/Volumes/Bari/Users". The "-p" was not strictly necessary here.

"rsync" is the Unix utility which detects when files are newer and older than each other and copies them around. It's a poor man's Time Machine. Here are the options:

• "--progress" tells rsync to tell me what it's doing. It just sits there otherwise, and with a restore as long as this, it really helps psychologically to see something happening besides the blinky lights on the disks.
• "--partial" tells rsync that if this is interrupted for some reason (I stop it, power outage, etc.), to pick up where it left off. Since many of these files are large video files, losing all progress on a big file would cost a lot of time.
• "-av" tells rsync to use "archive mode" and to be "verbose". I would suggest you look up rsync via its man page or via Google to dig into the details of archive mode, but basically, it's copy all newer files, treating folders and links correctly. "verbose" means I want to see output. "--progress" also gives me a count of files left, and shows other stats while the copy is going.
• "/Volumes/Beethoven/Backups.backupdb/Syd\'s\ iMac/Latest/Bari/Users/jazzman" is the path I want to restore from. I have to quote the apostrophe and spaces in the path with backslashes because Unix command-line utilities treat those characters differently.
• "/Volumes/Bari/Users/" is the destination path. The "jazzman" folder listed above will be synced inside of this folder.

So, I set this going:

[syd@macsyd ~]$ rsync --progress --partial -av /Volumes/Beethoven/Backups.backupdb/Syd\'s\ iMac/Latest/Bari/Users/jazzman /Volumes/Bari/Users/
building file list ...
595775 files to consider
jazzman/
jazzman/Desktop/
jazzman/ISOs/
 
...

jazzman/dev/sydvicious/trip/.git/logs/refs/heads/master
         175 100%    0.85kB/s    0:00:00 (xfer#505969, to-check=15/595775)
jazzman/dev/sydvicious/trip/.git/logs/refs/remotes/
jazzman/dev/sydvicious/trip/.git/logs/refs/remotes/origin/
jazzman/dev/sydvicious/trip/.git/logs/refs/remotes/origin/HEAD
         175 100%    0.85kB/s    0:00:00 (xfer#505970, to-check=12/595775)
jazzman/dev/sydvicious/trip/.git/objects/
jazzman/dev/sydvicious/trip/.git/objects/info/
jazzman/dev/sydvicious/trip/.git/objects/pack/
jazzman/dev/sydvicious/trip/.git/objects/pack/pack-4c25d7ded28170de85a7e5179cf1f2964004290a.idx
        2836 100%   13.71kB/s    0:00:00 (xfer#505971, to-check=8/595775)
jazzman/dev/sydvicious/trip/.git/objects/pack/pack-4c25d7ded28170de85a7e5179cf1f2964004290a.pack
      187147 100%  887.19kB/s    0:00:00 (xfer#505972, to-check=7/595775)
jazzman/dev/sydvicious/trip/.git/refs/
jazzman/dev/sydvicious/trip/.git/refs/heads/
jazzman/dev/sydvicious/trip/.git/refs/heads/master
          41 100%    0.19kB/s    0:00:00 (xfer#505973, to-check=4/595775)
jazzman/dev/sydvicious/trip/.git/refs/remotes/
jazzman/dev/sydvicious/trip/.git/refs/remotes/origin/
jazzman/dev/sydvicious/trip/.git/refs/remotes/origin/HEAD
          32 100%    0.15kB/s    0:00:00 (xfer#505974, to-check=1/595775)
jazzman/dev/sydvicious/trip/.git/refs/tags/

sent 2601869494866 bytes  received 11436104 bytes  42318360.71 bytes/sec
total size is 3644059082442  speedup is 1.40
 
I actually had to invoke this three times and stop the first two for various reasons. However, it did complete. It took 36 hours to restore 3.6 TB of data, but it's all there. When I get home, I can hook this back up to my iMac, and everything will be normal.

However, that Time Machine backup can't restore my system. That is worrisome, so I am going to have to nuke it and start over, let the backup finish several days later, and then see if I can restore a system with it. Nothing like having to perform a restore to find the problems with your backups.