All Hail Tony, The Digital Warlock

IMG_3291

This is the tale of a man who was faced with colossal data loss. As his family photos seem to fade in to the ether and with nothing to lose, he drew a line in the digital sand. He gained a persona that day, he was no longer just “Tony: Loving husband and father”. No no, that fateful day this man also became The Digital Warlock.

The Story of The Digital Warlock
Mr. Tony was but a simple genius-grade geek. He busily juggled time between multiple consulting gigs where his skillfully crafted software ran essential systems that each of his customer’s businesses depended on. (I’m not exaggerating, I know it sounds lavish. It’s also true. And I feel qualified to assert it.)

One day Tony went home to discover his faithful iMac would no longer boot. It’d turn on and eventually display a sadish looking hard drive icon. To his horror, Tony discovered a then recently documented defect had struck. This was the first time a hard drive had ever failed the tech savvy Tony. So he had not been overly concerned with backing up his family’s digital photos, which now seemed completely destroyed. He employed a handful of data recovery tools that all seemed promising, but ultimately failed to deliver or invoke any sense of confidence.

Feeling the nasty prospect of these irreplacable artifacts of his family’s history slipping away, Tony rolled up his sleeves. He realized that while the recovery tools seemed useless, the hard drive did still seem remotely alive. He managed to use a trusty tool (dd) to create a copy of whatever raw data the drive would still spit out. From there, he looked in to open source software that had been used for ipods on windows. This lent insight in to the workings of HFS+ file systems, a pre-requisite to completing his quest.

Ultimately, Tony managed to assemble a custom program that was able to interpret the raw data he captured and reconstitute the files in their original form. And just like that, The Digital Warlock came to exist.

 

The immediate aftermath
I know Tony from one of his consulting gigs. As he battled the data recovery project, we’d discuss his progress a bit. I’d try to offer what suggestions and ideas I could (dd was my idea). Though I must admit, in my opinion, I figured he was screwed. I held no practical hope for him ever seeing those images of his children as little pip-squeaks again.

So, naturally, I was stunned when a few days after he started writing his own recovery program, he came to me and excitedly reported he’d solved it and got his data back! After the initial cheer and ovation I bestowed, it occurred to me this may have been a silent victory. I asked him if he’d told others and how had they responded. I wasn’t surprised to hear that while he’d told people, no one found it interesting nor impressive. Being something of a (talented?) geek myself, I can think of many such times in my own life. It feels kind of horrible when you really do something you think is profoundly awesome and people are irritated by your attempts to even explain what you did. It’s a lonely place and probably a fact of life for many people.

I decided to do something about it.

 

This will not stand
I figured it must be rare that someone gets an award for something that really really really means something. It seems to me that many awards are simply handed out, and someone will be selected for each regardless. Further, when it comes to highly technical situations, I imagine that most awards are assigned by people not overly qualified to assess what was done. In the case of Tony defeating a rather obscure a tricky technical demon, I felt I was qualified to make a stand and not let his impressive feat vanish from recognition.

I decided a trophy of sorts was required. After finding a simple base at a local trophy shop, I came up with Tony’s well-earned title, The Digital Warlock. I then salvaged a recently retired hard drive of mine and retrofitted it on to the trophy. The photos say it all.

Suffice to say, the drive was loaded with one fresh file. The best quality Rick Roll I could dredge from the darkest corner of the internet. Hey, I don’t make the rules.

IMG_3248IMG_3250IMG_0184IMG_3284IMG_3286

1 comment to All Hail Tony, The Digital Warlock

  • Tony Enerson

    Tony’s “Programmer’s Guide to not Losing Important Data on your Mac”:

    Option 1:
    – Enable Time Machine

    Option 2:
    – Wait for a hard drive failure
    – Follow steps outlined in the detail below
    – Enable Time Machine

    The Details:

    The iMac 1TB Drive Replacement Program came too late for me.

    Just before going on a summer vacation (literally an hour before we left) I powered on Mac to have a quick look at a google map to verify some detail – and… the mac… didn’t… boot. The journey starts there.

    After our return then the fun with this drive really started. I dragged my Mac into work where a fellow Mac user and proficient system admin (James) helped me troubleshoot it a bit.

    I didn’t know this, but holding down the option key during startup allows you to boot off the CD or a USB drive. Thank goodness my generation of Mac still came with install CDs, because they don’t anymore. Once I was up an running off a external 2TB drive and a base install of Snow Leapord… I could at least have a look.

    Assuming it was a minor file corruption problem in some OS file, and not a problem with the integrity of the drive, my plan was just to copy off what I wanted from my user folders into a clean install. I kind of wanted to upgrade to the latest OS anyway, so this would have been a decent excuse to rebuild my files and do some house cleaning before the upgrade.

    I was wrong. The disk utility would see the drive – but the filesystem wouldn’t mount. Any attempt to use the repair tools would hang.

    So I escalated to consumer brand data recovery tools. Did you know that the Apple App store doesn’t install software onto external drives? I didn’t, and it became a real headache while attempting to buy data recovery products. It seems to me most emergency users of data recovery products would be in this configuration, so selling data recovery apps through the app store might be a bit.. um.. trying for those people, but I digress. Thankfully most of the packages had demos available that could be downloaded independently, and there are reviewed comparisons between the top contenders online. The extra bad news for me was that the ones I tried claimed the drive was damaged and hardware intrusion was the only option.

    Now I was worried. My personal backups were old. I didn’t really know what I had backed up, there were certainly going to be holes… mostly my pet projects and family photos were at risk. The very thought of the sorry-honey-but-your-computer-professional-husband-lost-our-family-photos discussion made me shudder in horror. I had to make this work, and my personal toolbox wasn’t empty yet.

    Of course over the few days while this was going on in the evenings I had been casually chatting with people at work about it. James was really helpful, having a decent system administrator’s background (which I find myself lacking) he would make helpful suggestions. He had mentioned that he had heard the first rule of data recovery is to get an image of the failing hardware, and dd was for that sort of thing.

    So I tried to dd the whole drive into a file onto my external drive. It would hang. But it would also copy the start of the drive. It would always hang in the same spot of the disk copy. So I would skip a few blocks and try again. Ultimately I discovered reads from certain addresses (about 50Mb near the start of the volume) would hang – but once I was past that I could copy the rest of the volume to a raw file.

    Now I had a small pile of files with partial disk dumps. This was now a software problem. Thankfully, I love software problems, but I know little of file systems.

    Enter the open source community.

    Doing some research it turns out the preferred file system on the Mac is HFS. It is also the file system Apple uses on their iPod products. Given that Windows and linux don’t have native support for HFS, anyone wanting to look at the files on their iPod using their Linux or Windows machine needs to use downloaded software to do it. These programs have all of the HFS structures and conventions coded within them…. luckily there are a few open source HFS file system browsers available.

    Java is my tool of choice – so I looked for Java implementations of HFS. The absolute best for what I needed was catacombae’s HFS Explorer ( http://www.catacombae.org/hfsx.html ). Erik’s layered treatment of the filesystem made sense to me, and he had already factored the random access to the underlying device behind an interface. In theory I could build a new implementation that stitched the dogs breakfast of file dumps I had into something coherent enough for his routines to work. Also I had a working model of HFS to learn from. Making a library out of it proved easier than I first thought because he had designed it well.

    So I started writing programs to get what I could out of the disk dumps.

    The first step in using the HFS file system is to get the file system header. Thankfully that was intact, and the reason why disk utility and other tools could see the disk.

    It turns out the crashed 50Mb landed in extents allocated to the catalog. The catalog is pretty important. There is a catalog entry for every file and folder on the volume, within each entry there are the addresses of the file data. That means few thousand files had been toasted. It is possible to recover those files, but without a catalog entry you are generally forced to look through disk blocks and try to recognize and match content. To use a book analogy, the catalog entry is like a binding on a book – so without it you just have loose pages lying on the floor (often consecutive, but without page numbers). This task would be the equivalent of trying to reconstitute a small public library from piles of unnumbered pages lying on the floor.

    However I finally got some good news – due to their positions in the catalog it is likely the damaged files were added early in the disk’s life… like in the factory.

    I wrote another program to troll the catalog and dump out a text file of filenames of what was readily recoverable, complete with some semblance of the parent directory structure up to the point we lost it. The trick here was recognizing that when folders pointed to parents that didn’t exist anymore – they ended up in a special “unrooted” folder.

    I discovered my user files were bound in the good catalog entries I had… so I didn’t have to go deeper into the entries that were lost (thank goodness).

    I then filtered the list using command line tools (sort, vi) to build a new file of what I really wanted to recover.

    Program 3 would go pass through my filtered file list to build a live directory tree structure in my local file system for the files I wanted to recover.

    The final pass would actually only deal with files – it would read the catalog entry from the damaged catalog and copy the file data from the drive image into appropriately named files of my recovery tree.

    I finally had all of the files I wanted properly named in a directory tree on my local disk.

    At this point it is all downhill – Copy the files back to the users that originally owned them – with a few quirks iPhoto and other apps just picked everything up as though it was there all along (the caveat is that your Library folder is quite important in holding the settings you held near and dear).

    The short conversation with my wife at the end of this:

    “Oh yeah, I got all the files back, your email should work now.”
    “Really – that sure took a while…”

    Ironically, the SMART status of the offending disk finally issued a warning at some point in time after the end of this process, despite the fact I hadn’t been using the disk for some days while I hacked away at recovery. Apparently the computer finally had detected the drive was failing.

    Two weeks later I heard the 1TB drive replacement program was extended to cover my model of iMac. When I brought my Mac in to get it’s new hard drive I tried to tell the abridged version of this story. But the genius just sort of stared at me. I guess this escapade was to be a private victory.

    The motto of this story – enable Time Machine.