Death by PEBKAC evaded by amazing ZFS snapshot CTRL+Zery

Tonight I was doing a little development work towards a telemetry system I’m building for a thing. Along the way I managed (like a 10/10 n00b) to delete a bunch of vital configs on my reverse proxy server that handles all my traffic. Thanks to the amazing ZFS snapshot function made easily available by FreeNAS, I was able to recover from this otherwise devastating fckup, super fast and without service disruption.

Read more “Death by PEBKAC evaded by amazing ZFS snapshot CTRL+Zery”

LVM Recovery Bedtime Story: The tale of the journey home for some long lost bytes


Above: To the right is my hovel at the cottage I setup during my holiday-time-well-spent playing with ddrescue


I had an awesome LVM for nearly 15 years. It saw me through high school, 8 years of post-secondary nerding and then some years after. The drives involved came and went, as I perpetually upgraded it. I loved it dearly. At its peak, it was comprised of 7 drives. Inevitably, the Seagate-reaper came to visit and the LVM was no more. It might have persisted, but I was younger and crazier and I took the striped path to ruin.

When it finally died, 2/3 drives were the Make & Model and both died pretty such in the same moment. They’d lived together in parallel their entire (upsettingly brief) lives. I used the opportunity to learn of ddrescue and I managed to recover about 99% of one of the drives. The other drive was a lost cause (and I tried some serious hardware stuff with it, as I have some skills in that arena). I know how I might recover THAT data, but it wasn’t worth the effort as I had backups of most of the good stuff and the whole thing was just an archive anyway.

After my efforts with ddrescue, 2.5years passed. I had other projects and pressures so the whole thing sat. Today I said it’s time to use the remaining good hardware. And so, I attempted to access what data I could. ddresuce had rendered me a file representing the one drive. I easily mounted it as a loopback device. I followed this guide to deal with the other fully missing drive. With that done, the LVM was alive again, but of course the filesystem was in ruin. It turns out, running FSCK against a large volume takes a lot of memory. There is a flag to getting fsck to store its data structures on disk, but it seemed with each run, it was gradually fixing different issues before running out of memory. So I threw-down yet another ugly little script (here). After a few runs of that, I could mount my LVM-backed file system again! And I was surprised to see a large amount of my data was accessible once again.

So if you too have lost one or more drives from you LVM. Take your time and give recovery a shot. Also, go use FreeNAS (or similar).

Now, to scale my freenas to accommodate a silly burst in data.

All Hail Tony, The Digital Warlock

IMG_3291

This is the tale of a man who was faced with colossal data loss. As his family photos seem to fade in to the ether and with nothing to lose, he drew a line in the digital sand. He gained a persona that day, he was no longer just “Tony: Loving husband and father”. No no, that fateful day this man also became The Digital Warlock.

Read more “All Hail Tony, The Digital Warlock”