Above: To the right is my hovel at the cottage I setup during my holiday-time-well-spent playing with ddrescue

I had an awesome LVM for nearly 15 years. It saw me through high school, 8 years of post-secondary nerding and then some years after. The drives involved came and went, as I perpetually upgraded it. I loved it dearly. At its peak, it was comprised of 7 drives. Inevitably, the Seagate-reaper came to visit and the LVM was no more. It might have persisted, but I was younger and crazier and I took the striped path to ruin.

When it finally died, 2/3 drives were the Make & Model and both died pretty such in the same moment. They’d lived together in parallel their entire (upsettingly brief) lives. I used the opportunity to learn of ddrescue and I managed to recover about 99% of one of the drives. The other drive was a lost cause (and I tried some serious hardware stuff with it, as I have some skills in that arena). I know how I might recover THAT data, but it wasn’t worth the effort as I had backups of most of the good stuff and the whole thing was just an archive anyway.

After my efforts with ddrescue, 2.5years passed. I had other projects and pressures so the whole thing sat. Today I said it’s time to use the remaining good hardware. And so, I attempted to access what data I could. ddresuce had rendered me a file representing the one drive. I easily mounted it as a loopback device. I followed this guide to deal with the other fully missing drive. With that done, the LVM was alive again, but of course the filesystem was in ruin. It turns out, running FSCK against a large volume takes a lot of memory. There is a flag to getting fsck to store its data structures on disk, but it seemed with each run, it was gradually fixing different issues before running out of memory. So I threw-down yet another ugly little script (here). After a few runs of that, I could mount my LVM-backed file system again! And I was surprised to see a large amount of my data was accessible once again.

So if you too have lost one or more drives from you LVM. Take your time and give recovery a shot. Also, go use FreeNAS (or similar).

Now, to scale my freenas to accommodate a silly burst in data.