Tonight I was doing a little development work towards a telemetry system I’m building for a thing. Along the way I managed (like a 10/10 n00b) to delete a bunch of vital configs on my reverse proxy server that handles all my traffic. Thanks to the amazing ZFS snapshot function made easily available by FreeNAS, I was able to recover from this otherwise devastating fckup, super fast and without service disruption.
Read more “Death by PEBKAC evaded by amazing ZFS snapshot CTRL+Zery”
Above: To the right is my hovel at the cottage I setup during my holiday-time-well-spent playing with ddrescue
I had an awesome LVM for nearly 15 years. It saw me through high school, 8 years of post-secondary nerding and then some years after. The drives involved came and went, as I perpetually upgraded it. I loved it dearly. At its peak, it was comprised of 7 drives. Inevitably, the Seagate-reaper came to visit and the LVM was no more. It might have persisted, but I was younger and crazier and I took the striped path to ruin.
When it finally died, 2/3 drives were the Make & Model and both died pretty such in the same moment. They’d lived together in parallel their entire (upsettingly brief) lives. I used the opportunity to learn of ddrescue and I managed to recover about 99% of one of the drives. The other drive was a lost cause (and I tried some serious hardware stuff with it, as I have some skills in that arena). I know how I might recover THAT data, but it wasn’t worth the effort as I had backups of most of the good stuff and the whole thing was just an archive anyway.
After my efforts with ddrescue, 2.5years passed. I had other projects and pressures so the whole thing sat. Today I said it’s time to use the remaining good hardware. And so, I attempted to access what data I could. ddresuce had rendered me a file representing the one drive. I easily mounted it as a loopback device. I followed this guide to deal with the other fully missing drive. With that done, the LVM was alive again, but of course the filesystem was in ruin. It turns out, running FSCK against a large volume takes a lot of memory. There is a flag to getting fsck to store its data structures on disk, but it seemed with each run, it was gradually fixing different issues before running out of memory. So I threw-down yet another ugly little script (here). After a few runs of that, I could mount my LVM-backed file system again! And I was surprised to see a large amount of my data was accessible once again.
So if you too have lost one or more drives from you LVM. Take your time and give recovery a shot. Also, go use FreeNAS (or similar).
Now, to scale my freenas to accommodate a silly burst in data.
Back in 2013 I cobbled together a little guide within a post on the FreeNAS users forum on how to add drive to an existing ZFS volume and convert it in to a mirrored volume. I can’t at all remember even writing this little thing, but recent discussion in the old forum over it brought in to doubt whether or not the method would still work. So I spooled-up a VM with the latest FreeNAS (FreeNAS 11) and tried it out. Yup. Still works, see:
Read more “FreeNAS 11 Add a drive to create a mirrored ZFS volume”