Above: To the right is my hovel at the cottage I setup during my holiday-time-well-spent playing with ddrescue
I had an awesome LVM for nearly 15 years. It saw me through high school, 8 years of post-secondary nerding and then some years after. The drives involved came and went, as I perpetually upgraded it. I loved it dearly. At its peak, it was comprised of 7 drives. Inevitably, the Seagate-reaper came to visit and the LVM was no more. It might have persisted, but I was younger and crazier and I took the striped path to ruin.
When it finally died, 2/3 drives were the Make & Model and both died pretty such in the same moment. They’d lived together in parallel their entire (upsettingly brief) lives. I used the opportunity to learn of ddrescue and I managed to recover about 99% of one of the drives. The other drive was a lost cause (and I tried some serious hardware stuff with it, as I have some skills in that arena). I know how I might recover THAT data, but it wasn’t worth the effort as I had backups of most of the good stuff and the whole thing was just an archive anyway.
After my efforts with ddrescue, 2.5years passed. I had other projects and pressures so the whole thing sat. Today I said it’s time to use the remaining good hardware. And so, I attempted to access what data I could. ddresuce had rendered me a file representing the one drive. I easily mounted it as a loopback device. I followed this guide to deal with the other fully missing drive. With that done, the LVM was alive again, but of course the filesystem was in ruin. It turns out, running FSCK against a large volume takes a lot of memory. There is a flag to getting fsck to store its data structures on disk, but it seemed with each run, it was gradually fixing different issues before running out of memory. So I threw-down yet another ugly little script (here). After a few runs of that, I could mount my LVM-backed file system again! And I was surprised to see a large amount of my data was accessible once again.
So if you too have lost one or more drives from you LVM. Take your time and give recovery a shot. Also, go use FreeNAS (or similar).
Now, to scale my freenas to accommodate a silly burst in data.
Back in 2013 I cobbled together a little guide within a post on the FreeNAS users forum on how to add drive to an existing ZFS volume and convert it in to a mirrored volume. I can’t at all remember even writing this little thing, but recent discussion in the old forum over it brought in to doubt whether or not the method would still work. So I spooled-up a VM with the latest FreeNAS (FreeNAS 11) and tried it out. Yup. Still works, see:
I picked up a Dell 690 from an e-recycler for $11. I picked it out knowing that I can kit-it-out with 64GB of slow DDR2 FBDIMM RAM for about $80 and I can replace the CPUs to get to 8 threads for another $30ish. I somewhat foolishly assumed VMware ESXi 6.5 would just run on this rig. I was wrong, kind of.
Annoyingly, this motherboard seems fully electrically compatible with CPUs that are both dirt-cheap and supported by ESXi, however after a fair search of the internet, I found that the bios does not support such CPUs. The most conclusive findings I encountered were these.
I considered modifying ESXi itself, but it appears it legitimately needs something not supported by the CPUs compatible with the 690’s board. I think I read somewhere that Live Migration depends on a certain CPU feature. I got extra annoyed since I never use Live Migration (as I live on free VMware Licenses in my home lab).
The “solution” that seems to be working perfectly fine was to simply install ESXi on another machine and put the OS drive on the 690 afterwards. I’ve done this and I can report that it boots fine and I created a quick Photon OS VM to confirm it works at all. I half-way expect to find something important doesn’t work, but the Web UI functions fine (which is why I’m pushing for 6.5, else 6.0 would have been okay) and VMs appear functional. So.. Mission Accomplished? This garbage rig is already better than a Quad Core rig I’ve used and depended upon for years.
I’m feeling good. I may upgrade the CPUs to lower TDP ones and get myself to 8 CPU threads along the way. I think I’ll go for 64GB of slower FBDIMM memory first. I couldn’t care less that it’s “slow”. It’s way faster than swap/page files. 😉
Now to get back to my actual project. Murah ha ha ha ha.