LVM Recovery Bedtime Story: The tale of the journey home for some long lost bytes


Above: To the right is my hovel at the cottage I setup during my holiday-time-well-spent playing with ddrescue


I had an awesome LVM for nearly 15 years. It saw me through high school, 8 years of post-secondary nerding and then some years after. The drives involved came and went, as I perpetually upgraded it. I loved it dearly. At its peak, it was comprised of 7 drives. Inevitably, the Seagate-reaper came to visit and the LVM was no more. It might have persisted, but I was younger and crazier and I took the striped path to ruin.

When it finally died, 2/3 drives were the Make & Model and both died pretty such in the same moment. They’d lived together in parallel their entire (upsettingly brief) lives. I used the opportunity to learn of ddrescue and I managed to recover about 99% of one of the drives. The other drive was a lost cause (and I tried some serious hardware stuff with it, as I have some skills in that arena). I know how I might recover THAT data, but it wasn’t worth the effort as I had backups of most of the good stuff and the whole thing was just an archive anyway.

After my efforts with ddrescue, 2.5years passed. I had other projects and pressures so the whole thing sat. Today I said it’s time to use the remaining good hardware. And so, I attempted to access what data I could. ddresuce had rendered me a file representing the one drive. I easily mounted it as a loopback device. I followed this guide to deal with the other fully missing drive. With that done, the LVM was alive again, but of course the filesystem was in ruin. It turns out, running FSCK against a large volume takes a lot of memory. There is a flag to getting fsck to store its data structures on disk, but it seemed with each run, it was gradually fixing different issues before running out of memory. So I threw-down yet another ugly little script (here). After a few runs of that, I could mount my LVM-backed file system again! And I was surprised to see a large amount of my data was accessible once again.

So if you too have lost one or more drives from you LVM. Take your time and give recovery a shot. Also, go use FreeNAS (or similar).

Now, to scale my freenas to accommodate a silly burst in data.

FreeNAS 11 Add a drive to create a mirrored ZFS volume

Back in 2013 I cobbled together a little guide within a post on the FreeNAS users forum on how to add drive to an existing ZFS volume and convert it in to a mirrored volume. I can’t at all remember even writing this little thing, but recent discussion in the old forum over it brought in to doubt whether or not the method would still work. So I spooled-up a VM with the latest FreeNAS (FreeNAS 11) and tried it out. Yup. Still works, see:

Continue reading FreeNAS 11 Add a drive to create a mirrored ZFS volume

VMware ESXi 6.5 Hypervisor on Dell 690

I picked up a Dell 690 from an e-recycler for $11. I picked it out knowing that I can kit-it-out with 64GB of slow DDR2 FBDIMM RAM for about $80 and I can replace the CPUs to get to 8 threads for another $30ish. I somewhat foolishly assumed VMware ESXi 6.5 would just run on this rig. I was wrong, kind of.

Annoyingly, this motherboard seems fully electrically compatible with CPUs that are both dirt-cheap and supported by ESXi, however after a fair search of the internet, I found that the bios does not support such CPUs. The most conclusive findings I encountered were these.

I considered modifying ESXi itself, but it appears it legitimately needs something not supported by the CPUs compatible with the 690’s board. I think I read somewhere that Live Migration depends on a certain CPU feature. I got extra annoyed since I never use Live Migration (as I live on free VMware Licenses in my home lab).

The “solution” that seems to be working perfectly fine was to simply install ESXi on another machine and put the OS drive on the 690 afterwards. I’ve done this and I can report that it boots fine and I created a quick Photon OS VM to confirm it works at all. I half-way expect to find something important doesn’t work, but the Web UI functions fine (which is why I’m pushing for 6.5, else 6.0 would have been okay) and VMs appear functional. So.. Mission Accomplished? This garbage rig is already better than a Quad Core rig I’ve used and depended upon for years.

I’m feeling good. I may upgrade the CPUs to lower TDP ones and get myself to 8 CPU threads along the way. I think I’ll go for 64GB of slower FBDIMM memory first. I couldn’t care less that it’s “slow”. It’s way faster than swap/page files. 😉

Now to get back to my actual project. Murah ha ha ha ha.

Unix Magic Trick: Rename stuff in bulk

Here’s a magic-trick I rock from time to time. I only graze the basic abilities of the “rename” program, but even in my basic use of it, I find it super helpful.

In this case, I had a situation where my eBooks had been resorted from a massive flat directory in to one containing sub-dirs named with the prefix “Categories – “. I might have sorted them in another manner if I didn’t know about the rename command. When I was done sorting and thus left with my category-based directories, I then wanted to rename each dir to dump the prefix.

A simple call to rename is all it takes:
rename "s/[what to find]/[what to replace it with]/" *

Continue reading Unix Magic Trick: Rename stuff in bulk

Dear Diary: Samsung ML-1610 on macOS 10.12 Sierra

New Version

A newer version is here. I now call this ml1610-blaster.

I have a trusty Samsung ML-1610 printer that I think I bought for like $90 back around 2003. It’s been an outstanding little cheapie that so far has seen me through two undergrad programs and is now seeing my wife through her third such program. Overall, it’s unremarkable, but I like it enough to keep it going.

I’ve long-since used SpliX to get it to work under more recent versions of macOS. Lately, I’ve found splix’s installer fails to complete (I suspect SIP is in the way). And so this post exists to help me keep track of what I’ve done to install it on my wreckless fleet of macs.

Continue reading Dear Diary: Samsung ML-1610 on macOS 10.12 Sierra