Developing a storage strategy

I own a lot of things digitally, and have a highly mobile lifestyle. My CD collection must be 500+, my DVD collection pushing a 100, and I average traveling 300 miles at either end of the week (give or take a hundred depending on where I am). I can pick up a handful of Cd’s and a couple of DVDs to see me through the week on a Sunday night, but when I end up being away for a longer period you get sick of the same 5 albums not to mention the jewel cases being destroyed, and the extra weight to lug about through airports, train stations and on my back.

The answer of course, was to digitize the lot, which I duly did over the course of about a month. CDEX tore through my CD collection (two desktops and a laptop working in parallel). DVDdecoder, acidrip and handbrake sorted out the DVDs gradually. In total the whole lot comes to nearly 300 gigs (no point ripping DVDs to low quality after all). Any new buys are then fed into the system as and when I get them, and now I’ve gone through the pain of the initial digitization, it’s easily managed.

Now this comes with two problems. Firstly, legality. I have bought every one of these items, either first hand or second hand, and I’m not distributing them, so I can’t see that I’m trying to rip people off. I’m just trying to watch/listen to stuff I own.

Secondly, and more importantly (in my book), is how do I store and transport all this stuff? When I went though the initial digitization hard-drives were pretty expensive. You can now get an external 500gig drive for under a hundred quid, so having two drives, one as a backup, is not difficult, but USB drives really really don’t take kindly to being lugged all over the country. I’ve lost one drive so far, and I imagine I’ll lose another fairly so as it’s making some horrible noises. USB also have the problem that it’s single computer only, and it would be very cool to have my music available to any of the computers on my network. Initially I bought an NSLU2 with the idea of hacking the firmware and installing an iTunes server. This would have been nice, but it was fatally flawed (for me). Firstly you need a spare USB hard-drive to use to hack it (install their OS, remount on another machine and thrash the shadow password file, remount and have root) and… it’s only USB 1.1 and thus indexing a 50,000 track mp3 collection is painfully slow, in fact, virtually impossible. Also – no movie streaming, only sharing by samba, which is not ideal.

So it looks like a NAS (network attached storage) device is the key. Out of desperation to get started, I bought A buffalo drive. You trials and tribulations with it are well documented on here.

I currently own 4 external hard drives. All USB2, I have an 80gb, 160gb, 250gb and a 500gb. The 500gb just about has a copy of everything on the other drives, and thus my total storage requirement is about 500Gb. At the moment if it wasn’t for the 500Gb backup, a single disk crash would loose my films, music or photos.

I need a setup which is resilient to single drive failure (well, preferably multi-drive failure but hey I’m not buying Netapp kit!). I’d like everything to be available everywhere via the Internet in a secure manner, but I’m willing to loose this. I’d like to have a couple of weeks of music and films with me portably because I end up working away form home in hotels quite a bit at times.

What I’m planning to have in there end it the following setup:
A Synology 407e cube, which will provide RAID5 across 4 SATA II drives (probably 250Gb each, total storage roughly 750GB). RAID gives me some fail-over at the cost of a about a quarter of the disk space. The 407e will also provide an iTunes server (locally) and media streaming for films, but luckily you can ssh to it securely over the net, so I should be able to use some ssh-tunnel magic to access to the music remotely. It has built in USB ports, so I can get the family machines sharing the printer over it too which is handy, and you can also backup it’s content across the USB to off-site it every so often and prevent a fire wiping out everything I own (though it would take out every other bloody thing I own!).

So that should cover all my on-site needs, but I still travel and would like some local content too (trains, planes and automobiles not having wireless access). At the moment a USB external drive is chunky, but brings with it a wedge of stuff, however it is also heavy and externally powered, making using it on trains etc difficult unless you carry a powerstip (ironically, I often do). I think what I’ve going to do it buy a bloody great USB flash pen. 4gb would be 2 films and 20 Cd’s (a film is roughly a gig, a CD roughly 100megs). This is a little more than I need for a week, not quite enough to 2 weeks, however if I had remote ssh access to the rest of the collection every so often because I was at work, I could top up. An 8 gb keyring would give me more than enough for 2 weeks however.

The total cost of this setup – it’s not cheap. The NAS would be 400 notes barebone. The 4 SATA 250GB drives, about another 120 quid. 50 quid should easily cover the USB pen, and I can reuse my 500GB and 250GB drives, leaving me 2 free smaller portable drive to flog/hold in reserve/ give away to family. Phew.

With all of us storing more and more things digitally, I can’t help but thing that most people are unlikely to shell out nearly 800 quid (once you’ve factored in everything and a bit more) to provide a terabyte of storage, but at the moment, a single drive failure (of a re-install of XP or Vista) would wipe out most peoples legitimate content (iTunes downloads for example) and family photos!

Possibly Related Posts:

• Home-brewed time-lapse fungus growing 101
• Getting Redis and CodeIgniter to play together
• Mushrooms, live!
• Train + wifi + bored + guardian api + unix tools = ….
• E by gum