Digital Media | Spotlight: Photography | Inside Lightroom | Blogs
A New Years Storage Resolution
New Year's is always a good time for resolutions. Sometimes, the best resolutions are the small approachable ones that solve big problems in your life. For example, one of my big problems right now is storage for my photo library. For most of 2007, I've been limping along with my previous system while dreaming big about what I wanted to do next. For some reason, I've been wanting to keep all of my photographs on a single logical volume and since my total storage needs are right at the size of the largest single disks available, I thought that some sort of RAID system would be in my future.
The problem is that the moment you go into RAID storage, things get more expensive and more complicated, and they do so very fast. I'm no stranger to this land having set up many a Xserve RAID in the last few years for my consulting clients. And the number one lesson that I've learned is that RAID is not the same thing as backup. Even if you have a monster array of drives, you still need backups. And, if you've got a 3TB RAID, then the easiest and simplest backup is another 3TB RAID. Then, what if you want to have an offsite backup? Carrying a RAID array down to the bank's safe deposit box isn't very practical. There are solutions, but they're a bit out of my current budget. And, worst of all, when you run out of space on a 3TB RAID, there are very few solutions will currently allow you to gracefully and incrementally grow.
Not having a five-figure budget to dedicate to solving the storage issue, I've been waffling along keeping my eyes on options as they appear and dreaming of the day where ZFS is a realistic option under Mac OS X without having to setup a rack of equipment and servers in the back room. I've also been watching out for Drobo to release a version of their very interesting storage robot in a form that provides high speed access, say via a nice fast eSATA connection. In the meantime, I've been making do and shuffling things around as needed without really buying anything new and committing to a solution. As of right now, however, I'm out of time. I need a solution that works and that will scale linearly over time.
So, my New Years resolution is to implement a storage solution that works, provides on-site and off-site backups, and does so for my data needs for 2008 and 2009 at under $2k. To do this, I'm going to take advantage of the fact that you can get 750GB drives, both external and internal, fairly inexpensively these days. And, some of the external drives available from Seagate and others come with eSATA connections which means that you can access data on external drives at high speed and economically. Most of all, I'm going to give up the dream of holding all of my photos on a single logical volume. It was a good dream, but it's time to just get real and move on. I've got a plan and a course of action that should be easily implemented.
I'll report back in the next week or two how things turn out. It should be interesting. In the meantime, if you've hit the wall with simple single disk storage solutions, and have solved it in an interesting way, please leave your approach in the comments.
Comments (13)
Not something I've done yet, but one of the local photographers I know uses Amazon's S3 service as a remote backup for his photographs. At last tally, he had 30 GBs stored there and paid less than $5. Not sure if that's per month.
check readynas by infrant (now netgear). similar to drobo but a network solution. pro and cons of that. slower than USB, but accessible from everywhere....
I'm researching S3 and think I will be using it in the very near future. My nephew has an S3 account and uses Jungle Disk https://secure.jungletools.com as a front end tool.
It looks really slick and you can't touch the price or the uptime and the fact this is available from anywhere.
I'm interested in how things turn out. With photos spanning 4 drives, I need to, at minimum, consolidate the Library somehow. I don't mind if I get down to 2, but I also need to get everything back to a date sorted order (which I've been doinf manually for a while now).
You should realy try Windows Home Server.
It's made for this!
We have installed about 25 of these at our client sites:
http://x-equals.com/blog/?p=8
Amazing little device, and can be made accessible anywhere via ftp. In addition to its outbound messaging capabilities to warn of error conditions and "drive close to full" thresholds, it's a steal at $2k.
Thanks for this - I am keen to see how you go as I am in a parallel position and have been umming and ahhingh about RAID, and steered clear of Drobo as proprietary right? (i.e. not an open system??).
I'm curious whether you've considered a Linux-based fileserver with an LVM filesystem to create the "single logical volume" you want. Is this just too slow to meet your needs? Too complex? It's certainly not too expensive.
Open Solaris/ZFS is awesome!
Why donʻt you use an open solaris server? Just buy an off-the-shelf workstation from Dell or better yet build your own and stuff the sucker full of 1TB drives.
Create a fat zpool raid, slice it up using quotas and compression, configure snapshots, and install an apple share server for your mac and samba for your PCs (iʻm thinking youʻre not that bad off).
Itʻs not that hard, and the documentation is everywhere.
Both our 2 ~6TB Xserves are collecting dust du to the arrival of our Thumper (Sun x4500).
What I really canʻt wait for is iSCSI support from apple!
Brooks, Anthony: S3 and other solutions like Strongspace have long been interesting to me. The problem is that I've got right at 1TB to backup. And growing. Much of it could be pruned, but I haven't had the time as of it. :) 30GBs is reasonable price wise, 1TB, not so much. Also, there's the transmit time.
alberto: I've thought about going networked with a ReadyNAS or other network storage solution. The problem with those is that they are slower than local solutions. They'd be find for primary backup where the data flow is mostly one way and not in the critical path, but they're not a place where I can put primary storage onto.
Christer: I've actually thought about Windows Home Server.. But I've got the same concerns with it as any other home server solution.
Jay: I've not considered going the drobo route in the short term not because of proprietariness. I actually really love the idea. It just that I know several people with them and they're not speedy at all. Again, a fine solution for a backup drive, but not as primary storage.
About setting up a home server--I've considered it. I have several concerns with that. The first is just speed. Most of the server solutions aren't super fast. And if you want fast, you do pay for it. The second is that I don't need another machine to administer. I've spent the last 20 years of my life tinkering with too much hardware, including running my entire Internet presence at home for a decade. Anymore, I need fewer machines to admin so I can spend more time taking photos. :)
Third, and most importantly, I've been trying to consume less power in my house. Setting up a home server goes against this desire.
On the other hand, if I could put a Thumper in with a fat pipe connection to it, I'd love it. The problem with that is that I don't have a server closet to fit it in and my office is at home. Maybe in the next generation of hardware or so. And yes, iSCSI in the MacPros would be nice. As would 8 drive bays. :)
As far as trying to do it on the cheap, it's a necessity of sorts. I've already got a MacPro in the budget for '08, and possibly a Canon 1Dsmk3. And there's lots of other things going on. So, if I can do a safe setup at a reasonable price, I'll take it.
Anyway, I'm in progress right now with moving things around on some new primary disks and setting up somethings. More to report later.
Keep the ideas flowing! :) And thanks!
I use a Sun Ultra 40 (slightly under $2K when bought using the Sun Startup Essentials program). It is expensive for a PC, but it has 8 hard drive slots and most importantly it is very quiet (it runs in my living room a few yards from where I sleep, and I can't hear it) and power efficient. I stuffed it with 6 750GB Seagate SATA drives in a RAID-Z2 (dual parity) setup.
RAID + ZFS is a winning combination, as snapshots are the real backup, and they cost almost nothing to take. I backup my Mac to it using rsync over Gigabit Ethernet, it takes less than 5 minutes to perform an incremental backup (my backup set is about 500GB in size).
I then perform additional offsite backups to a repurposed old PC with 2x400GB PATA drives, also running Solaris. Even on pokey DSL upload speeds, you can transfer about 1GB/night of incremental backups, so my offsite backup is at worst trailing by a few weeks. As a third layer of redundancy, I have a DND-323 NAS with 2 WD 1TB SATA drives in the same place as the offsite backup machine, and to which it rsyncs over NFS (the DND-323's CPU is not really up to high-volume transfers over SSH).
I hear you. I keep it super simple, and it works fine for me. I have 1.5 TB in internal drives for local work and swap space (right now everything fits on those two drives). I keep that backed up on a variety of external drives (USB2/FW400) that I actually keep offsite. I also keep one external drive connected at all times as my kind of "real time" backup. Whenever I do a ton of work, I make a copy to the external that's always connected. Then, every once in a while, kind of depends on the amount of files accumulated from the last time, I bring one of my offsite drives back to the house, copy anything new to that drive, and wipe that stuff off the "permanent" external drive. Then, it goes back offsite, and my in-home backup external is freed up to go back to work.
With drives becoming cheaper and cheaper, it seems just buying another simple cheap external drive is an easy way to go, and it works, and although a bit complicated, not really. A server or nas comes with it's own admin headaches, cost, etc. and being on a network kind of sucks if you're talking loads of big files. USB2/FW is bad enough. And, I kind of don't trust the drobo, although I have no reason to believe there's anything bad about it, I just trust a cheap, simple, system more for some reason.
Once I run out of space internally, I'll have a bigger problem, but it'll probably just mean taking some of the files "offline", and doubling them up on external drives. I'm hoping to hold out just long enough until some major breakthrough happens in storage. I bet in the next five years, some amazing things will no doubt happen in this area. We'll see.
How about the Apple X-Serve RAID it would fit with your platform?
Personally I just use many disks with a sync rather than a RAID, but disk sizes need to keep growing!
It terms of backup media it sounds like Blu-Ray is going to win the game for HD - and you can fit 25-50GB on one of those which will be handy!