Digital Media | Spotlight: Photography | Inside Lightroom | Blogs
Storage Solutions for Photographers
In last weeks blog I asked for feedback and the topic with the most requests was storage solutions. So I thought this week I would cover my storage solution for my images and business files. Back in November 2007 I covered my main back up solutions but I will elaborate here.
About a year and a half ago I did some research on storage solutions because I wanted to find the most robust and least expensive method to back up my images and business files. I found that the cheapest method was to burn multiple DVDs but that, while cheap, was also time intensive and didn't lend itself well to accessing images. The second least expensive method was to buy multiple hard drives and either mirror the drives or manually make sure they contain the same data. Last was the RAID 5 option which was the most expensive by far and even more expensive to back up. Basically with RAID 5 the big question is how do you back it up? The answer is another RAID 5 or multiple hard drives - both of which are not inexpensive.
There has been a lot of talk about RAID 5 solutions and while RAID 5 is robust and more secure than a single hard drive, it is not a back up solution on its own. In my experience, a minimum of three backups is ideal. And since I wanted to create the most robust back up strategy I have opted to put all of my images on two hard drives and also burn them to DVDs. The DVDs go offsite just in case of a natural disaster or the office burns down. My DVDs aren't in another state - just across town at a colleague's office. We trade DVDs so we both have offsite storage. With three independent copies of my images I am covered quite well. This past year I had two hard drives fail and had to go dig up the DVDs - thank God for those DVDs.
The hard drives I use are chosen for their speed and portability. All of my images are stored on internal Seagate Barracuda 750 GB SATA drives inside my main Apple MacPro. The Backup drives are Seagate 3.5-inch Pushbutton 750 GB External Hard Drives that plug in via Firewire. With this set up I have a super fast SATA drive that can be used to access images quickly and a fairly fast Firewire connection for the secondary drives.
My images are downloaded to the drives from a working drive and then never altered. I might save new versions of images to the drives but in general photo shoots are loaded onto the drive and then left untouched. This solves any fragmentation issues and keeps the drives working well. The drives are only used for accessing images and I load up to 80% of the hard drives capacity maximum on each hard drive to avoid failures.
My main reason for using SATA drives is the connection speed. Firewire is decent but pretty slow in comparison to SATA. USB 2.0 in real world usage is ridiculously slow in my experience and I avoid USB drives like the plague. I wrote an article last year on Five Ways to speed up Lightroom where I talked about how the connection speed of the hard drive can slow Lightroom down - hence I like to reference images from SATA drives as those have the fastest connection speed available.
The secondary back up drives are Firewire 400/800 which is perfect because my back up computer is a laptop and if for some reason I need to access my images on the laptop I can use the Firewire drives - my laptop does not have a SATA connection port.
Now I realize there are a million different ways to back up your images. I have looked into NAS (Network Attached Storage) as well but the connection speeds are so slow that it was more of a nightmare than I was willing to deal with. An Apple Xserve RAID is another great solution but it is quite pricey and it still needs to be backed up. At the moment I have 4 TB worth of drives which is not overwhelming to deal with since they are all 750 GB or 1 TB drives.
In the future I would love to get the Drobo system - that seems like a sweet setup but the huge problem is that it is currently USB 2.0 only and that is just too slow of a connection speed to be productive for my needs. If they come out with a Firewire 800 or SATA connection then I would be first in line to get one. It seems like an easy RAID 5 system that is perfect for most small businesses.
Of course there is a ton of great information online about the different backup solutions available. One of the best sources of information I have seen is Apple's Storage Solutions primer. The primer lays out all of the options available quite nicely and then it is up to you to decide which is the most cost effective and robust for your needs.
One last tidbit, on the topic of hard drive choices, I have talked with a lot of IT and network specialists, and all of them seem to say that Seagate is the most reliable brand. All hard drives will fail at some point. But in my experience I have found Seagate drives to be very robust and have never had one fail on me yet (knock on wood). For external drives I also like the G-Tech brand quite a lot. They are certainly not the cheapest but they are very well built. I have seen many other brands fail - multiple LaCies, a Western Digital and a Hitachi drive. I am sure that all of us have different experiences but this is just one lone voice giving you the straight skinny on his experiences. I would suggest that you seriously consider how the hard drive is cooled as this affects lifespan more than just about any other factor. If there is no fan or a decent cooling system in your external drive then start looking elsewhere.
That's it for this session. See you next week.
Adios, Michael Clark
Comments (10)
Is there any scientific basis for using 80% of the drive? This sounds a little like an urban legend.
I like Seagate also, I've had much more hassle with other brands.
Only filling up the drives to 80% is not going to affect the reliability in any way. It's heat, usage and aging that cause drives to die.
Drobo is not a RAID-5 solution. It is a proprietary solution which ensures that you can lose a disk at any time and not lose any data. RAID-5 is similar, but requires 3+ disks of equal size and spec's. Neither solution protects you from the loss of 2 drives simultaneously, but in both cases, losing one drive and then replacing it quickly will result in no loss of data.
Hi Michael,
If you want to go Drobo but don't like the USB 2 option (as many don't), you might be interested in Droboshare:
http://www.datarobotics.com/products_droboshare.aspx
It was literally just released and adds (among other things) gigabit ethernet connectivity, essentially making the Drobo a NAS.
I know you mentioned you didn't like NAS solutions because of their speed, but Gigabit ethernet for simply transferring your images around should be fine.
Personally I have all my photos on my local drive, synced (with rdiff-backup) over night to RAID-5 (which is not that expensive if you go with slightly older hardware) on the local network and replicated to Amazon S3 (with s3sync). Works like charm.
Don't forget that it's not a backup if you keep all copies at the same physical location IMHO.
ZFS file system is something that may be helpful for storing large amounts of data in a year or two.
http://trac.macosforge.org/projects/zfs/wiki/whatis
Wohoo - talking about storage solutions always seems to get some comments.
Well, Perhaps the 80% is urban legend - I don't know - that is just what I have been told by the networking specialists that I have talked to.
And yes, I realize the Drobo is not a RAID 5 apparatus but it pretty much simulates one exactly so pardon the syntax.
Rob, thanks for the info on the gigabit ethernet Drobo connection - interesting but still pretty slow - I can wait.
And yes, I did say that I have the DVDs off site. Thanks for the comments - keep 'em coming.
80% doesn't really have much to do with the reliability of a drive. Drives can still be reliable as they get full.
However, as you fill drives up above 80%, they will start to get slower. As the drive goes above say 90%-95% capacity it will start to slow down significantly. Maybe not a huge deal for a backup drive, but certainly for a drive you're using all the time.
Partially this can be because drives get more fragmented as they fill up, but mainly the issues are due to physics -- drives are faster near the outside of the platter. This is because they spin at a constant rotational velocity (typically 7200 RPM for a desktop drive), but the linear velocity on the outside is substantially higher. Also, as drives fill, the heads must move along the entire radius of the drive to seek data, while when they're less full the heads movement is constrained to the outside of the disks.
Personally I think that mastering dvd is pretty useless if you are a pro, because it's after a couple of year you need to do backup job again, considering very short life of a dvd.
About Drobo: ok it's great, but is too much expensive if you want to have cheap solution...
If you want to ensure a very secure backup you could consider services like Photoshelter, unfortunately VERY expensive but so secure. The only problem with this solution is a really slow connection speed...unless you work in a IT farm like a friend of mine with 4Gb/upload.
Personally, I've just installed a RAID 1 (hardware) solution in an old ubuntu machine, where one 320gb hd is mirrored on another one, real-time; this pc is connected through my router so it can act like a file-server or used with sftp. After this, I'll copy a "best of" raw gallery directly on a usb drive, just in case. For current year, I'll use my notebook hd to work on actual project, making backups of them every time.
2 320 hd: €160
1 raid hw: € 20
1 160 hd + box: € 90
if you think that cables, raid hw and 160gb hd were given to me for free from friends, ok it's very cheap and secure!
Bill -
Thanks for the explanation - that makes perfect sense and kind of upholds the 80% rule when one considers operational speed. Good stuff....
Cheers, Michael Clark