Digital Media | Spotlight: Photography | Inside Lightroom | Blogs
XMP Sidecar Files - What's Missing?
It’s likely one of the first things you learned about Lightroom was its non-destructive nature, and that your work was stored in The Catalog. Since this was so contrary to the bulk of your past image editing experience, naturally your next question was, "Why aren't the changes just stored in the file?"
So you then learned about the proprietary nature of camera raw files, and the dangers of writing to files with undocumented internal formats. "But," you said, "I don't like that idea, I want my changes to be portable and compatible and secure!"
Enter Ctrl-S, 'Save Metadata to File'. "File, what file? You just told me I couldn't save to the file!" And then you learned about XMP sidecar files, that kludgy work-around that allows file based storage of Parametric Image Editing data, along with traditional image metadata, while preserving the sanctity of the original raw file.
So, you feel much better now, because you can point to an actual file, external to The Catalog, and if you can just keep track of it, you've been granted your wish of 'portable and compatible and secure!' Or have you?
I’m not going to talk today about what's inside that XMP file. I will say that, if you're interested, it's easy enough, and very instructive, to take a peek at the plumbing inside with almost any text editor. (Working with a disposable scratch copy is probably best.) No, what I'd like to talk about today is which Lightroom data is not included in the XMP file. Rest assured, all of your image adjustment data is included. But some very valuable workflow data is not.
What's Missing?
Virtual Copies: Nope, not included in XMP. Workaround: export 'em and reimport 'em as actual images. Workaround: Create Snapshots instead, snapshots are saved in XMP. I know some folks who refuse to use VCs, simply because they aren't implemented in XMP.
Develop History: "What? How can that be? Do you mean my changes won't be preserved?" The final state of the edited image is preserved in XMP, but the individual intermediate steps of the Develop History are lost. Again, if it’s important, save it in a Snapshot.
Collection Membership: If your image has been assigned to one or more standard 'dumb' collections, its membership(s) will not be saved in XMP.
Pick Flags: Nope, not preserved in XMP.
Stacks: These don't even really make sense in the context of individual files and are not saved in XMP.
A couple of other housekeeping details are lost as well. If you've used any of the Develop Panel effect off/on switches to disable an applied adjustment, the status of the switch is not saved. If you've saved metadata for an image while displayed at a zoom level larger than Fit view, its pan location in the display is not preserved.
I've encountered many folks who use the XMP sidecar files as a part of their backup strategy. I do it myself, but it's important to recognize what data is not conserved through this method. Next time, we'll look at XMP metadata and RGB file types.
(Credit where credit is due: While I’ve spent a lot of time tinkering inside XMP files, much of this information was confirmed in discussion at various Lightroom forums. Thanks to those who contributed to my education.)
Thanks Brad as this question is raised in the forums so frequently...
Thanks Brad this was really informative! Do you have the same degree of usefull info about saving metadata to dng-files? What metadata get saved to a dng-file when you press ctrl-S "Save metadata to file". Is this different than when you save to a XMP? What does the "Update DNG previews & metadata" do that ctrl-S don't? And last, is there any use of ticking the "Automatically write changes into XMP" (under Catalog Setting => Metadata) if you only have DNG-files? Is this eaven a bad idea?
Thanks in advance // Jani
Jani - it's precisely the same information. No difference whatsoever.
Good article, Brad, glad to see new life breathing into this blog.
Jani, Gio is correct, the saved information is the same for DNGs as well.
Answering your other question:
Updating 'DNG previews & metadata' is a little different. The DNG format includes a location to save a thumbnail image in JPG format within the file. This allows external image applications to show previews of the file, even if they aren't capable of fully rendering the DNG image on their own, for instance, Windows Explorer.
So, leaving AutoWrite XMP turned on does have an effect on DNGs in that your adjustment data is automatically saved to the DNG file. The preview updates however have to be requested manually(which also refreshes the XMP data.)
Good or bad idea? Personally I always leave autowrite XMP turned on. I like the security of the 'extra' backup. Since day one, there has been debate concerning the performance ramifications of this option. I believe with each new release, this has become of lesser concern.
Thanks Gio and Brad. I'm very greateful for your answers. Now I can rest assure that the metadata is uppdated in my DNG-files when I have "Automatically write changes into XMP" turned on. I don't have an performance issue with this option on. And I only have to use "Update DNG previews & metadata" to update the previews inside the DNG's.
I'm in the midst of a fairly big reorganization of my photos after installing a new hard drive and consolidating file locations.
I am using the Lightroom 2 Library to move the files on my hard drive. I noticed that when I select photos (filtered by date), then create a new folder on my hard drive moving the selected files - the CR2 or jpg files are moved to the new folder but the xmp files remain in the old folder.
As a test, I made some Lightroom 2 edits to both a CR2 and a jpg file, moved the CR2 and jpg, then deleted the xmp file from the old folder. The changes I made to each file were still intact after restarting Lightroom 2. Is the xmp file not necessary to keep the exif data and Develop module edits in Lightroom 2?
Darryl, the edits are always preserved in Lr's *.lrcat catalog database file. As long as that file and its links to the relevant image file are intact, the edits will be safe and sound. The embedded XMP or sidecar files are an added layer of security, and also provide some measure of portability of the edit data, but they are not mandatory.
How much larger would the xmp file be if it was able to store the additional information of say a single virtual copy together with its final settings. Can't be much of an increase and certainly no more than double. Such a feature would be a real bonus. Snapshots are cumbersome and you only know they are there if you look in the panel.
I've searched everywhere and can't find the answer, so I'll try here. I shoot raw and import into my Vista OS using a 9in1 multi reader. On import, I select to import using Photoshop Elements 5. Up until just a few weeks ago, everything was fine. I would get the raw file and an xmp sidecar. Now, all of a sudden, using the same card reader as always it doesn't allow that. I have to import and save to a location (I use dated folder in Windows Explorer). That, in itself, is not a problem. However, none of my raw files import with an xmp file any longer. The metadata is still there in the NEF file, but no sidecar. What am I losing? I can still edit the files, but I don't know if I still really have a raw file any longer. Can anyone help?