26 Apr 2017 The Mobile Photographer’s Image Management Strategy (Podcast 570)
It’s been two years since I explained my image management strategy as a traveling photographer, and I’m finding myself explaining what’s changed a lot in email conversations, so today, I’m going to walk through this with you again, and update you on the changes I’ve made.
First of all, allow me to explain the problems that I’m overcoming with my workflow, so that this all makes sense as we work through my thinking.
Problems to Overcome
One question I get asked about a lot, and why I often send people to this post, is how do I move smoothly between computers when I get back from a trip. People tend to make the process of getting images from a trip back into their main library a very painful process.
The good news is, if you build your workflow around the premise that you will travel, you don’t have to do anything special. I’ll go into details shortly, but basically I have to click one button when I get home after a trip, to initiate a backup of my images, and I’m done. In fact, I have to click that same button whenever I go to my desktop computer, so nothing changes. I literally transition between my laptop and desktop computer with zero effort, as I’ll explain.
Another problem people often come up against, is keeping track of what is backed up to where. I’ve found that it’s very important to decide which hard disk contains your working data, and which hard disks are just a backup. If you work on images in separate locations it soon becomes a real pain to keep them synchronized, so we build this into our strategy.
Finally, I think it’s vitally important that we have a multiple backups of our precious photos at home, as well as a copy in the cloud. Having everything under one roof could be a recipe for disaster, if indeed, disaster should strike. Should something catastrophic happen to your house or business premises containing all of your local backups, having the ability to contact someone to receive a backup of all your data could be the only possible way to rebuild your image library, as you rebuild your life.
Same Strategy, Different Software
One other major change over the last few years, is that I’m now using Phase One’s Capture One Pro as my raw processing and image management software. The details regarding what I did in Lightroom are still in the original post, so you can certainly still reference that post for details, and as you’ll see, most of what we’ll cover doesn’t really change depending on the software you are using in your workflow.
Move Catalog Drive Rather Than Synching Computers
The cornerstone of our digital workflow is our image catalog, or now that I’m using Capture One Pro, catalogs, in the plural. I have more than one now unfortunately. But, I’ve found that keeping track of multiple catalogs and keeping them backed up has not been a problem.
The important thing is that I keep my catalog on an external hard drive, and this needs to be relatively fast. A USB 3.0 hard drive generally won’t cut it. I’ve actually changed my hard drive twice since my fist post. I used a Drobo Mini over Thunderbolt, and then 4TB Western Digital Thunderbolt drive a long time, but as I bought a new MacBook Pro with USB-C ports at the end of last year, I picked up a Sandisk Extreme 900 portable SSD drive, and have been very happy with it. These are expensive drives though, currently retailing at $787 on B&H.
These external SSD drives over USB-C 3.1 Gen2 are incredibly fast though, and remove any and all stress related to running your image catalog and images on an external drive. The Western Digital thunderbolt drives were fast too, but nowhere near as fast as these SSD drives. The Extreme 900 comes with both a USB-C to USB-C cable and a USB 3.0 Type-A to USB-C cable, so I can plug it straight into either my new MacBook Pro or my older iMac. The speed is actually pretty respectable over USB 3.0 on my iMac as well. The problem with other USB 3.0 drives is that the 2.5 inch hard drives are slow, but that isn’t the case with SSD.
The downside, is that the largest available volume at this point in time is 1.92TB, so I had to rethink a few things. I can just about fit my Finals and current year of images on this drive, although it will be tight. I’ll explain this in more detail shortly, but the important thing to note here, is that I have all of the work that is important to work on at the current time on this drive, and I run my Capture One Pro catalogs from this drive. When I move computers, I simply unplug the drive from one, and plug it into the other. When I reopen Capture One, I’m taken right back to the location that I left off when I closed the catalog on the other computer.
When I moved to Capture One, I found that it couldn’t handle all of my images in a single catalog, so I split my images into multiple catalogs. Each year of images has its own catalog, except for the first six years, from 2000 to 2005, because I didn’t have that much work, and so I was able to fit this all into a single catalog.
So, I currently have one catalog called 2000-2005, and separate catalogs for each year from 2006 onwards. These year catalogs contain every image that I shot for each year. I do all of my initial editing and image rating in these catalogs, until I have finalized my selection. Once I have finalized my selection, I copy my images to another catalog, called Finals. This catalog contains a separate folder for each year. I also copy the physical images to a Finals folder with one subfolder for each year. I’ll cover this in more detail later.
Although I wasn’t happy about having to split up my catalog initially, in practice, it hasn’t been that bad. You can easily get to each catalog from a pull down in Capture One Pro (right) and I have all of the photos that are worth a hoot in my Finals catalog anyway, so most of the time I tend to flick between my Finals and the current year catalogs.
Master and Backup Copies
As I mentioned, I learned from experience that it’s really better to avoid having multiple copies of folders and catalogs that you work on, so it’s really important to decide where you are going to put your catalogs and folders of images, and decide which on is the master, and which ones are just for backup purposes.
If you work on a copy of your image library on one computer, and then work on a different copy of your image library on another computer, at some point you are going to wonder which copy is the most recent, and you’ll have forgotten. Even when using software such as ChronoSynch that we’ll look at later, which has the ability to synchronize the latest files between locations, there will come a point when you have two copies of the same file that have both been worked on, and when you select one copy, you throw away what you did to the earlier copy, or you keep both copies and that’s avoidable, so I prefer not to.
My Master and Backup Copy Strategy
For me, I’ve found it best to have my current years worth of images and all of my Finals on my Sandisk Extreme 900 SSD drive, and I have all previous years on a Drobo 5D, which is attached to my iMac in my studio. I never need to access my original photos from previous years while traveling, and I because I do travel with all of my Final selects, I can get to those if necessary.
Let’s map this out and start to visualize my strategy, starting with my desktop computer, in the studio. When I have my Sandisk Extreme SSD drive, which I call Traveller, attached to my iMac, I can see, open and edit every image I’ve ever photographed. We’ll build this out like a presentation slide deck, so excuse the blank space on the right side of Diagram #1 (below).
At it’s bare minimum, my workflow starts with shooting images, and transferring them to my Traveller drive. This drive contains the master copy of all of the current year’s images and all of my Finals, which is every photo I’ve ever made that I consider good enough to use, and my current year and Finals Capture One catalogs. These things all live on my Traveller drive, so that I can easily move this to my laptop, as we’ll see shortly.
I use an application called ChronoSync from Econ Technologies to synchronize my files and catalogs around. I used Robocopy when I was on Windows, but it’s not all that intuitive. Other Windows applications that were suggested following the last post I did on this are SyncBack and GoodSync, which is multi-platform.
With ChronoSync you can create synchronization jobs and bundle them together, and schedule a batch of jobs to run whenever a specific drive is attached to a computer. Let’s walk through this
I have two sync jobs that mirror my current year and my Finals folders to my Drobo. Here is a screenshot of my current year sync job (below) and this is simply going to copy everything new in my 2017 folder on my Traveller SSD to a 2017 folder in a folder called Photo Originals on my Drobo.
Using the Mirror option will also delete anything that I have deleted from my Traveller. This is important, because as I remove images from my main copy, I don’t want to leave them in my backup. I also create a Rule to not copy the hidden .DS_Store files to my Drobo. They are specific to each drive, so I don’t want them to be included.
I have a similar job to Mirror my entire Finals folder to my Drobo as well. I don’t Mirror just the current year of my Finals, because as I work on images, I sometimes change images from previous years, so I want to keep this all synchronized with my Drobo.
I also have two special jobs that synchronize only things that have changed inside my Finals and current year Capture One catalogs. To do this, turn on “Allow package file selection” when you are locating the drive and folder to synchronize, and then ChronoSync will treat the package files as a folder, and synchronize the contents.
If you don’t do this, ChronoSync will synchronize the entire package, and that would cause a very large file to be copied to my Drobo every time I sync, and it would cause the entire catalog to be unnecessarily backed up to the cloud every time I sync.
Group Jobs Together in a Container
Once I have all of my sync jobs created, I wrap them up in what’s called a Container, as we can see in this screenshot (below). After adding all of the relevant jobs to a container, you can click the Add to Schedule button and schedule these jobs to run automatically.
You can schedule sync jobs based on various actions, or simply have them run at a set time each day, but for this purpose, I select to run the job “When An Independent Volume Mounts” and this enables me to select my Traveller drive (below). I also select “Prompt user before running”. I want to be prompted, because I don’t necessarily want or need to synchronize my drive every time I plug it in to my iMac.
Once I have this set up, whenever I plug my Traveller drive into my iMac, I see a little popup like this (below) that asks me if I want to synchronize my Traveller with my iMac.
So, although it takes a little bit of time to set up, I can now with one click automatically backup my images and catalogs to my computer. Let’s continue to build out the slides to check where we are in our backup strategy (below).
We now see that as soon as I attach my Traveller drive to my iMac, my catalogs are automatically backed up to my iMac. I keep a backup of my catalogs on my iMac hard drive for a number of reasons. The first is because if I put them on the Drobo, it would take me longer to backup my Drobo, because I’d need to create separate ChronoSync sync jobs to avoid copy entire catalogs, as I mentioned earlier. The second reason for doing this is that I also set up Time Machine to backup my iMac drive, so that would be an easy way to get back to a working copy if anything went wrong.
We can also see from this diagram that as soon as any new images are copied to my Drobo, they are automatically backed up to my Backblaze account. Backblaze has been great, and for just $50 per year, you can get unlimited storage in the cloud. If I ever had some kind of catastrophic disaster that took out all of my local copies of my images, I could have Backblaze send me hard drives with my 12TB of data on them, and I’d be back up and running in no time.
Location of Files
Let’s also recap on where everything is now. The master copy of my current year’s work and my Finals library of images, and the working catalog for this work is all on my Traveller SSD. The master copy of all of my previous years work is sitting on my Drobo, and whenever I reference these images, I launch the catalogs from my iMac hard drive.
An easy way to look at this, is everything from previous years is based in my studio, on my desktop workstation. Everything that travels with me, is on my Traveller drive.
Local Fault Tolerance
Let’s move on to look at one last slide from my studio setup (below). Although my Drobo has fault tolerance built in, and I can have one drive fail without losing my data, there is always a slim but real chance that more than one drive dies at the same time, or that the entire device could die on me. Because I don’t want to rely on Backblaze sending me my work on hard drives just for a drive failure, I actually have a second Drobo 5D, which is a straight mirror of my first.
I have a reminder scheduled on my computer to remind me to turn on my second Drobo once a week, and mirror my first Drobo to it. I use ChronoSync for this as well. When I set up my first Drobo 5D I used drives that turned out to be very noisy, so I demoted that Drobo to the backup, and have bought quieter Western Digital Red drives for the main Drobo, as that’s turned on most of the time. They also use quite a lot of power running five 3.5 inch hard drives, so I only turn on the second one when necessary.
I know that some people have had bad experiences with Drobo drives, but I have been very happy with mine, and have never had any problems. But, technology does fail, so I just don’t want to have my main copy of all of my work to exist in just one place locally.
Mirroring Entire Drives with ChronoSync
To close the loop on the last diagram before we move on, please note that to mirror the contents of my first Drobo 5D to my second with ChronoSync Task, because we will mirror the root of the drive, I set up a few Rules to prevent ChronoSync from copying and overwriting some important system files, as we can see in this screenshot (below).
Let’s Get Mobile
OK, so now let’s move on and look at what happens when I’m traveling, or simply working away from my studio. As you can see, I just plug in my Traveller SSD drive, and continue working. If I had actually closed down Capture One Pro on my iMac with that Snow Monkey photo displayed, it would have opened at the same location when I move to my MacBook Pro.
I also travel with three USB 3.0 hard drives. These are too slow to run my catalogs and images from, but as backups they work fine. One is just a Time Machine backup, that I plug in at the hotel every few days, usually over night, if I can leave my MacBook Pro plugged in to the electricity.
Carry Spare Backup Drives
I then have two backups, which I once again automatically synchronize with ChronoSync. I have a schedule set up to detect each drive as it is attached to my computer, and it asks me if I want to mirror my images and catalogs to the hard drive. I make two backups, simply because one drive could fail. Actually, all of my drives could fail, but having spent many months on the road, over the years, I’ve actually had just one drive fail on me. That was in Antarctica though, and believe me, when there are no shops around, having a spare is very important.
It’s a little nerve racking to only have one master copy and one backup of my images while traveling, but if my second backup drive was to fail, I’d kill my Time Machine backup, and continue to make a backup of my images. Thinking of it this way, carrying these three drives is more to give me backup drives, than an actual backup of my images, but as it’s all automatic once I’ve plugged the drive in, I just keep them up to date at the end of each day.
Keep Your Copies Separated
One other important aspect of having these backups, is that I feel it’s very important to keep these separate as you travel. I always travel with a photographer vest, and keep my master copy in my vest, on my person, at all times. Even when I go to the bath when traveling domestically here in Japan, I take my Traveller drive with me and put it in a locker.
Having three backups of your images isn’t going to help you at all if they are all in the same bag, and you lose that bag. I generally keep one backup in my bag, and the second in my suit case. These means something would have to happen to all three copies in separate locations for me to lose my entire library of images while traveling.
Turn Off Cloud Backups While Traveling, Please!!
I don’t do cloud backups while traveling, partly because I only pay for one computer on Backblaze, and it’s better to make that my iMac as it’s always on and connected to the Internet. Also, hotel Wifi is usually not good enough to bear up to uploading large numbers of raw files. Many people have automatic backups turned on now, and you can literally watch the network go down as a bus load of photographers get to their rooms after a days shooting. I wish more people would turn this off while on the road.
Synching Settings Files
One other thing that I need to mention before we move on, is that to make moving between computers totally seamless, I also synchronize my Capture One Pro settings folder by moving it to my Dropbox, and creating a symbolic link in the original location. This isn’t officially supported by Phase One, but I’ve been working this way for 10 months now, and haven’t found any problems.
Here is the code I use with my name replaced by USER_NAME. This assumes that you’ve moved the “Capture One” preferences folder under the “Application Support” folder to a folder called “Capture One Prefs” inside a folder called “Capture One” in your Dropbox. This code only works on a Mac, and please do this at your own risk.
ln -s "/Users/USER_NAME/Dropbox/Capture One/Capture One Prefs" "/Users/USER_NAME/Library/Application Support/Capture One"
You of course have to do this on all computers that you will work on, to ensure that your preferences are copied between each via your Dropbox. If you don’t know how to create a symbolic link in Windows, this tutorial will probably help.
Set a Hard Drive Letter for Windows
If you use this strategy in a Windows environment, you’ll probably also need to ensure that the drive letter of your Traveller drive doesn’t change as you move it from computer to computer. Here’s another tutorial on how to do that. Just ensure that you select the same letter on all computers you work on. Give yourself some room too, so that you can still have lots of dynamically lettered drives on your desktop. T for Traveller would be a good option.
Not Really a Cross Plastform Solution
I should also mention that this solution may not ideal if you switch between Windows and Mac regularly. The catalog can be taken from one operating system to the other and will open, but Windows and the Mac OS reference drives differently, so you’d need to tell the other OS where your files live each time you open the catalog on the other system.
Once you have all of this in place, you will literally be able to move your Traveller hard drive from computer to computer, and continue working as though you were on the same computer. Because you have your Capture One Pro settings syncing too, even all of your presets are available on both computers. They essentially become identical.
Exporting Original Format Images
Let’s start to wrap up now, with a few other pieces of advice based on my own workflow.
To get my final select images from my original photo folders to my Finals folder I select the images that I want to export and right click one of the thumbnails, and from the shortcut menu, select Export > Originals. You can also get to this option from the File menu.
I don’t change the image name on export, because I change it on import. After checking the destination, I ensure that Include Adjustments is turned on, then click the Export button, as you see in this screenshot (right).
I don’t package my images as EIP or Enhanced Image Package format files, because the thought of wrapping my images in something non-standard scares me. I just want my raw images in a new location, that’s all.
Synchronize Finals Folder
Once the export process has completed, I switch to my Finals catalog, locate the folder for the year I exported my images to, then right click that, and select Synchronize. Capture One will then go and look for anything new in my current year folder, and import them into my Finals catalog. As long as you turn on the Include Adjustments checkbox on export, any changes made to your images will also be applied to your new copy.
Starting a New Year
At the start of each new year, I have a little bit of cleaning up to do, to prepare for starting to photograph the new year. First of all, I ensure that I have completed all edits that I want to do on my previous year’s images, and ensure that I have run my backup to mirror these images to my Drobo.
Then, I close the catalog in Capture One Pro, and delete the folder from my Traveller drive. After that, when I reopen the catalog in Capture One, my folders all show up as missing, as you can see in this screenshot (below).
To fix that, and complete the process, right click the top level drive or folder, and select Locate from the shortcut menu. You’ll then be able to navigate to the copy of your year folder that was your backup copy until a few minutes ago. After spending some time locating all of your images in the new location, you are ready to continue to use your catalog. From this point on, this becomes your master copy, along with all of the other previous years.
Create a New Year Folder
And of course, you also need to create a new year folder to ingest all of the new work that you’ll make. Remember that this will live on your Traveller drive for the current year, along with your Finals, if that’s how you work, and you’ll just proceed as you did in the previous year. It’s all quite easy once you have gotten your head around it.
Having spent many years tweaking and developing a smooth workflow, I’m very happy with how I work, so I hope this helps you to smooth out any possible kinks that you might have in your own workflow. As I mentioned earlier too, if you don’t use Capture One Pro, the techniques and strategy that I covered should be pretty transferable to whatever program you use to manage and edit your photographs.
Sandisk Extreme 900 SSD: http://mbp.ac/sd192TB
Western Digital Red Drives on B&H: http://mbp.ac/wdr8tb
Music by Martin Bailey
Subscribe in iTunes for Enhanced Podcasts delivered automatically to your computer.
Download this Podcast in MP3 format (Audio Only).
Download this Podcast in Enhanced Podcast M4A format. This requires Apple iTunes or Quicktime to view/listen.