The Importance of Backups

How many backups do you have of your important files? I have several backups of my photo archive:

* A local rsnapshot (incremental) backup made every six hours.
* A local copy of my photos directory, which is a straight copy, including deletions. Synced daily.
* A copy on another server synced daily.
* An online cloud backup on Backblaze.

And I still managed to lose a few days worth of photos.

Late last month I took delivery of a new 8TB external drive. I benchmarked it and it was just a bit better than any of my existing drives so I decided to make that the primary location for my photos. I called it “cats”.

Previously my photos lived on a drive imaginatively called “data”, but that would now become my secondary copy.

With my photo archive copied over (all 2.3TB of it), I edited my backup script so it would not accidentally erase my photos on cats, especially not any newly imported photos. I updated rsnapshot so it used cats rather than data, and made sure that Backblaze was backing up cats too. It all looked like it would work perfectly!

The next day I didn’t take any photos, or the next, I took a couple of photos with my phone but didn’t transfer them to my laptop immediately. Then one morning I took a photo of the sunrise, and the following day I did the same.
I edited a few of the images and uploaded one on the same day. Nothing wrong at all.
The next day I went into Cork to photograph Glow Cork and got some nice long exposure photos of the lights. I went in the next day on December 3rd and did the same.

I worked on a couple of shots, but I was busy, and didn’t have time to work on photos until this evening when my world fell apart as Lightroom reported it only had the smart previews for my latest photos. With a sinking feeling and forced calm I checked the 2017 folder. It all looked good, until 2017-11-19 that is. Nothing there after that.

I frantically checked my backup script again and again, poring over each line to make sure the right files were being copied to the right place but it looked ok.
I checked my rsnapshot folder. That should surely have the missing files. No. No, it didn’t.
I checked my other server and the files weren’t there either. They stopped at 2017-11-19.

I checked Backblaze, and the backup made today ended in 2017-11-19 too.

Dumbfounded and a little desperate I started clicking on the end date in Backblaze, going back a few hours, then a day, and another day when I found something weird. I found my December 1st and 2nd photos, but not in the cats drive, but in the data drive!

I’d forgotten to update my Lightroom import preset. Lightroom was still copying my images into the data drive, and then my backup script would delete them the next day. Rsnapshot wasn’t backing them up either because it was now looking at the cats drive.

I started a restore of my December photos off Backblaze. The 3.7GB of data is still downloading. Luckily I hadn’t formatted my camera SD card so my December 3rd photos were still there too, waiting to be copied over again. Once converted to DNG, copied into place, renamed and metadata data updated, Lightroom was happy with those files.

I learned a lot about Backblaze. Without it I would have completely lost my December photos, including this one and the photo at the top of this post.

However, I could not find backups of photos shot from November 20th to 30th. Most of those were sunrise photos, copied to the laptop after the daily backup script had run so Backblaze should have had a day to back those photos up. The restore process offers hourly backups over the last two days, then daily for the next 7 days, then weekly going back a month.

I suspect that Backblaze considered the daily backup done sometime around 7am UTC right after my backup script had deleted the files but that doesn’t explain how I was able to find my December shoot because that was past the hourly deadline. I need to contact Backblaze to find out why.

I still recommend using Backblaze. Last week several factors conspired to make my backup fail:
1. My DSL has a relatively slow 1Mbps upload.
2. I had my laptop unplugged from my external drives for several hours.
3. Backblaze doesn’t immediately back files up.

Then there’s the bad luck to find this problem after more than a week when Backblaze changes to weekly snapshots. It has it’s limitations but it still saved a bunch of my photos and I am very thankful to have it!

I’ve already changed my backup script so it won’t mirror deletions. If I’m going to delete files I’m going to do it soon after copying them onto my laptop and I remove files so rarely it’s not needed in the backup script.

The Automattic Photo, 2017 Edition

Last September almost 600 employees of Automattic travelled to Whistler in Canada for our annual Grand Meetup. As is now a company tradition a photograph was taken of everyone who attended. It’s a challenge taking a photograph of that many people. Remember how hard it was taking that family photo? The difficulty of getting everyone to look at the camera at the moment the shutter closed, or even just getting everyone in one place is multiplied when you have this number of people.

The last company photo was also taken in Whistler and we took a photo in the that same location again but it wasn’t good enough. The camera wasn’t up high enough, so people in the back were all but invisible in the photo. This photo was taken around the corner, where the photo was taken from a first floor balcony. Plenty of height and a perfect vantage point.

Taking the photo was one challenge, but then came development of the image. The camera had to be handheld because of the location on the balcony but the photographer took plenty of shots so I had a lot of material to work with.
The first issue was lining up the photos and I tried using Affinity Photo. Unfortunately it wasn’t quite up to the job (at least with the knowledge I have, it’s an excellent tool otherwise) but I learned a lot about working with layers and layer masks as I swapped parts of different images around.

Eventually I used Photoshop, loaded 3 photos in a stack and the auto align did a great job of correcting for changes in each photo.

What turned out to be the biggest pain point in developing a photo like this was removing people from the background. Here’s a few of them:

Blond Guy
This guy stayed in the crowd for about ten images. I only realised he wasn’t part of the group when I saw him with his friends in an earlier photo.

Two guys showed off their bikes for the camera. It was easy to tell they weren’t part of Automattic!

This lady thought it would be fun to join us in the photo. Her friend was laughing in the background and can be seen in the photo below.

This man stood mostly still throughout the photos that were taken making it more difficult to remove him, but he’s gone from the final image. πŸ™‚

I’m already planning for next year. We’ll get some shots of the background to make it easier to to clone bystanders out, and have more volunteers to help organise the shoot so we make best use of the available space.

Google Photos Overload

The Google blog announced that Google Photos will now allow you to share whole albums or just the photos of particular people with someone else.

I tried it this morning and it works, but there’s one big stumbling block. Too many photos.

When I take photos I’ll take 2 or 3 photos of the same scene sometimes, or if there are people in the photo I’ll take photo a few times just so everyone is looking at the camera. Or if kids are involved they’ll be looking every which way except at the camera.

So, what should really be one representative photo it’s really five or six images.

No. Using machine learning and image recognition is a huge step, but what a lot of non photographers want is a curated collection of “the best” photos.

I’m sure it’s only a matter of time before Google learns how to select from near identical photos to create this “best of” album. Their daily albums are a good indication that they can do this already.

The other problem with machine learning is that it might try too hard. This morning it suggested i share street photos I took a few weeks back. Unfortunately one of the suggested people was a former colleague who lives in Colorado and was not in Cork on the day. Another suggestion was my niece who also was not roaming the streets!

Anyway, I love Google Photos. The image recognition is amazing. I’ll search that for an image I need before diving into Lightroom 5 to edit it. πŸ™‚

How to take a photo with a camera phone

(mild spoilers for Designated Survivor)
.
.
.
.
.
.
.
.
This is how you take a photo of the US President and Press Secretary on the steps of Air Force One.

Taken from S01E19 of Designated Survivor where they have camera phones that see through fingers.

Forever McDonalds

Google Photos created an animation from a number of photos I made outside the McDonalds in Douglas last year around this time of the year.

Spots at f/22

It might be time to clean the sensor of my camera again. The circles in the photo above are the spots I removed in Lightroom. They’re caused by little specks of dirt or dust on the camera sensor.

Thankfully they’re really only visible when shooting with a closed down aperture like f/22. At f/8 I see nothing! The aperture of a lens describes how big the hole in the lens is that lets light in to the camera sensor or film. Paradoxically, small f numbers are big holes, so f/1.8 lets in lots of light, while f/22 creates a tiny hole and not a lot of light gets through.

Even if you never take the lens off your camera, you might still get spots on your sensor. A zoom lens has bits that go in and out. Air goes in and out and there’s a (tiny) chance that dust will get sucked in. Dust in the lens itself is nothing to worry about as it’ll never show in photos but if you shoot a lot at small apertures like f/22 you can clean your sensor.

It’s actually not that hard to clean the sensor. Last year I wrote a blog post on how to clean your camera sensor including a video and step by step instructions. I’ll probably get around to that this week again. Go have a look if you see spots, or go see a doctor if you’re not looking at a photo at the time …

MacOS Sierra and Lightroom 5

If you updated your Mac to MacOS Sierra and you use an old version of Lightroom you may get a shock when you try to import anything.

Lightroom Import Dialog

The destination and rename panels are missing from the sidebar! Luckily there’s an easy way to fix it, at least temporarily, thanks to The Lightroom Queen who figured out how. Right click on one of the panel headers and you can enable the missing panels again in the menu that appears!

missingpanels

Unfortunately the change doesn’t stick and the next time you import photos you’ll have to enable those panels again if you want to check those settings.

I can’t imagine Adobe will update LR5. I didn’t think Lightroom would start to break down so quickly after an OS upgrade as the app isn’t that old but I guess we’ll all have to jump on the Creative Cloud bandwagon sooner or later.

Automattic in 2016

Automattic

This is (most of) Automattic. We’re in the town of Whistler, Canada for the annual Grand Meetup. The company is growing every year, from tiny origins when a small group of us hacked away on a couple of servers to almost 500 now!

In the old days the company portrait was easy. The first one was us seated around a large table. I remember well the one in Breckinridge, Colorado. The whole company fit on the stairs leading up to one of the houses we had rented.

In recent years it’s been a challenge. This year Rose Goldman Simon and I scouted the town of Whistler for suitable locations. Rose had already been around and picked out several so I helped to narrow it down. We’d need:

  • Somewhere high up for the photographer to stand so as many people as possible could be seen.
  • Tuesday, the day this was made, was a lovely bright day, but that would play havoc with the photo. We needed a large shaded area.
  • It needed to be close to where we’re staying. The further away we had to walk the more complicated it would become.

We narrowed it down to two locations. One in front of a local store where there’s a nice courtyard and this location above. This won out despite the fact that sunlight would be shining on the camera lens but a large whiteboard helped keep the sun off. Ironically as people dispersed after the shoot, the shadows crept up past where the camera was which would have simplified things!

Final development of the photo was in Lightroom and Affinity Photo.

You’ll also find the photo on Matt’s blog, and shared by many of my colleagues elsewhere. I’ve also uploaded the full resolution image on Cloudup.com for your viewing pleasure.