The Importance of Backups

How many backups do you have of your important files? I have several backups of my photo archive:

* A local rsnapshot (incremental) backup made every six hours.
* A local copy of my photos directory, which is a straight copy, including deletions. Synced daily.
* A copy on another server synced daily.
* An online cloud backup on Backblaze.

And I still managed to lose a few days worth of photos.

Late last month I took delivery of a new 8TB external drive. I benchmarked it and it was just a bit better than any of my existing drives so I decided to make that the primary location for my photos. I called it “cats”.

Previously my photos lived on a drive imaginatively called “data”, but that would now become my secondary copy.

With my photo archive copied over (all 2.3TB of it), I edited my backup script so it would not accidentally erase my photos on cats, especially not any newly imported photos. I updated rsnapshot so it used cats rather than data, and made sure that Backblaze was backing up cats too. It all looked like it would work perfectly!

The next day I didn’t take any photos, or the next, I took a couple of photos with my phone but didn’t transfer them to my laptop immediately. Then one morning I took a photo of the sunrise, and the following day I did the same.
I edited a few of the images and uploaded one on the same day. Nothing wrong at all.
The next day I went into Cork to photograph Glow Cork and got some nice long exposure photos of the lights. I went in the next day on December 3rd and did the same.

I worked on a couple of shots, but I was busy, and didn’t have time to work on photos until this evening when my world fell apart as Lightroom reported it only had the smart previews for my latest photos. With a sinking feeling and forced calm I checked the 2017 folder. It all looked good, until 2017-11-19 that is. Nothing there after that.

I frantically checked my backup script again and again, poring over each line to make sure the right files were being copied to the right place but it looked ok.
I checked my rsnapshot folder. That should surely have the missing files. No. No, it didn’t.
I checked my other server and the files weren’t there either. They stopped at 2017-11-19.

I checked Backblaze, and the backup made today ended in 2017-11-19 too.

Dumbfounded and a little desperate I started clicking on the end date in Backblaze, going back a few hours, then a day, and another day when I found something weird. I found my December 1st and 2nd photos, but not in the cats drive, but in the data drive!

I’d forgotten to update my Lightroom import preset. Lightroom was still copying my images into the data drive, and then my backup script would delete them the next day. Rsnapshot wasn’t backing them up either because it was now looking at the cats drive.

I started a restore of my December photos off Backblaze. The 3.7GB of data is still downloading. Luckily I hadn’t formatted my camera SD card so my December 3rd photos were still there too, waiting to be copied over again. Once converted to DNG, copied into place, renamed and metadata data updated, Lightroom was happy with those files.

I learned a lot about Backblaze. Without it I would have completely lost my December photos, including this one and the photo at the top of this post.

However, I could not find backups of photos shot from November 20th to 30th. Most of those were sunrise photos, copied to the laptop after the daily backup script had run so Backblaze should have had a day to back those photos up. The restore process offers hourly backups over the last two days, then daily for the next 7 days, then weekly going back a month.

I suspect that Backblaze considered the daily backup done sometime around 7am UTC right after my backup script had deleted the files but that doesn’t explain how I was able to find my December shoot because that was past the hourly deadline. I need to contact Backblaze to find out why.

I still recommend using Backblaze. Last week several factors conspired to make my backup fail:
1. My DSL has a relatively slow 1Mbps upload.
2. I had my laptop unplugged from my external drives for several hours.
3. Backblaze doesn’t immediately back files up.

Then there’s the bad luck to find this problem after more than a week when Backblaze changes to weekly snapshots. It has it’s limitations but it still saved a bunch of my photos and I am very thankful to have it!

I’ve already changed my backup script so it won’t mirror deletions. If I’m going to delete files I’m going to do it soon after copying them onto my laptop and I remove files so rarely it’s not needed in the backup script.

3 months free Backblaze again

A long time ago in 2013 Backblaze had a “sign up and get an extra three months free” offer and they’re back at it again.

For the next 32 days. click on this Backblaze signup link, you’ll get started on a free trial for a month to test it out, but if you decide to pay you’ll get an extra three months.

I love the Backblaze service. It’s a cloud backup service where you can backup your files. I won’t go on and on about it. I did that in my 2013 post, but if it’s any indication of how good they are, I’m still a customer all these years later with over 2TB of data backed up there now.

The signup link is an affiliate link of course (I get three months free as well if someone signs up) but if you don’t want to use it and don’t want that extra three months sign up through this link instead.

And to think, I used to rely on backing up my photos to CD..

Ages …

Time Machine Initial Backup

Well, look. At least it’s not like in Windows where it tells you it’ll be done in ten minutes and then five minutes later it’s gone up to twenty ..

I expect this will take a few days.

Restoring From Physical Media

dancing in the street

Remember DVDs? Last night while looking for the original photo of this dancing in the street photo I was horrified to find an empty directory!

Empty Directories

I had no idea when they were deleted. I searched my backups but of course the directories had been synced a long time ago and they were gone. I checked Backblaze and there was no sign of them. I even checked Google Plus for their backups but still no sign of them. I must have deleted them more than two years ago.

I hate that. I have multiple copies of every photo just because this might happen and yet it wasn’t enough! My backups sync every night so any deletes were synced within twenty four hours.

Almost. Later, I remembered that I had backed up photos to DVDs before I started using multiple drives and Backblaze. They were up in the attic!

DVDs

It didn’t take long to get them, the first DVD worked and my photos from The St. Patrick’s Festival in 2006 were restored, but the photos from April 20th were on the second DVD. The second DVD didn’t work. Nooooo!

The rest of the photos did survive their stay in my cold, damp attic and were restored. I’m thinking now about monthly offline backups to a drive I’ll plug in occasionally.

Do I have any photos from April 20th, 2006? I remember the day well because I was in St. Finbarr’s Hospital with my wife helping a family member. While the others waited I went outside to take photos of a derelict building at the back of the hospital grounds. People passed me going to work while I was take pictures of the daisies, flowers and old architecture. Soon enough a security guard showed up but he just asked me to stop taking photos. I do have a few photos. We called to a friend later in the day and I got some shots of her cat so that’s here too. 🙂

What’s the take home message from all this?

  • You can never have too many backups.
  • You need versioned backups (of a few days at least) in case you notice a mistake a few days later after your backups have been synced.
  • You need an offline backup that’s synced periodically.
  • You need cloud backup that will keep your files safe for a length of time after they’re deleted. Backblaze holds on to your files for thirty days after they’re deleted!
  • When you export files from Lightroom, Photoshop or whatever graphics app you use, always make sure you export a full size original version, not just a web version.

I’m sad that those files are gone, but glad I have a few reminders of an eventful day.

Fill and span DVD archives with Discspan

I have a huge archive of photos. I shoot tens of thousands of photos every year. Storage requirements for all those photos was bad enough when I shot in Jpeg but then I switched to RAW and space usage jumped! Here’s what the last 3 years looks like:

169GB of data is a lot of stuff to store. Originally I had them all duplicated on two external drives but then I bought a 500GB internal drive for my laptop for speedier access. Unfortunately that drive simply wasn’t big enough. I need to convert some of my RAW files to Jpeg to save space. To preserve the original RAW files I want to archive them somewhere permanently. I have a DVD writer so that was an obvious choice.

Burning data to lots of DVDs is tiresome. You can use tar, zip or another archiver to split the data but then you have to run through all the DVDs to pick out a file to restore. I like having the files directly accessible but that means endless selecting files, making sure they’re as close to the DVD size as possible, burning them, moving on to the next bunch. In the bad old DOS days I had a program to fill floppy disks if you pointed it at a directory but I’ve spent years searching for a similar Linux script. Last week I found one.

Enter Discspan. My 2007 archive was already burned to DVD, and I wish I had this script while doing it. I’ve burned my 2008 archive with Discspan and it was a doddle. Point it at the right directory, feed it some details about the DVD drive and let it go. 26 DVDs later and my 2008 archive is safe on DVD!

The script scans the directory, figures out how many DVDs are required and it fills each DVD with data, spanning my digital archive over multiple DVDs.

Be aware when using it that you should let Linux detect the next blank DVD before pressing return. The first time I ran it the script bombed out when growisofs didn’t see media to write to. You also need to patch it because it doesn’t detect the right size of DVD+R’s but it’s a simple one-liner.

Another Linux project, Brasero promises to span disks too but it didn’t for me. It’s the default CD/DVD burner in Ubuntu now and it’s a shame this functionality is broken in it.

Hopefully Brasero will be fixed for the next release. I’d offer to help but my C/C++ is very rusty.

Backups save the day

Marina City, Chicago A few weeks ago I blogged about my backup system. How I have two 1TB Iomega external drives and how one drive is a duplicate of everything on the other drive, and how I backup everything on my laptop and VPS accounts. It sometimes seem excessive but I’m paranoid.

This morning I’m very glad I went to such lengths. I wanted to copy some stuff onto my Macbook, and there’s nothing like the bandwidth available from a directly connected disk. I unmounted my drive, at least I tried. Something was keeping it mounted. Instead of following my own advice and checking what program was keeping the drive busy, I used “umount -l” instead. Turns out it was Rhythmbox, but I didn’t realise that until later.

Anyway, I disconnected the usb cable, plugged in the one connected to the Macbook (BTW – Ext2 for Mac OS X is useful for reading ext2/ext3 filesystems) and kaboom. The light on the external drive went out. Oh oh.

Long story short, the drive refused to mount again on the Linux box for several minutes. Eventually it did, but with errors. I’m running fsck.ext3 on it but it’s giving me tons of errors and won’t run automatically. I need to buy another drive this morning.

So what’s lost? My 8 years of photos? All the family videos shot over the last 2 years? My mp3 collection? Nope. They’re all backed up. Murphy’s Law states that if something can go wrong, it will. This was the worst time ever for a drive to fail as I had just reinstalled the operating system, and my backup system wasn’t running properly yet. Thankfully nothing irreplaceable was moved onto the broken drive in that narrow window of time when things weren’t being backed up.

I now want to get that new drive installed before the second goes belly up! Paranoid? You betcha!