Easy Automated Snapshot-Style Backups with Rsync

Have rsync, have RAID volume, have ssh connection to server. What’s the best way to back it up? Here’s one way. Use hard links to make “duplicate” archives of remote content with the minimum of wasted space.
I’ve adapted the idea with the following:

for t in `ls remote/`
do
    for i in `ls remote/$t/`
    do
        export older=7
        rm -fr remote/$t/$i/7
        for n in `seq 6 1`
        do
            if [ ! -d remote/$t/$i/$n ]; then
                mkdir -vp remote/$t/$i/$n
            fi
            mv remote/$t/$i/$n remote/$t/$i/$older
            export older=$n
        done
        cp -al remote/$t/$i/0/. remote/$t/$i/1
    done
done

rsync -v -v --progress -az --delete --exclude-from=exclude.txt -e 'ssh' www.example.com:/home/ remote/www/home/0/
rsync -v -v --progress -az --delete --exclude '*logs/*' -e 'ssh' www.example.com:/usr/local/apache/ remote/www/apache/0/
rsync -v -v --progress -az --delete --exclude-from=exclude.txt -e 'ssh' mail.example.com:/home/ remote/mail/home/0/

Using this script you can have multiple hosts (here we have www.example.com and mail.example.com) and multiple directories(/home and /usr/local/apache) on each host backed up.
This script and post will be updated as I refine the backup procedure.
Later… After reading through the linked article above, I see that the author uses a slightly different way of rotating the archive snapshots. Instead of re-using the oldest one, he creates a hard link copy of the latest snapshot, “0”, in the “1” snapshot. When rsync downloads changed files it “deletes before copying” so that the old file is preserved in “1”, but the new file is now in “0”. Read the “hard links” section of the above for more on how that works!
Here’s a diff of what I changed:

7,12c7,8
<         if [ ! -d remote/$t/$i/7 ]; then
<             mkdir -vp remote/$t/$i/7
<         fi
<     
<         mv remote/$t/$i/7 remote/$t/$i/7.tmp
<         for n in `seq 6 0`
---
>         rm -fr remote/$t/$i/7
>         for n in `seq 6 1`
20,21c16
<         mv remote/$t/$i/7.tmp remote/$t/$i/0
<         cp -al remote/$t/$i/1/. remote/$t/$i/0
---
>         cp -al remote/$t/$i/0/. remote/$t/$i/1

Later Still… I found and installed Backuppc via delicious this morning and I’ve got it working. It uses the same idea of space saving hard links but also provides a web based interface to the backup and restore procedures. I think it can be automated and there are loads of other features I couldn’t possibly hope or wish to duplicate in a timely manner!
Installation is relatively well explained, even down to installing the required Perl modules from CPAN, but configuration is slightly harder. Just make sure to override the defaults in config.pl with config.pl in the $backupdir/pc/$host/config.pl file. That took som figuring out where that file was.
It’s already backed up two Windows machines and it’s working on a Linux box. Backup contents can be examined over the web and because it uses a “pool” mechanism, it can find duplicate files, even among different backups! That should save a lot of disk space and network bandwidth as time goes by!

First Post From Fedora Core 2

No, I don’t know yet if I’ve been hit by the WinXP/Kernel 2.6 bug but I do have the recovery commands written out in good old dead tree format just in case things go sideways when I reboot later.
First impressions? It’s buggy, a little. Nautilaus just crashed there and it’s not responding correctly yet (I’ll probably need to restart X), I had to enter the Oceanfree dns settings in /etc/resolv.conf as kppp didn’t pick them up from pppd for some reason (they appeared in /var/log/messages however), and when I first went online Mozilla and Moz FF both crashed when they received keyboard input. That seems to be solved now by restarting X (obviously, if you’re reading this!)
Later… Booting into XP seems to be OK!

Apart from that, it looks nice, the usual arial-type fonts are installed although they look bigger than FC1 but maybe that’s down to some tweaking I did previously in FC1.
The first thing to do of course is install apt, do an “update”, and install “xmms-mp3” as well as a few updates that are available.
Open Office works as well as can be expected, Gnucash is slightly updated but nothing new. I briefly played with the new spatial features of Nautilaus but it’s crashed now so that experimentation was short lived!
Is it worth upgrading from FC1? Probably not, especially if you have a Kernel 2.6 update for FC1 then you get all the speed improvements without the bugs. 🙂
Later Still… I’ve burned CDs, copied photos from my camera (it’s handy that a “camera” icon appears automagically now), and Nautilus seems to have settled down somewhat. It generates thumbnails of image folders really quickly, almost as quickly as WinXP, and certainly much faster than Gthumb (and Gthumb can use the same thumbnails AFAICS).
This is a handy tip for using the Gnome 2.6 GUI. Not obvious and terrible usability wise however:

You can type a file name into the new file selector.

Ctrl + L will open the filename box, with autocomplete and all.

Fedora Core 2 and WinXP Problems?

Looks like a fairly major bug in Fedora Core 2 from RedHat. If you have WinXP installed before you install FC2 grub, or something, will destroy your partition table and possibly other data. There are a few solutions mentioned on the page above and there’s more on slashdot and the Fedora Test mailing list. I think I’ll hold off a few days before trying this release.

Sed one-liners

Cool! Several handy one-liners for use in Sed, the stream editor.

# double space a file
sed G

# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
sed ‘/^$/d;G’

# triple space a file
sed ‘G;G’

# undo double-spacing (assumes even-numbered lines are always blank)
sed ‘n;d’

The SBPost and Me!

about 10 days ago a journalist from the Sunday Business Post rang asking me about Linux in TradeSignals. A brief interview followed, but then 2 days later a photographer called out and took a picture of me stuggling with a laptop while smiling bravely. I prefer being on the other side of the camera..
Those of you who didn’t buy the newspaper on Sunday are spared the picture of me, and the article (unfortunately), but there is an article about a CEO who tried Linspire on his laptop and was surprised to find it easy to install and use.
They also put their obligitory Microsoft FUD article online too which is a bit disappointing as it’s quite misleading and repeats the usual MS line. Oh well!
Later… This was mentioned on The ILUG mailing list on Sunday, and here too, along with some more thought about the pro-Microsoft article.

Linux: Tuning Swappiness, eh?

Via slashdot, I thought this thread on how swappy Linux should be would be informative but it appears to be very subjective.
I tested with swappiness set high and low, along with Mozilla FF, Thunderbird, xchat and mgt, I opened 3 2560×1920 images in the GIMP and performed a few operations on each. Neither setting affected performance much, except that with swappiness set high it may have been a little slower. That said, the GIMP maintains it’s own cache, and I have that set at 256MB.
I have swappyness set to 10 now. Not sure if that’s good or bad, or whatever.. *shrug*