How China Digital Times moved from MT to WordPress

Can you improve performance when moving from a statically generated site to a dynamic environment? You can if the conditions are right. In the case of CDT, publishing times were a nightmare with Movable Type. Search performance was horrible, and the comment spam problem caused such a drag on the server that we’d had to disable commenting altogether. Now, with the site fully tag-enabled, searchable and comment-able, loads are down dramatically and publishing times have dropped from 15 minutes to a few seconds.

Notes on a massive WordPress migration. Scot moved the China Digital Times site with 16,000 posts and 6,000 tags from Movable Type to WordPress and saw a huge performance increase. Nice.

Phoar! What a spike!

I released WP Super Cache 0.5.3 on Friday last. From the stats it looks like people are upgrading which is good news!

This version has a number of fixes and improvements:

  • If your blog is installed in a sub-directory you’ll want to upgrade. This version fixes the mod_rewrite rules that search for the cached files. If upgrading, make sure you delete the Super Cache rules so they’ll be upgraded. (Thanks Otto42)
  • With a click of a link in the backend page you can view your mod_rewrite rules to check that they are ok. This may help the adventurous who want to upgrade those rules manually too.
  • The plugin now warns if your blog’s root directory is writeable. Most of the time there’s absolutely no reason for this so it’s good to be reminded to fix it.
  • Check that $mutex is set. This is really only useful if your server is borked and the filesystem is mounted read-only but it’s good to be complete.

Wondering about the title? Check out this traffic graph Scott Beale posted a few weeks ago and you’ll understand. One of his posts hit the front page of Digg (twice) then Slashdot.org, and was covered by lots of other blogs and media. Wow.

On December 12th our blog hit a record high of 222,523 views in one day.

Best Buy Cease & Desist Traffic Stats

Digg users will love this

Version 0.5.1 of WP Super Cache is now available! This release of the plugin will be especially useful for Digg and Slashdot users who experience really huge traffic spikes.

This post has been dugg! Add your Digg here! I doubt it’ll get anywhere near the front page at this stage as it’s only collected 3 diggs in 7 hours. Once it hits 24 hours it disappears forever.

supercache-directfiles

After submitting a site to Digg, some people do the following to get every last ounce of performance out of their WordPress blog, especially on an underpowered server:

  1. Clear the cookies from their browser so the comment form won’t be filled in. (or use a second browser).
  2. Visit the page they submitted to Digg and save it to their desktop.
  3. Open an ftp programme, and recreate the path to the page. Then upload the saved file as “index.html” to that directory.
  4. Finally, after the Digg subsides 24 hours later, remember to remove the directory structure and index.html.

The new version of WP Super Cache automates all the above. You do have to make your blog’s root directory writable by the webserver, but you’re warned continually that this is a major security risk and reminded to make it read-only again.

Download it here: wp-super-cache.0.5.1.zip

How does it perform versus the regular static files the plugin creates? In most situations you won’t notice any difference, but when there are tens of thousands of requests hitting your server for one particular page, I find that Apache has trouble keeping up.

In other developments, I added checks for PHP safe_mode. Unfortunately safe_mode stops WP Super Cache working properly. I’m glad to see Mark applied my patch for Subscribe to Comments! No more stray emails if you use the moderation queue to approve comments from many posts!

100,000 page views in 5 minutes

Now, that’s why you can’t believe benchmarks. Sure, this server was able to serve 100,000 page views in 282 seconds but:

  • Requests were made from a VPS in the same datacenter. No need to worry about slow clients, or maintaining network connections to many remote clients.
  • I used Litespeed Web Server instead of Apache.
  • Was it realistic? Even a digg that sends you say, 8,000 page views in an hour, isn’t going to exercise your server that much unless your page is chock full of graphics, css and Javascript. (oh wait, web 2.0 ..)

So, Litespeed’s webserver is the one to go for? Maybe not. I can’t for the life of me get compression of the static cache working. When I do, the browser tries to display the gzipped data directly. I can enable the webserver’s gzip function but from tests I don’t think it caches the resulting gzipped file. (btw – mod_deflate, the Apache2 module that does the same thing suffers from this problem too!) Later – testing this again. Litespeed allows you to set a a gzip cache directory. For normal traffic it’s worth doing so pages load faster.
The mod_gzip site is a great resource if you want to find out more about compressing HTTP content.

How did Apache cope? I was serving 100 concurrent requests and Apache didn’t cope too well. It did serve all the file requests eventually but the load average jumped to just over 50 and the site was unavailable to anyone else. It’ll serve 1000 requests for a static file fine, even 10,000 too, but under constant load the server starts to wilt. Unless you have the RAM to keep enough Apache child processes going all the time you’re going to start swapping.
Meanwhile, Litespeed hardly caused a blip in the server’s load average. I’m quite impressed and I’m running it now. It’s also what powers WordPress.com. Even if you’re not using WordPress, you should look at alternatives to Apache.

This leads me nicely on to announce WP Super Cache 0.4! Download it here!

Major new features include:

  • A “lock down” button. I like to think of this as my “Digg Proof” button. This basically prepares your site for a heavy digging or slashdotting. It locks down the static cache files and doesn’t delete them when a new comment is made.
  • Automatic updating of your .htaccess file. (Backup your .htaccess before installing the plugin!)
  • Don’t super cache any request with GET parameters. You really need to use fancy permalinks now.
  • WordPress search works again.
  • Better version checking of wp-cache-config.php and advanced-cache.php in case you’re using an old one.
  • Better support for Microsoft Windows.
  • Properly serve cached static files on Red Hat/Cent OS systems or others that have an entry for gzip in /etc/mime.types.
  • The Reject URI function works again and now uses regular expressions!

Support queries should go to the forum. Make sure your posts are tagged “wp-super-cache”, but if you post from that link they will.

Donncha's Thursday Links

Reports of an exploit in WP Super Cache are being investigated but details are vague at this stage. There are only 3 reports of this out of hundreds who installed the plugin. Email me at donncha at ocaoimh.ie if you find files from outside your blog in wp-content/cache/supercache/
Tell me the following if you can:

  1. Plugins installed on your blog.
  2. WordPress version.
  3. The output of ps auxw and lsof if it’s installed.
  4. If you notice any strange processes running, check that they are not shell or php scripts.
  5. Anything strange in your log files? Look for the string “=http” for funky stuff. There will always be strange entries, so don’t be too alarmed if you see them, see my perl bot post. They’re fishing for an exploit on your server.

On to the links ..

  • You can now make your Google Reader tags public and add those sites to your blog as a blogroll. Nice! I still want a “timely dozen” like Photomatt used to have. Must look at Simple Pie ..
  • I’m shocked. Alkos is writing on his blog. I thought the world ended when he used colour but this? What will happen next? Oh, great photography, as usual!
  • Robert’s girlfriend has only a few days to go and his nerves are starting to go. Good luck with the birth! He has also posted Super Cache benchmarks. Good results!
  • Any world leaders reading? If you want to justify an outrageous pay rise, talk to Ireland’s Prime Minister/Taoiseach. His pay rise takes him to the top of the list of top 30 OECD countries. Well done Bertie! Now where’s the health service gone?

    Ireland €310,000
    The US €276,000
    Britain €267,500
    Germany €261,500
    Switzerland €256,000

How well did Super Cache handle the digg?

I must admit making the front page of Digg.com wasn’t the nail biting experience I expected.

$ grep "GET /2007/11/05/wordpress-super-cache-01/" access.log.1|grep digg -c
4686

digg.com

My Super Cache announcement only drew 4686 visitors which is an ultra-light Digg. The Digg page for the post received 808 diggs as of a few minutes ago which is great. Thank you for voting! Judging by the sheer number of comments on that post, there’s a lot of interest out there in the plugin.
What about traffic graphs? The spike at the end of the first graph is my nightly Backuppc service kicking in. The second is from Google Analytics. My server could certainly handle a lot more traffic!

digg.com traffic
analytics-digg

A quick look at my uptime shows the server hardly broke a sweat dealing with the extra traffic except where some idiot spammer bots tried to download my archives a few times. Unfortunately the first time that happened the archives weren’t cached and the load climbed.

For maximum performance, download Xcache and install it. The Xcache WordPress plugin uses Xcache to cache data structures and makes WordPress much faster, even if you don’t use any other caching tool.

WordPress Super Cache 0.1

It’s time to lift the veil of secrecy on my latest project. With help from friends who diligently tested and reported bugs on this I can now present version 0.1 of WP Super Cache!

It is an extensive modification of the famous WP-Cache 2 plugin by Ricardo Galli Granada. This plugin creates static html files that are served directly by the webserver as well as the usual WP-Cache data files. It also goes one step further fixing a couple of bugs, adding some hooks and new features and making WP-Cache more flexible.
From the plugin page, here are some of the major changes and updates:

  • A plugin and hooks system. A common complaint with WP Cache was that hacking was required to make it work nicely with other plugins. Now you can take advantage of the simple plugin system built in to change how or when pages are cached. Use do_cacheaction() and add_cacheaction() like you would with WordPress hooks. Plugins can add their own options to the admin page too.
  • Works well with WordPress MU in VHOST or non-VHOST configuration. Each blog’s cache files are identified to improve performance.
  • Normal WP-Cache files are now split in two. Meta files go in their own directory making it much faster to scan and update the cache.
  • Includes this WP-Cache and protected posts fix.
  • Automatically disable gzip compression in WordPress instead of dying.
  • As Akismet and other spam fighting tools have improved, the cache will only be invalidated if a comment is definitely not spam.

If your server is struggling to cope with the traffic your site gets this plugin could be just right for you. If your site regularly gets hit by spikes of traffic like a digging or slashdotting it’s definitely the right choice, and even for everyday use, you may very well notice your webserver is a little bit more responsive.

I contacted Ricardo last week and sent him on an earlier copy of the plugin but I haven’t heard from him yet however. I’d love to know what he thinks of my modifications!

Update! this post has been dugg, please digg it and we can really test the cache out!

Nov 6th: WP Super Cache 0.2 is out! I think all the bugs mentioned below are now fixed. I applied Tummbler’s patch (from Elliott and Reiner) that enables gzip compression of the WP-Cache data files and fixes feed content types.
Please note: PHP’s internal zlib compression must be disabled for this to work. Look in your php.ini for the zlib.output_compression and zlib.output_compression_level directives and comment them out by placing a “;” at the start of each line.

Check the plugin page above for the download link.