Tag Archives: irishblogs

Varnish is an open source, state of the art web application accelerator.

What it does is make your existing site faster by caching requests so your web server doesn’t have to handle them. This helps because your web server may be a lumbering giant like Apache that is loaded up with extra functionality like PHP, the GD library, mod_rewrite and all the other tools you need to make your website. All these modules unfortunately make your general purpose web server slower and heavier so by avoiding it your site spits out pages much faster!

Varnish sits in front of your webserver. Most documentation I’ve read on the subject suggest having Apache listen on any port other than port 80 and then have Varnish listen on port 80 of the external IP address. There’s no need to do this as I configured Apache to listen on port 80 of the 127.0.0.1 or localhost address while Varnish sits on the external IP.

Installing Varnish

Setting up Varnish is fairly easy. I’m going to assume that you’re already using Apache and On a Debian based system just use this to install it (as root)

apt-get install varnish

Apache

You need to configure Apache first. It has to listen on port 80 of the localhost interface. Edit /etc/apache2/ports.conf and change the following settings:

NameVirtualHost 127.0.0.1:80
Listen 127.0.0.1:80

Normally Apache listens on port 80 of all interfaces so you’ll probably just have to add “127.0.0.1:” in front of the 80.

Varnish

By default Varnish won’t start. You need to edit /etc/default/varnish. Change the following options in that file:

START=yes

DAEMON_OPTS="-a EXTERNAL_IP_ADDRESS:80 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

Replace EXTERNAL_IP_ADDRESS with the IP of your external IP address.

Now edit /etc/varnish/default.vcl. The file should already exist but most of it is commented out. First of all change the “Backend default”:

backend default {
    .host = "127.0.0.1";
    .port = "80";
}

This tells Varnish that Apache is listening on port 80 of the localhost interface.

I’m going to define several functions in the default.vcl now. Comments in the code should explain what most of it does.

# Called after a document has been successfully retrieved from the backend.
sub vcl_fetch {
    # Uncomment to make the default cache "time to live" is 5 minutes, handy 
    # but it may cache stale pages unless purged. (TODO)
    # By default Varnish will use the headers sent to it by Apache (the backend server)
    # to figure out the correct TTL. 
    # WP Super Cache sends a TTL of 3 seconds, set in wp-content/cache/.htaccess

    # set beresp.ttl   = 300s;

    # Strip cookies for static files and set a long cache expiry time.
    if (req.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$") {
            unset beresp.http.set-cookie;
            set beresp.ttl   = 24h;
    }

    # If WordPress cookies found then page is not cacheable
    if (req.http.Cookie ~"(wp-postpass|wordpress_logged_in|comment_author_)") {
        set beresp.cacheable = false;
    } else {
        set beresp.cacheable = true;
    }

    # Varnish determined the object was not cacheable
    if (!beresp.cacheable) {
        set beresp.http.X-Cacheable = "NO:Not Cacheable";
    } else if ( req.http.Cookie ~"(wp-postpass|wordpress_logged_in|comment_author_)" ) {
        # You don't wish to cache content for logged in users
        set beresp.http.X-Cacheable = "NO:Got Session";
        return(pass);
    }  else if ( beresp.http.Cache-Control ~ "private") {
        # You are respecting the Cache-Control=private header from the backend
        set beresp.http.X-Cacheable = "NO:Cache-Control=private";
        return(pass);
    } else if ( beresp.ttl < 1s ) {
        # You are extending the lifetime of the object artificially
        set beresp.ttl   = 300s;
        set beresp.grace = 300s;
        set beresp.http.X-Cacheable = "YES:Forced";
    } else {
        # Varnish determined the object was cacheable
        set beresp.http.X-Cacheable = "YES";
    }
    if (beresp.status == 404 || beresp.status >= 500) {
        set beresp.ttl = 0s;
    }

    # Deliver the content
    return(deliver);
}

sub vcl_hash {
    # Each cached page has to be identified by a key that unlocks it.
    # Add the browser cookie only if a WordPress cookie found.
    if ( req.http.Cookie ~"(wp-postpass|wordpress_logged_in|comment_author_)" ) {
        set req.hash += req.http.Cookie;
    }
}

# Deliver
sub vcl_deliver {
    # Uncomment these lines to remove these headers once you've finished setting up Varnish.
    #remove resp.http.X-Varnish;
    #remove resp.http.Via;
    #remove resp.http.Age;
    #remove resp.http.X-Powered-By;
}

# vcl_recv is called whenever a request is received
sub vcl_recv {
    # remove ?ver=xxxxx strings from urls so css and js files are cached.
    # Watch out when upgrading WordPress, need to restart Varnish or flush cache.
    set req.url = regsub(req.url, "\?ver=.*$", "");

    # Remove "replytocom" from requests to make caching better.
    set req.url = regsub(req.url, "\?replytocom=.*$", "");

    remove req.http.X-Forwarded-For;
    set    req.http.X-Forwarded-For = client.ip;

    # Exclude this site because it breaks if cached
    #if ( req.http.host == "example.com" ) {
    #    return( pass );
    #}

    # Serve objects up to 2 minutes past their expiry if the backend is slow to respond.
    set req.grace = 120s;
    # Strip cookies for static files:
    if (req.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$") {
        unset req.http.Cookie;
        return(lookup);
    }
    # Remove has_js and Google Analytics __* cookies.
    set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");
    # Remove a ";" prefix, if present.
    set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
    # Remove empty cookies.
    if (req.http.Cookie ~ "^\s*$") {
        unset req.http.Cookie;
    }
    if (req.request == "PURGE") {
        if (!client.ip ~ purge) {
                error 405 "Not allowed.";
        }
        purge("req.url ~ " req.url " && req.http.host == " req.http.host);
        error 200 "Purged.";
    }

    # Pass anything other than GET and HEAD directly.
    if (req.request != "GET" && req.request != "HEAD") {
        return( pass );
    }      /* We only deal with GET and HEAD by default */

    # remove cookies for comments cookie to make caching better.
    set req.http.cookie = regsub(req.http.cookie, "1231111111111111122222222333333=[^;]+(; )?", "");

    # never cache the admin pages, or the server-status page
    if (req.request == "GET" && (req.url ~ "(wp-admin|bb-admin|server-status)")) {
        return(pipe);
    }
    # don't cache authenticated sessions
    if (req.http.Cookie && req.http.Cookie ~ "(wordpress_|PHPSESSID)") {
        return(pass);
    }
    # don't cache ajax requests
    if(req.http.X-Requested-With == "XMLHttpRequest" || req.url ~ "nocache" || req.url ~ "(control.php|wp-comments-post.php|wp-login.php|bb-login.php|bb-reset-password.php|register.php)") {
        return (pass);
    }
    return( lookup );
}

Notes:

1. Varnish caches Javascript and CSS files without the cache buster ?ver=xxxx parameter. Varnish doesn’t cache any url with a GET parameter so those files weren’t getting cached at all.
2. The code removes the Cookies for Comments cookie after it checks for GET and HEAD requests. This improved caching significantly as web pages are not cached with and without that cookie. They are all cached without it. The cache hit/miss ratio went up significantly when I made these two changes.
3. I have a private site on this server that requires login. I had to stop Varnish caching this site as the privacy plugin thought I wasn’t logged in. See the example.com code above.
4. If pages were purged Varnish could store cached pages for much longer.

As I didn’t modify WordPress so it would issue PURGE commands there are probably issues with the cache keeping slightly stale pages cached but I haven’t seen it happen or receive complaints about that.

PHP

Since all requests to Apache come from the local server PHP will think that the remote host is the local server. By using an auto_prepend_file set in your php.ini or .htaccess file you can tell PHP what the real IP is with this code:

if ( isset( $_SERVER[ "HTTP_X_FORWARDED_FOR" ] ) ) {
        $_SERVER[ 'REMOTE_ADDR' ] = $_SERVER[ "HTTP_X_FORWARDED_FOR" ];
}

You’ll see a huge improvement if you use Apache, especially if you don’t use a full page caching plugin like WP Super Cache on your WordPress site.

To see exactly how well Varnish is working use varnishstat and watch the ratio of cache hit and miss requests. This will vary depending on your TTL and by how much time Varnish has had to populate the cache. You can also configure logging using varnishncsa as described on this page:

varnishncsa -a -w /var/log/varnish/access.log -D -P /var/run/varnishncsa.pid

Now use multitail to watch /var/log/varnish/access.log and your web server’s access log.

I used a number of sites for help when setting this up. Here are a few:

• Tutorial: Setting up Varnish with Apache.
• Putting Varnish In Front Of Apache On Ubuntu/Debian
• WordPress + nginx + Varnish + Apache 2
• MediaWiki Manual:Varnish caching
• Using Varnish So News Doesn’t Break Your Server

I have tried Nginx in the past but could not getting it working without causing huge CPU spikes as PHP went a little mad. In comparison, Varnish was simple to install and set up. Have you tried Varnish yet? How can I improve the code above?

Edit: It looks like someone else has done the hard work. I must give the WordPress Varnish plugin a go.

This plugin purges your varnish cache when content is added or edited. This includes when a new post is added, a post is updated or when a comment is posted to your blog.