I use php-cgi and don’t have much problems, plus I use memory caching with memcached and object-cache.php which enables memcache to work.

Oh you know what, php-cgi carries a high load sometimes a /etc/init.d/php-cgi restart sometimes helps