I had cause to research search engine bots yesterday and I came across a few nuggets:
- Help the Googlebot understand your web site
- Google Information for Webmasters
- Nine things you can do to make your web site better
- If-Modified-Since and dynamic content
- Issuing Correct HTTP Headers
- Are you using If Modified Since?