Upgraded to PHP 4.3.1 and added Robotcop to Apache on the site. Robotcop will allow us to “trap” misbehaving web crawlers and block them from using the site. Recipezaar is different in that it contains millions (billions?) of pages that you can browse through, all dynamically-generated from a database. So irresponsible (meaning they ignore our robots.txt file) web crawlers stupidly spend weeks crawling the site which causes an unnecessary load on the site, which is problematic for us in a number of ways. It’ll be interesting to see how well Robotcop works.