I agree that in principle a second googlebot that mimics IE would be a great advance for Google. If that ‘bot only crawled top results for major keywords, it might not even be so hard for Google.

But a couple problems:

1) A far-reaching bot, that crawls deep, would be long, slow, and bandwidth-intensive. Google’s existing crawl takes, what, a month (?), though really it’s a never-ending task at this point. All indications are that Google is continually facing a constraint as to how much of the web to crawl — would want to crawl more if it could, seemingly, but continually drops low-pagerank sites in an attempt to focus efforts where they’re most needed.
2) Google doesn’t look favorably on folks tampering with HTTP headers. But the crawler you propose would have fake HTTP headers (User Agent, Referer, etc.). Google might be uncomfortable with this.
3) The crawler would still come from Google’s IP blocks. Some cloaking is based on HTTP headers (user agent or, in WhenU’s case as to the research I just posted, Referer header or lack thereof). A second ‘bot could catch these tricks. But cloaking on the basis of ‘bot IP would be much harder to find: Google would have to use another set of IPs for this project, and once word got out which IPs those were, the smart cloakers would adjust their behavior accordingly.

All in all, I think it’s actually not such an easy task. Wouldn’t want to be in Google’s shoes!

Ben Edelman
benedelman.org

You haven’t tried it lately, have you?