December 17, 2002

Googlebot's curious DNS

What is it with Googlebot? Googlebot is the technology that Google uses to 'crawl' web pages as it builds its index of the web. Every so often, it crawls www.mohea.com, enabling people to find the pages on our web site with underaged boys in underwear. Wait, no, that's not it.

A little over a week ago, I changed the DNS entries for mohea.com to point to the new server (the one hosting the page you're reading now). Usually, it takes about 72 hours for all the Domain Name Servers worldwide to flush their cache and get the new entries. Looking at the logs of our old site, I can see that most people got switched over sometime on the 7th. www.mohea.com went from 2072 hits on 12/6 to just 48 on 12/8.

But by yesterday, 12/16, it still got 36 hits. All of the hits from about 12/10 on have been from Googlebot. For some reason, the DNS servers that Google uses aren't clearing the cache and getting the updated information. Weird.

Posted by Mike at December 17, 2002 09:03 AM