Tracking Web Spiders / Search Engine vists to your site.

Aug 7, 2001

Search engine traffic is some of the most important traffic that your website gets for a whole host of reasons. First off, the majority of web servers start their quest for information at a search engine. Having a high placement in the search engines will generate a lot of traffic to your website. This is why there are so many Search Engine Optimization companies out there. Another reason why search engine traffic is so important to your site is that it's free. Advertising is expensive, even at the dirt cheap banner ad prices these days. (Banner ads aren't working anymore anyhow.)

So you've gone out and spent lots of time and/or money on getting the pages of your website optimized for the search engines. The next question is when will the search engines pick up your modifications? That's a waiting game and like any waiting game, it can be nerve wracking. Patience isn't my strong suit either. One option is to pay for a frequent scan of your website. AltaVista has an Express Inclusion Service that guarantees weekly scans of your website. This is probably the way to go if you want your weekly/monthly specials to be in the search engine databases. However if you're a small site like us, then your site doesn't change to often and it probably isn't worth the money.

One little thing I do to make the wait easier is to track and log any visits from web spiders or crawlers to my website. Knowing that Goodle or Fast came through the site recently is a help. How is this done?

What I do is to create a PHP page that will send me email when it is requested from the server. I include a bunch of server variables in that email, such as REMOTE_ADDR and HTTP_USER_AGENT which tells me the IP address and the browser type of whatever is requesting that page. Most spiders are good enough to announce their identity via the HTTP_USER_AGENT variable. The only link to this page is hidden on my site by a single pixel transparent gif. Human surfers stand a slim chance of finding this link, but any spider that is looping through all of the links on that page will easily locate it.

I'm using PHP to implement this, it would be just as easy to do it in ASP, Perl, ColdFusion or whatever scripting language you have available. Automate like crazy and stay on top of what's happening to your website.

      Rene

If you have any comments or questions, please feel free to drop Rene an email at rene@astutecomputing.com. If you would like assistance in setting up this kind of feature on your website, please contact us for a quote.