Hehehe…is my title effective to make you read this article? sorry about that, just playing with ‘ya. I just check my page stats and found out that there is a lot of bots especially from google crawls on my site. Then I notice there is one called Inktomi Slurp. Without having an idea of what the hell it was, I run some googling and found out that it is one of many web crawlers. Then I managed to find a good article on this topic. Hope it helps.

Inktomi Slurp, Yahoo Slurp, Google Slurp, they are all crawlers or spiders or bots which came to our site and collecting information about the site. Optimizing web pages for search engines such as Inktomi and Google is a great start, but what good is all your hard work if those search engine spiders never stop by for a visit? The two primary spiders to focus on are Inktomi’s “Slurp” and Google’s “Googlebot”. Below are details on 3 top search engine crawlers.

Inktomi Slurp

Slurp is Inktomi’s web-indexing robot that stroll the Internet collecting information from individual web pages and compiles them into a searchable index used by companies such as HotBot and MSN for search results. Slurp is a standalone UNIX based robot with a sole purpose defined as indexing / statistics and adheres to the Robot Exclusion Standard.

If you simply create a web page and host it on a server you’ll have to patiently wait for Slurp to stop by for a visit and collect your information to be included in Inktomi’s search results. This means checking server logs for specific IP addresses on a regular basis to see if and when Slurp stopped by for a visit. You also have to wait longer to see your optimized web pages show up in Inktomi results.

Googlebot

Googlebot is Google Inc’s search engine spider. Googlebot is a standalone Linux based robot with a sole purpose of indexing web pages and adheres to the Robot Exclusion Standard. Unlike Inktomi which changes results every few days, Googlebot crawls web pages over the course of a few weeks and then update the main index in Google Search Directory. This usually takes place once a month. Google results will change depending on which data serving the results are pulled from. The results change sporadically at first and then settle down as the dance ends usually lasting between 3-5 days.

Googlebot loves new content! New web pages are spidered by Googlebot often, varying around every few days to check and see if new content has been added. If so, you can expect frequent Googlebot visits. If your web pages rarely change and have no updates, Googlebot my place you into its monthly visit, right before the dance. Why should Googlebot come for a visit if your web page has nothing new to say?

Yahoo Slurp

Yahoo Slurp is Yahoo’s search engine spider (similar name to Inktomi’s - completely different spider). Similar to Googlebot, Yahoo Slurp follows every instance of HREF links and dynamic links but not SRC links such as image and frames. If you want to make sure your individual pages get spidered, make sure your linking menus are in working order and just to be safe create a site map. Yahoo Slurp crawls your web pages to be indexed in the Yahoo search database and in return listed in its SERP’s.

More info on site optimization at Web-Page-Optimization.com

Written by: nyunyu @ http://www.nyunyu.info