The Other Search Engines

**RichTC** · 11-27-2004, 09:55 PM

Hi all,

despite paying the fee to be included in the Yahoo dirctory last year i find that the Inktomi slurp bot when it visits takes very little data with it. Im not listed in yahoo, well not anywhere worth talking about anyway!

I get something like 100 hits from the slurp bot v 2000 Google, 1000 MSN

I find that Yahoo lists only 500 pages compared with 100,000 on google. I just dont get it. Ive tried to contact yahoo but it looks like the inktomi slurp bot just doesnt index volume.

I also notice in the US that Yahoo expect you to pay via overture for "submit express" to get the bot to index you, meanwhile in the UK overure dont currently offer the service.

Anyone have any ideas how i can get the inktomi slurp bot to fully index my site?

Thanks in advance

**Jamesanderson** · 11-28-2004, 11:48 AM

Sounds like the Yahoo Slurp Black Hole to me!

**RichTC** · 12-03-2004, 07:50 PM

James,

Looks like i am in a Yahoo black hole. These were the spider / robot stats for the whole of November on my site (Most active First):-

Robots/Spiders visitors

32 different robots Hits Bandwidth Last visit (ALL November)

Googlebot 51494 798.19 MB 30 Nov 2004 - 10:11
MSNBot 14878 392.22 MB 30 Nov 2004 - 23:27
Unknown robot 4478 77.03 MB 25 Nov 2004 - 18:48
LinkChecker 2314 49.55 MB 30 Nov 2004 - 18:12
Alexa 1712 28.87 MB 30 Nov 2004 - 02:59
Inktomi Slurp 1379 21.53 MB 30 Nov 2004 - 22:52
Fish search 920 12.52 MB 29 Nov 2004 - 13:48
InternetSeer 672 104.53 KB 30 Nov 2004 - 23:18
MSIECrawler 421 3.48 MB 29 Nov 2004 - 23:31
UdmSearch 139 3.67 MB 29 Nov 2004 - 04:19
Mirago 119 876.41 KB 16 Nov 2004 - 07:43
LinkWalker 67 960.44 KB 30 Nov 2004 - 04:55
AskJeeves 58 1.52 MB 30 Nov 2004 - 15:21
psbot 53 534.33 KB 22 Nov 2004 - 19:36
larbin 52 1.61 MB 24 Nov 2004 - 22:07
Nomad 48 223.27 KB 23 Nov 2004 - 20:50
Boris 35 233.33 KB 09 Nov 2004 - 12:51
Unknown robot 28 83.86 KB 30 Nov 2004 - 23:36
LinkScan 26 577.25 KB 30 Nov 2004 - 10:21
Combine System 21 232.60 KB 03 Nov 2004 - 09:51
Templeton 19 107.88 KB 10 Nov 2004 - 13:15
Unknown robot 11 207.01 KB 25 Nov 2004 - 02:36
Fluid Dynamics 7 176.38 KB 10 Nov 2004 - 12:37
ExactSeek 5 87.10 KB 03 Nov 2004 - 19:01
NetMechanic 4 0 30 Nov 2004 - 08:00
legs 4 108.90 KB 03 Nov 2004 - 16:11
EchO! 2 98.83 KB 30 Nov 2004 - 23:32
Netcraft 2 0 06 Nov 2004 - 11:29
CoolBot 1 17.42 KB 03 Nov 2004 - 15:53
ObjectsSearch 1 21.60 KB 10 Nov 2004 - 06:53
Walhello appie 1 27.38 KB 26 Nov 2004 - 11:30
Ultraseek 1 21.60 KB 11 Nov 2004 - 23:16

May i ask what you conclude from these stats? the picture has been similar most months. The only real interest im having is by Google (which is worth sweet FA until after the next update?) MSN which is only any good if it goes live? Yahoo after that is listing next to nothing and the rest are a waste of time.

Any thoughts on what i could do to get Yahoo interested? is this all down to DMOZ who are not listing my site?