The Other Search Engines

**DavidatWork** · 03-05-2004, 10:55 AM

Hi all,

My web site is being crawled on a regular basis by an ever growing number of SE's. When trying to identify who these SE's are I'm beginning to wonder....

Do people write crawlers that can be purchased by other people? If so, does that mean that the same bot name (ex... aspseek) does not necessarly come from the same SE or place?

ex...
Can aspseek bot crawl my site for aspseek.net and also....
aspseek bot crawl my site for SE.net that purchased the aspseek search engine software.

If this is the case , how do you get a handle on who is crawling your site and if they are really a search engine or just another company crawling your site, etc.

Hope I made sense in my question...its easier to think it then to write it down...

thanks....

**rocky1** · 03-06-2004, 03:49 AM

Well it doesn't work just exactly like that. Several engines may purchase the results of a single bot, but to my knowledge they aren't sent out on any engine specific crawling missions.