The Other Search Engines

**Jeashale** · 07-30-2006, 10:34 PM

I was just wondering... how exactly do search spiders work? Is there a lot of work involved in creating a spider, or can it easily be done?

Dave

**chrishirst** · 08-03-2006, 06:31 AM

very simple things spiders and very simple to make.

all they do is grab the HTML source of the page then write it somewhere else. Could be to a database or to another text file. Then it moves on to the next page wherever it may be and repeat the process.

**hyipo** · 04:05 PM

This is my definition of "Search Spiders"

A search spider is a robot that crawls a site and information into their search databases; A program that looks at the page in text and cannot see images but only if an ALT tag is specified in <IMG...> area; A search spider also may be blocked or go through depending on the robots.txt; A search robot is like a robot visiting your website to determine what the sites about; A search spider crawls, sitemaps, .php pages(not the source code), xml, html, htm, txt, pdf, or almost anything except images, or stuff that cannot not read.

Also search spiders crawl the web. It tries to crawl sites to organize in a search engine

Is a good enough definition?

Here is page also explaining about what a search spider is:
http://www.searchtools.com/robots/
What is spider? - A Word Definition From the Webopedia Computer Dictionary