How do search engine bots work? Like the Google bot. How would you even go about programming something like that? How would you program something to click on links and grab text off pages to be stored in databases?

Just curious,
Nathan

Pretty much the same way a browser or rss reader works; by using http. Here's a code sample.

You wouldn't program a search bot to "click" links. Even the "Link Walker" type bots don't actually "click" things.

The spider is simply a retrieval agent. It goes out, reads the HTML source code from the URL and stores it.
Other software agents then read the stored version and break it down. Any URIs found in the code are then added to a crawler schedule database to be crawled at a later stage.

Hmm.. So essentially if I learn how web browsers work (getting the html file etc.) I'll have some understanding of how a spider would be programmed.

Thanks for the input,
Nathan

The entire code that makes up my ASP driven spider.

function GetPageCode(strURL,strUserAgent )
 Response.Buffer = True
  Dim objXMLHTTP, xml
  Set xml = Server.CreateObject("Microsoft.XMLHTTP")
  
 	xml.Open "GET", strURL, False
	xml.setRequestHeader "User-Agent", strUserAgent 
	xml.Send
	if xml.status = 200 then
	GetPageCode = xml.responseText
	else
	GetPageCode = cstr(xml.status) & " (" & xml.StatusText & ")"
	end if
  Set xml = Nothing
end function

Call the function with two parameters, the URL to be grabbed and the user agent to identify itself with in the remote servers logs and it comes back with either the response code or the source code from the URL.

It is that simple.

The robots.txt parser, database storage, URL parser etc are a bit more complicated but the crawler is very basic indeed.

Thanks! I appreciate you sharing that with me.

-Nathan

Could it be as simple as using php's fopen function and then just analyzing the data that comes out of that, although HTTP commands are probably better.