|
There are so many spiders out there that hit my site every day. Is there one of them that makes its data available to others to use in building a search engine? Something like an open source crawl data site.
Anyone have any ideas?
Ultimately I want to develop (learn as I go) a search engine for my site. I don't want to send a spider out to gather everything that someone else has already captured.
If this option is not viable, does anyone know of another option? I see all these sites that have a search engine, and it seems almost like they are integrating data from many search engines. How do they do that? Is it legal?
Frank
Last edited by neorunner; 08-29-2006 at 10:42 PM..
Reason: Added a question or four.
|
