|
How long is a piece of string! Having worked on a search engine I can tell you that there are significant challenges. You will be working with huge datasets, and getting it to perform well will be a huge challenge unless you have some serious investment in hardware, and bespoke software. Crawling sites is an almost exponential process - start with microsoft.com for example and you will easily have a million websites in your search engine within 24 hours. (assuming you had the bandwidth and software available to processs this many)
|
