So i'm seeing a lot of price comparison tools and i'd really like to create one for a specific group of projects i'm interested in..

lately i've been seeing a bit more general engines that are searching sites that I was unaware had data feeds.. so this makes me curious about how they're doing this..

are most of these sites just scraping regular searches? do they cache their results in the db to reduce the number of searches.. or are they pretty much all using data feeds of some sort?

are there any good frameworks out there already written for something like this.. ?

to be honest... i really dont know unless they are using hyper submit Meta Extension wich allows you to do that. Costs alot of money thou.

hmm.. i know with amazon i can just use their api to get a soap feed and parse that.. but for other sites.. i can easily write perl to extract all of the data i need.. but in some cases we're talking 2-3k page views to parse.. won't most sites get angry with this or block my ip? once i have the data i can put it in the db and then just parse the "newest" materials after that..

Originally Posted by ju1ce

hmm.. i know with amazon i can just use their api to get a soap feed and parse that.. but for other sites.. i can easily write perl to extract all of the data i need.. but in some cases we're talking 2-3k page views to parse.. won't most sites get angry with this or block my ip? once i have the data i can put it in the db and then just parse the "newest" materials after that..

Hmmm.... Ya know... ur beyond what i know lol, im only 12.. i know what your talkin about but dunno how to do it