Hi - I'm looking for a way to query the site orbitz.com and get flight information. I'm trying to complie a list of the longest non-stop flights offered in the world, and I rather do it in an automated fashion. What I'd like to do is enter a list of major cities like JFK, HKG, etc and then query Orbitz if someone offers non-stop service between those two cities on a specific day. If so, I'd like to extract the flight time and distance (miles) of the flight and enter that data into a database.

Any ideas how to accomplish this? What tools should I be looking at to query html and build my own db from the results?

Thanks,

I doubt Orbitz.com will allow you to try to query their database, and if the server has a firewall, which I'm sure they do, the firewall will block it..

Your best option would be to get in touch with Orbitz and see if they have any sort of data feed you can use. Although with flights, Orbitz might be directly querying the airlines information, in which case you might need to get in touch with the airlines themselves. It could end up being fairly expensive.

I know Orbitz would not let me query their database directly - I am not asking for a mysql query I can run on port 1443 against their database - I am asking for a way to systematically query web pages on port 80 and with the results build my own database tables.

Any ideas?

The firewall would not block port 80 access. That's just web surfing.

it may do if you start scraping their site

so you are just looking to build your own database of the time it takes to get from one place to another and how many miles between the two places?

and you are looking for a script that will grab that data from orbits using a web form and then have the output stored into your database?

you need a bot or a spider....not sure if they have any online for what you want it to do. I have never made one but I am sure you can find a person who has...good luck with that


place to start your search

Web Bots & Spiders

or you might find some ideas for one here

http://securitydot.net/exploits/index.php?dir=bots/

Do you have web scraping tasks that need attention, but you don't have the attention to give them?

Give us a try @ http://www.scrapegoat.com.

We build custom scraping bots that grab the data you need and store it conveniently in sql format, csv format, excel, and many others.

We only do legitimate work. No content theft will be entertained.