Tycoon Talk
Become a Big fish!
The number 1 forum for online business!
Post topics, ask questions, share your knowledge.
Tycoon Talk is part of Freelancer.com - find skilled workers online at a fraction of the cost.

PHP Forum


You are currently viewing our PHP Forum as a guest. Please register to participate.
Login



Freelance Jobs

Reply
Question about scraping
Old 05-08-2010, 01:18 PM Question about scraping
Experienced Talker

Posts: 33
Trades: 0
Hello everyone, I would like to set up a certain function for my site and was wondering if this is even achievable/possible. I am just now getting into PHP so you can definitely classify me as a newbie so if this question sounds ridiculous it's not because I'm dumb, it's because I don't know any better at this stage of my PHP proficiency.

Alright, so this is what I am hoping to set up:

from selected 20 sites, each of which features tens of articles daily, I want my site every day at a set time to be able to automatically scrape articles from these sites which meet certain conditions (for example articles that use certain keywords). Then I want my site to post links to all these relevant articles from these 20 sites that meet the conditions I have set.

Is this even doable or am I just a newbie that's way in over his head?

Last edited by Frank Drebin; 05-08-2010 at 01:19 PM..
Frank Drebin is offline
Reply With Quote
View Public Profile
 
 
Register now for full access!
Old 05-08-2010, 01:36 PM Re: Question about scraping
lynxus's Avatar
Awesomeo-Maximo

Posts: 1,618
Location: UK
Trades: 1
Its "doable" as any data on any website is scrapable.

However,
DOING it is no small task ( getting the data is easy )
Knowing what to get from where on the other hand is slightly more tricky.

It would simply be a case of getting the sites html and parsing out the bits you dont want.

Then storing it locally to then be displayed on your site.
__________________

Please login or register to view this content. Registration is FREE

Please login or register to view this content. Registration is FREE


Please login or register to view this content. Registration is FREE

Please login or register to view this content. Registration is FREE


lynxus is offline
Reply With Quote
View Public Profile Visit lynxus's homepage!
 
Old 05-08-2010, 02:26 PM Re: Question about scraping
Experienced Talker

Posts: 33
Trades: 0
Quote:
Originally Posted by lynxus View Post
Its "doable" as any data on any website is scrapable.

However,
DOING it is no small task ( getting the data is easy )
Knowing what to get from where on the other hand is slightly more tricky.

It would simply be a case of getting the sites html and parsing out the bits you dont want.

Then storing it locally to then be displayed on your site.

Thanks a lot for your reply lynxus, I really appreciate it.

Alright, this gives me hope for what I'm hoping to accomplish.
Frank Drebin is offline
Reply With Quote
View Public Profile
 
Old 05-08-2010, 03:08 PM Re: Question about scraping
chrishirst's Avatar
Missing! presumed drunk.

Posts: 42,384
Name: Chris Hirst
Location: Blackpool. UK
Trades: 0
Apart from the consideration that you would be STEALING the content.
__________________
Chris. ->>
Please login or register to view this content. Registration is FREE
<<-

A foolish consistency is the hobgoblin of little minds
Thought for today:- Is SEO the only industry where all the cowboys are Indians?
chrishirst is online now
Reply With Quote
View Public Profile Visit chrishirst's homepage!
 
Old 05-08-2010, 03:21 PM Re: Question about scraping
ThailandForum's Avatar
King Spam Talker

Posts: 1,415
Name: Sir Richard Cranium
Location: Bangkok Thailand
Trades: 0
^You get too hung up on minor little details Chris You never know, they might not mind him stealing from them
__________________

ThailandForum is offline
Reply With Quote
View Public Profile Visit ThailandForum's homepage!
 
Old 05-08-2010, 07:24 PM Re: Question about scraping
Experienced Talker

Posts: 33
Trades: 0
Quote:
Originally Posted by chrishirst View Post
Apart from the consideration that you would be STEALING the content.


"Stealing"? Did you not read my original post or are you just looking to pick a fight just because?

Let me assist you in comprehending what I said in my original post:

I would be LINKING to their site whenever they post something that is fits my scraping criteria, meaning I would be driving traffic TO their sites
Frank Drebin is offline
Reply With Quote
View Public Profile
 
Old 05-08-2010, 07:41 PM Re: Question about scraping
chrishirst's Avatar
Missing! presumed drunk.

Posts: 42,384
Name: Chris Hirst
Location: Blackpool. UK
Trades: 0
When you are scraping content from another site you are stealing it, regardless of your intent.
__________________
Chris. ->>
Please login or register to view this content. Registration is FREE
<<-

A foolish consistency is the hobgoblin of little minds
Thought for today:- Is SEO the only industry where all the cowboys are Indians?
chrishirst is online now
Reply With Quote
View Public Profile Visit chrishirst's homepage!
 
Old 05-08-2010, 08:23 PM Re: Question about scraping
Experienced Talker

Posts: 33
Trades: 0
I would not be scraping anything from any site without their consent first
Frank Drebin is offline
Reply With Quote
View Public Profile
 
Old 05-08-2010, 11:35 PM Re: Question about scraping
ThailandForum's Avatar
King Spam Talker

Posts: 1,415
Name: Sir Richard Cranium
Location: Bangkok Thailand
Trades: 0
Without consent linking or not linking to their site it is still stealing, some don't mind as it does drive traffic to them, so do mind and will quite likely contact your host, quite often this results in the thiefs site being taken down till the issue is resolved.
__________________

ThailandForum is offline
Reply With Quote
View Public Profile Visit ThailandForum's homepage!
 
Old 05-10-2010, 10:17 AM Re: Question about scraping
Experienced Talker

Posts: 33
Trades: 0
Quote:
Originally Posted by ThailandForum View Post
Without consent linking or not linking to their site it is still stealing, some don't mind as it does drive traffic to them, so do mind and will quite likely contact your host, quite often this results in the thiefs site being taken down till the issue is resolved.


as I mentioned in my last post....



Quote:
Originally Posted by Frank Drebin View Post
I would not be scraping anything from any site without their consent first
Frank Drebin is offline
Reply With Quote
View Public Profile
 
Reply     « Reply to Question about scraping
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off





   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML



Page generated in 0.68122 seconds with 12 queries