Content & Writing Tycoon

**mason** · 07-31-2006, 06:18 PM

How does scraping work? I'm looking for a method to automatically update my clothing list for my clothing site and someone suggested it on a diff forum.

for instance, how do I scrape each clothing item, their brand, their price, and the url in which it is located on the site?

**Karl89.co.uk** · 07-31-2006, 06:20 PM

Well you would make a script to do it for you. Ones I have done have been scripted in PHP and I've used them to gather a lot of valuable information and content.

**mason** · 07-31-2006, 06:28 PM

i just don't understand how a script can "pull" information from other sites. I prob just need to look at some code examples to get a better idea.

**Karl89.co.uk** · 07-31-2006, 06:28 PM

It's easy enough to do, there must be some code examples around. I'll see if I can find any.

**Lpspider** · 07-31-2006, 06:51 PM

Interesting... I'd never really heard of this until now. (well, I heard of it but didn't know what exactly it was).

**Karl89.co.uk** · 07-31-2006, 06:58 PM

Quote:

Originally Posted by Lpspider

Interesting... I'd never really heard of this until now. (well, I heard of it but didn't know what exactly it was).

I'm about to create one now to pull lots and lots of images from a website.

**Lpspider** · 07-31-2006, 07:12 PM

Quote:

Originally Posted by Karl Evans

I'm about to create one now to pull lots and lots of images from a website.

Really... that would be... very beneficial. Something very beneficial indeed... hehe.

Seriously, any chance I could get my hands on that? :devil:

**Karl89.co.uk** · 07-31-2006, 07:32 PM

Quote:

Originally Posted by Lpspider

Really... that would be... very beneficial. Something very beneficial indeed... hehe.

Seriously, any chance I could get my hands on that? :devil:

Haha. Maybe.

I'll see how many images it pulls and things first, see how much space it uses. :P

**Lpspider** · 07-31-2006, 09:16 PM

Quote:

Originally Posted by Karl Evans

Haha. Maybe.

I'll see how many images it pulls and things first, see how much space it uses. :P

Well, tell me how it goes regardless. :P

**Tran** · 07-31-2006, 09:26 PM

To find out how to "scrape" all that information..

You would have to figure out what to parse from the whole page..

**Decepti0n** · 08-01-2006, 12:27 AM

I think you can use file_get_contents() in PHP, or fopen() on an exact url (http://...)

Then all your basic regex functions, and string functions to search for things, like <img etc...at least thats how I'd go about doing it.

Of course, this only gets the HTML code

I might make a proper script later on that I can pretty much do whatever with, have some checkboxes to grab all the images and whatnot.

**imported_alan** · 08-01-2006, 06:23 PM

Perl is the classic programming language for scraping.

Check out Perl and LWP and Spidering Hacks by Oreilly for more info.

**Lpspider** · 08-02-2006, 12:10 AM

Does anyone have a practical way to do this - namely images?

**imported_Max** · 08-02-2006, 02:02 AM

I hate to be the 'whitehat' here, but has anyone even thought about the legalities?

**Shpigford** · 08-02-2006, 11:09 AM

Quote:

Originally Posted by Max

I hate to be the 'whitehat' here, but has anyone even thought about the legalities?

It's illegal. Unless the author of any content you "scrape" has given you permission to do so, it's illegal in every way.

**Atomm** · 08-02-2006, 12:18 PM

Look, I can think of some legitimate uses of scrapping a website. In fact, one of my projects would greatly benefit from the ability to scrape because it is just a content repositories for other sites who want to be on my site. Some of these sites do not have RSS, so that leaves either email or scraping to automate the content retrevial process.

Then there is wikipedia which uses the Creative Commons license. You can scrape all you want as long as you credit the source.

So, I want to know about scrapping. It is not a black or white issue. It is a grey area and some of us have legitimate uses for it.

**Shpigford** · 08-02-2006, 12:25 PM

Quote:

Originally Posted by Atomm

It is not a black or white issue. It is a grey area and some of us have legitimate uses for it.

Umm...no, it IS black or white. It really is a simple case of unless you have permission (whether it be via Creative Commons or the author themselves), it is illegal.

**lyndoman** · 08-02-2006, 03:36 PM

Ignore this, I screwed up the post

**lyndoman** · 08-02-2006, 03:39 PM

Hmmm, don't remember giving Google permission to put my site in their cache?

So are you saying Google is breaking the law?

Or is it the same old thing that if you have tons of cash you can do whatever the hell you like?

btw Google, if you read this, I really don't mind that you pinch my stuff. So keep sending the peeps.

**Shpigford** · 08-02-2006, 03:52 PM

If you don't want your page cached by Google: http://www.google.com/support/webmas...y?answer=35306