Hey all,

I am trying to extract plain text from within HTML tags.

The script starts out with a spider:

PHP Code:

<?php
class spider  
    {   
     // This class grabs the content from the sites                  
     function setup()    
         {
         $cookieJar = 'cookies.txt';                 
         curl_setopt($this->curl,CURLOPT_COOKIEJAR, $cookieJar);         
         curl_setopt($this->curl,CURLOPT_COOKIEFILE, $cookieJar);        
         curl_setopt($this->curl,CURLOPT_AUTOREFERER,true);        
         curl_setopt($this->curl,CURLOPT_TIMEOUT,30);        
         curl_setopt($this->curl,CURLOPT_CONNECTTIMEOUT,25);        
         curl_setopt($this->curl,CURLOPT_FOLLOWLOCATION,true);        
         curl_setopt($this->curl,CURLOPT_RETURNTRANSFER, true);          
         }        
     function get($url)    
         {         
         $this->curl = curl_init($url);        
         $this->setup();                 
         return $this->request();    
         }        
     function request()    
         {        
         return curl_exec($this->curl);    
         }  
     }
    $spider=new spider();
    $link=$spider->get("http://www12.statcan.ca/census-recensement/2006/dp-pd/prof/92-597/P3.cfm?Lang=E&CTCODE=4707&CATYPE=CMA");
    ?>

This works fine. The problem I am having is identifying the correct functions to use for actually parsing the information.

I was trying to use 'substr()' and a loop, but I dont know how I should go about identifying the start point ('<td class="alignBottomRight col_1"' plus an increasing row id that changes which I will need to take into account) and the end point ('</td>').

Do you guys have any suggestions as far as how I can set this up to retrieve the data, or of any good pre-made scripts that might be of use here?

Thanks!

How about going about it in reverse?

http://php.net/manual/en/function.strip-tags.php

Hey Chris,

The problem is that the first half of the script gives me the source code for the entire page. I am then trying to pull data out of one specific column on that page. So by keeping the tags I am able to use that to identify the correct column, but I am just not sure how to go about doing this.

Thanks for you help!

Can the document be parsed as a XML tree?

Chris how would I do that?

I am pulling the data from a government website and this page doesnt use XML. They do have an option to export to CSV but unfortunately they use a unique identifier on the link that doesnt seem to have any relevance other than as an ID.

If the page uses a XHTML Strict DTD you should be able cURL the source to a variable and read the node content using the ID attribute.

(in theory)

Wow that sounds intense . Do you know of a good turotial or example that explains this?

Shouldnt there be a simpler way to do it using substr() and some other function?

http://www.php.net/manual/en/domdocu...lementbyid.php

Nice.

So I gave that a try:

$doc = new DomDocument;

// We need to validate our document before refering to the id
$doc->validateOnParse = true;
$doc->Load($link);

echo "The element whose id is books is: " . $doc->getElementById('col_1')->tagName . "\n";

The problem is that they dont have a unique ID for them

I tried using getElementsByTagName() but it didnt return anything.

Chris you rock!

The problem is that they dont have a unique ID for them

That's quite unsporting of them

What does the document source look like and is there something unique about the section you want?

Regular expressions are "greedy" by nature so may get too much, so it will be a case of using strpos to get the start position and the length, then use substr() to extract the data.

Haha that is rather unsporting of them isn't it?

I am trying to pull information from Statistics Canada on different census tracts. Here is an example of a page for one of the census tracts: http://www12.statcan.ca/census-recen...707&CATYPE=CMA

I want to take all the information from the column marked '0109.01 (CT)' and add it to an array so that i can output it to a CSV file once I have done this for all the necessary census tracts (about 100).

If all else fails I can spend a Saturday copy pasting and then formatting everything but that kind of sucks, plus I would like to figure this out in case I ever need to do something like this again in the future.

strip_tags()?

How about looking at

http://wonshik.com/snippet/Convert-H...to-a-PHP-Array and http://www.phpbuilder.com/board/show...php?t=10313404

Chris the links from the second site to the script dont seem to work but the first script looks great!

I won't have a chance to try it until Friday but I will let you know how it works out when I do.

Thanks for all your help

strip_tags() for a small chunk of HTML. For large document such as an entire web page, preg_match() or preg_match_all() with regular expressions are better in dealing with complicated situations.

Sorry Chris but I could use some hand holding on this because I have never really worked with classes before and the instructions are confusing me more

Should 'var $source = NULL;' actually be equal to the URL of the page? And then just change the anchor variables up top there too?

Sorry, I feel dumb asking this question