PHP Forum

**ibbo** · 08:13 AM

Code:

<?php

// author: Mark Ibbotson
// email : 
error_reporting(E_COMPILE_ERROR|E_ERROR|E_CORE_ERROR);

class Search{

	var $_domain = "";
	var $_keyword = "";
	var $_links = Array();
	var $_matched = Array();
	var $_usefull=Array();

	function Search(){
	}

	function openSite(){
    // opens DOMAIN and scans the home page for potential site links 
	// looking for short links I.E "/\/[a-z0-9]*/i"	/support and not www.DOMAIN.com/support
		
		if(isset($this->_domain)){

			$page = file($this->_domain);

			foreach($page as $val){

				if(preg_match("/\/[a-z0-9]*/i", $val, $m)){
					// add matched short links to array for future consideration
					array_push($this->_links, $m[0]);
				}elseif(preg_match("/\/[a-z0-9]*\/[a-z0-9]*/i", $val, $mn)){
					array_push($this->_links, $mn[0]);
				}
			}
		}
	}

	function findPageOnSearch(){
		// provide a list of links of matched pages and search terms
		
		foreach($this->_links as $key=>$val){
		
			// remove duplicates from top level link array
			if(!in_array($val,$this->_usefull)){
				// usefull, unique list of short links
				array_push($this->_usefull, $val);
			}
		}

        // lets make sure we dont parse any pages that dont require parsing. 
        // css directory, image directory etc
                
		foreach($this->_usefull as $key=>$val){
			
			if($val != "/css" && $val != "/js" && $val != "/images"){
				// if none of the above pages grab the current page
				
				$page = file_get_contents($this->_domain . $val);

				// can we match our keyword in that page
				if(strstr($page, $this->_keyword)){
					if($val == "desktop") echo $val;
					// if so lets display that match and a link to the page
					echo "MATCHED : $this->_keyword<hr />";
					echo "<a href=" . $this->_domain . $val .">" . $this->_domain.$val . "</a> <br />"; 
					echo "<hr />";
				}
			}
		}
	}

}

// ensure POST is set before kick off
if(!empty($_POST)){

	if(isset($_POST['domain']) && isset($_POST['keyword'])){
		// new search obj.
		$search = new Search();

		// search vars
		$search->_domain = $_POST['domain'];
		$search->_keyword = $_POST['keyword'];

		// dog work methods
		$search->openSite();
		$search->findPageOnSearch();
	}
}
?>
<html>
<head>
<title>Site search</title>
<body>
 <form method="post" action="search.php">
  <table border="0">
   <tr><td>Domain</td><td><input type="text" name="domain" value="http://domain.com" /></td></tr>
   <tr><td>Keyword</td><td><input type="text" name="keyword" value="" /></td></tr>
   <tr><td></td><td><input type="submit" value="Go" /></td></tr>
  </table>
 </form>
</body>
</html>

This is in the prtotype stage and its CASE SENSATIVE (add str_tolower where appropriate if you use it).

It works like thus:

Enter domain name (http://) included.

The script then goes away and preg's all /[a-z0-9]* looking for short links like href="/support" which it then adds to $_links.

$_links is then polled and all duplicates are removed leaving us with $_usefull.

Poll $_usefull and open each page it referes to. A simple strstr on that page to match your keyword search and hey presto it lists linbks with your keyword.

Of course its far from been a google worthy search engine (and it seriously lacks features bar matching keywords).

Anyway its there for you all to chop to bits improve upon and use at your leisure.

Any mods or ideas to improve it would also be welcome.

Hope it helps some of you, it certainly is usefull to me.

Ibbo

**ibbo** · 08:12 AM

PHP Code:


			
<?php

// author: Mark Ibbotson
// email :

error_reporting(E_COMPILE_ERROR|E_ERROR|E_CORE_ERROR);

class Search{

    var $_domain = "";
    var $_keyword = "";
    var $_links = Array();
    var $_matched = Array();
    var $_usefull=Array();
    var $_case = fasle;

    function Search(){}

    /*
     * Purpose: Grabs a root webpage
     * Params : null
    */
    function openSite(){
        // opens DOMAIN and scans the home page for potential site links
        // looking for short links I.E "/\/[a-z0-9]*/i"    /support and not www.DOMAIN.com/support
        if(isset($this->_domain)){
            $this->findLinks(file($this->_domain));
        }
    }

    /*
     * Purpose: Poll a page for possible links
     * Params : $page page of interest
    */
    function findLinks($page){
        // we are looking for dir's to add to our list for future consideration
        foreach($page as $val){
            // 1 dir deep
            if(preg_match("/\/[a-z0-9]*/i", $val, $m)){
                array_push($this->_links, $m[0]);
            }
            // 2 deep
            if(preg_match("/\/[a-z0-9]*\/[a-z0-9]*/i", $val, $mn)){
                array_push($this->_links, $mn[0]);
            }
            // 3 deep
            if(preg_match("/\/[a-z0-9]*\/[a-z0-9]*\/[a-z0-9]*/i", $val, $mn)){
                array_push($this->_links, $mn[0]);
            }
            // 4 deep
            if(preg_match("/\/[a-z0-9]*\/[a-z0-9]*\/[a-z0-9]*\/[a-z0-9]*/i", $val, $mn)){
                array_push($this->_links, $mn[0]);
            }
        }
    }

    /*
     * Purpose: Parse link array, remove duplicates
     *          Investigate sub pages for further links   
     * Params : NULL
    */
    function parseLinks(){
        // provide a list of links of matched pages and search terms
        foreach($this->_links as $key=>$val){
            // remove duplicates from top level link array
            if(!in_array($val,$this->_usefull)){
                // usefull, unique list of short links
                array_push($this->_usefull, $val);
                // lets see if sub pages hold any links for us to explore
            }
        }
    }

    /*
     * Purpose: Poll usefull links and open each page therein.
     *          match keywords on that page, if a match is found echo a link
     * Params : NULL
    */
    function showResults(){
        echo "Please be patient this could take several mninutes<br />";
        echo "Results for search " . $this->_keyword;
        if($this->_case){ echo " using case insensitive match";}
        echo "<br />";

        // grab sub page links and restart, 
        // killer on time as each sub page needs to be polled for links within it
        // then those links need to be scanned for keyword match
        foreach($this->_usefull as $key=>$val){
            $this->findLinks(file($this->_domain . $val));
           $this->parseLinks();
        }
        
        foreach($this->_usefull as $key=>$val){
            // lets make sure we dont parse any pages that dont require parsing.
            // css directory, image directory etc
            if($val != "/css" && $val != "/js" && $val != "/images" && $val != "/W3C"){
                // if none of the above pages grab the current page
                $page = file_get_contents($this->_domain . $val);
                
                // convert to lowercase on case insesitive match
                if($this->_case){
                    $page = strtolower($page);
                    $this->_keyword = strtolower($this->_keyword);
                }
                // can we match our keyword in that page
                if(strstr($page, $this->_keyword)){
                    // if so lets display that match and a link to the page
                    echo "<a href=" . $this->_domain . $val .">" . $this->_domain.$val . "</a> <br />";
                }
            }
        }
    }
}

// ensure POST is set before kick off
if(!empty($_POST)){

    if(isset($_POST['domain']) && isset($_POST['keyword'])){
        // new search obj.
        $search = new Search();

        // search vars
        $search->_domain = $_POST['domain'];
        $search->_keyword = $_POST['keyword'];
        isset($_POST['case']) ? $search->_case = true : $search->_case = false;

        // dog work methods
        $search->openSite();
        $search->parseLinks();
        $search->showResults();
    }
}
?>

<html>
<head>
<title>Site search</title>
<body>
 <form method="post" action="search.php">
  <table border="0">
   <tr><td>Domain</td><td><input type="text" name="domain" value="http://www.ubuntu.com" /></td></tr>
   <tr><td>Keyword</td>
       <td><input type="text" name="keyword" value="" />
           <input type="checkbox" name="case" /> case insensitive
       </td>
   </tr>
   <tr><td></td><td><input type="submit" value="Go" /></td></tr>
  </table>
 </form>
</body>
</html>

A modified version that polls sub pages for links and follows them too.

Penalty, takes approx 3 minutes to search "BUT if its there i'm 91% sure it will find it".

Ibbo

**mgraphic** · 06-27-2006, 12:18 PM

Quote:

"BUT if its there i'm 91% sure it will find it"

The other 9% is sure that you forgot to put it there in the first place!

**ibbo** · 04:52 AM

PHP Code:


			
function findLinks($page){
        // we are looking for dir's to add to our list for future consideration
        foreach($page as $val){
            // 1 dir deep
            if(preg_match("/\/[a-z0-9]*/i", $val, $m)){
                array_push($this->_links, $m[0]);
            }
            // 2 deep
            if(preg_match("/\/[a-z0-9]*\/[a-z0-9]*/i", $val, $mn)){
                array_push($this->_links, $mn[0]);
            }
            // 3 deep
            if(preg_match("/\/[a-z0-9]*\/[a-z0-9]*\/[a-z0-9]*/i", $val, $mn)){
                array_push($this->_links, $mn[0]);
            }
            // 4 deep
            if(preg_match("/\/[a-z0-9]*\/[a-z0-9]*\/[a-z0-9]*\/[a-z0-9]*/i", $val, $mn)){
                array_push($this->_links, $mn[0]);
            }
        }
    }

Modify this method to take another param for depth. As it only searched 4 dir's deep at the moment I did not feel confident saying 100%. However if you pass a depth parameter to this method and loop a preg_match instead of the default 4 in there and you will soon start to reach your 100% success rate.

However the penalty will be an increase in crawl time (for thats all it is in essence). In its present state it takes just short of 3 mins to crawl 4 dirs deep. Increase this to six and your probably going to be sat waiting for 6+ mins.

Its a case of how low do you go.

Give it a try on any site, you will find it finds your keywords and will list the pages it finds them on.

Ibbo