Tycoon Talk
Become a Big fish!
The number 1 forum for online business!
Post topics, ask questions, share your knowledge.
Tycoon Talk is part of Freelancer.com - find skilled workers online at a fraction of the cost.

Coding Forum


You are currently viewing our Coding Forum as a guest. Please register to participate.
Login



Reply
Differentiating between crawler and genuine
Old 04-29-2009, 02:50 AM Differentiating between crawler and genuine
Skilled Talker

Posts: 64
Trades: 0
For simplicity sake I'm going to say that I'm making a hit counter.

However, because I have Google Ads on my page, the 66.249.73.XX IP of the Googlebot pops up on my logs a few seconds after a genuine hit because the ads have appeared. Obviously, the IPs of all other web crawlers that visit my site are logged as well.

Is there any way (through PHP preferably or I'm open to other methods) to differentiate between a genuine user visiting a site and a web crawler's hit? I want to only add hits for genuine people.

Blocking crawlers through robots.txt or any other method isn't wanted.
Petsmacker is offline
Reply With Quote
View Public Profile
 
 
Register now for full access!
Old 04-29-2009, 03:00 AM Re: Differentiating between crawler and genuine
nayes84's Avatar
Extreme Talker

Latest Blog Post:
Difference between ASP And JSP
Posts: 232
Name: John
Location: Tokyo
Trades: 0
You can distinguish between bots and normal users from posted http headers.
normal users would have words like firefox,ie,opera,Gecko, etc..
Code:
user-agent Mozilla/5.0 (Windows; U; Windows NT 5.1; ja; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8 
search bots should have some thing different mostly including the word bot
Code:
user-agent Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
You can't get 100% accurate data but you can over time improve your statistics accuracy by keeping adding search bots headers to your script to exclude them
__________________

Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE

if(I'm("Helpful")) Add_Talkupation("nayes84");

Last edited by nayes84; 04-29-2009 at 03:01 AM..
nayes84 is offline
Reply With Quote
View Public Profile
 
Reply     « Reply to Differentiating between crawler and genuine
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off





   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML



Page generated in 0.34493 seconds with 12 queries