Tycoon Talk
Become a Big fish!
The number 1 forum for online business!
Post topics, ask questions, share your knowledge.
Tycoon Talk is part of Freelancer.com - find skilled workers online at a fraction of the cost.

Website and Server Administration Forum


You are currently viewing our Website and Server Administration Forum as a guest. Please register to participate.
Login



Reply
When Crawlwer hit the Robot.txt file?
Old 02-27-2011, 11:28 PM When Crawlwer hit the Robot.txt file?
Novice Talker

Posts: 14
Name: brad John
Location: Usa
Trades: 0
Last day my friend and me ware discussing one point, i.e when crawler hit the robot.txt. Lot of people say that crawler hit first .htaccesse file and then hit the robot.txt. So my question is when will crawler hit the robot.txt file.
__________________

Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE
brad12 is offline
Reply With Quote
View Public Profile Visit brad12's homepage!
 
 
Register now for full access!
Old 03-02-2011, 02:49 AM Re: When Crawlwer hit the Robot.txt file?
jenifer143's Avatar
Novice Talker

Posts: 14
Name: Jenifer Gracious
Trades: 0
there is no fixed time that when the Crawler will hit the Robot.txt file
__________________

Please login or register to view this content. Registration is FREE
- datStruct
jenifer143 is offline
Reply With Quote
View Public Profile
 
Old 03-02-2011, 07:28 AM Re: When Crawlwer hit the Robot.txt file?
Banned

Posts: 53
Name: AlanMaster4
Trades: 0
its not fix that when crawler hit the robots.txt file.... its hits when crawler crawle our site.
alanmaster4 is offline
Reply With Quote
View Public Profile Visit alanmaster4's homepage!
 
Old 03-02-2011, 12:12 PM Re: When Crawlwer hit the Robot.txt file?
chrishirst's Avatar
Missing! presumed drunk.

Posts: 42,380
Name: Chris Hirst
Location: Blackpool. UK
Trades: 0
.htaccess is NEVER requested by useragents, robots or real visitors

On Apache servers .htaccess is read by the server on each HTTP request, so if a request is made for a file called robots.txt the server will parse the directives in the .htaccess to see how the files is to be delivered in the server response.
__________________
Chris. ->>
Please login or register to view this content. Registration is FREE
<<-

A foolish consistency is the hobgoblin of little minds
Thought for today:- Is SEO the only industry where all the cowboys are Indians?
chrishirst is online now
Reply With Quote
View Public Profile Visit chrishirst's homepage!
 
Reply     « Reply to When Crawlwer hit the Robot.txt file?
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off





   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML



Page generated in 0.33297 seconds with 12 queries