Tycoon Talk
Become a Big fish!
The number 1 forum for online business!
Post topics, ask questions, share your knowledge.
Tycoon Talk is part of Freelancer.com - find skilled workers online at a fraction of the cost.

Website and Server Administration Forum


You are currently viewing our Website and Server Administration Forum as a guest. Please register to participate.
Login



Reply
Old 06-24-2009, 10:43 PM Search Spiders
Experienced Talker

Posts: 30
Trades: 0
I am in the process of setting up my first website and wanted to start uploading my pages to a server soon. However, I wanted to test them on the server and make sure that everyone involved with the project was happy before any search engines crawl the page.

My question is, if I don't submit the page to Google, Yahoo and other search engines do I need to worry about them crawling my page or do I need to setup an htcacess file to keep them at bay?
samo1215 is offline
Reply With Quote
View Public Profile
 
 
Register now for full access!
Old 06-25-2009, 12:47 PM Re: Search Spiders
Skilled Talker

Posts: 62
Location: North Yorkshire
Trades: 0
If no one links to you and you don't tell the search engines then no one knows you exist so test away.
__________________

Please login or register to view this content. Registration is FREE



Please login or register to view this content. Registration is FREE
Scarthingwell is offline
Reply With Quote
View Public Profile Visit Scarthingwell's homepage!
 
Old 06-25-2009, 05:34 PM Re: Search Spiders
hairygunther's Avatar
Extreme Talker

Posts: 164
Trades: 0
Quote:
do I need to worry about them crawling my page or do I need to setup an htcacess file to keep them at bay?
To be safe, set them up in folders just for testing - then move the files to the main area. After all, there's various ways google can find a link to your site, eg your own carelessness. HTaccess is a bad idea because then if it gets crawled google has it pegged as blocked, and you have to wait for it to recrawl and who knows whether you'll get unlucky about the time delay. Leaving it as it is and taking a risk is also bad, simply because it is careless, unprofessional and shoddy.

This is a trivial question worth far less ascii than I have dedicated to it. I couldn't ever see myself ever having asked such trivial questions anywhere on my journey to development expertise... I think you need to try and rely more on your mind and more on your own drive and less on the (often incorrect) advice of those in an arena (social media / consumer generated media) with over 75% deceptive messages, written with the primary purpose of selling in mind.
hairygunther is offline
Reply With Quote
View Public Profile
 
Old 06-26-2009, 04:32 PM Re: Search Spiders
konetch's Avatar
Ultra Talker

Posts: 258
Trades: 0
even if you're careless it will probably take google more than a week to index your site, so if you can get everything done within that time you won't have a problem
__________________
Alex
konetch is offline
Reply With Quote
View Public Profile
 
Old 06-28-2009, 01:11 PM Re: Search Spiders
L a r r y's Avatar
Super Talker

Posts: 100
Name: Larry K
Location: Arizona
Trades: 0
Robots.txt should keep the spiders away from the test areas, and then just transfer agreed-upon content from the test folders into the final resting place on the site.
L a r r y is offline
Reply With Quote
View Public Profile Visit L a r r y's homepage!
 
Reply     « Reply to Search Spiders
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off





   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML



Page generated in 0.76535 seconds with 12 queries