Tycoon Talk
Become a Big fish!
The number 1 forum for online business!
Post topics, ask questions, share your knowledge.
Tycoon Talk is part of Freelancer.com - find skilled workers online at a fraction of the cost.

The Google Forum


You are currently viewing our The Google Forum as a guest. Please register to participate.
Login



Reply
robots.text validator
Old 11-14-2006, 01:57 PM robots.text validator
Novice Talker

Posts: 8
Name: Kenneth W. Lowry
Trades: 0
I have tried to make a robots.txt file but it won't validate. My main reason is because the site has some pages that are afilliate links and I read that google looks at them as portals.
In any event, I first wrote the file myself with first line being the User-agent: * line, which returned syntax errors at http://tool.motoricerca.info/robots-checker.phtml Then I used a generator which generated the file exactly like the one created with the first line # robots.text file created by. . . And that too returned syntax eroors for that first line. So i ended up with this:

# My robots.txt file
User-agent: *
Disallow: /Merchandise.html/
Disallow: /iTunes.html/
Disallow: /Mall.html/
Disallow: /Mall_2.html/

Which again returnded the first line error. What is wrong?

http://www.satinstyle.net
kloco is offline
Reply With Quote
View Public Profile
 
 
Register now for full access!
Old 11-14-2006, 07:06 PM Re: robots.text validator
vangogh's Avatar
Post Impressionist

Latest Blog Post:
Why Responsive Design?
Posts: 10,815
Name: Steven Bradley
Location: Boulder, Colorado
Trades: 0
I think you might just need to remove the trailing slash since you're trying to disallow specific files and not folders. Are you using a text editor to create the file to make sure there are no extra characters in the file?

You don't need that first line. It's just a comment so it's safe to remove it.
__________________
l Search Engine Friendly Web Design |
Please login or register to view this content. Registration is FREE

l Tips On Marketing, SEO, Design, and Development |
Please login or register to view this content. Registration is FREE

l
Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE
vangogh is offline
Reply With Quote
View Public Profile Visit vangogh's homepage!
 
Old 11-14-2006, 09:01 PM Re: robots.text validator
Experienced Talker

Posts: 40
Trades: 0
Example:

Analyzing file

http://multifinanceit.com/robots.txt

No errors found in this robots.txt file

Look at that file.
__________________
Kjell Gunnar Bleivik::
Please login or register to view this content. Registration is FREE


Please login or register to view this content. Registration is FREE


Please login or register to view this content. Registration is FREE
kgun is offline
Reply With Quote
View Public Profile
 
Old 11-14-2006, 10:31 PM Re: robots.text validator
Novice Talker

Posts: 8
Name: Kenneth W. Lowry
Trades: 0
I believe I shoukd not have it a validator- the above tool is an analyzer. When I take out the first line the User-agent returns an error. The (/) at the end passes the analyzer.
I took out both the final (/) and the first line.
so now:

User-agent: *
This line doesn't follow a correct syntax. The correct syntax is: <field>:<optionalspace><value><optionalspace> , where "field" can be "User-agent" or "Disallow". Please refer to Robots Exclusion Standard page for more informations.

Regardless of what is in the first line, this is the same error message. However, without another first line the rest of the script brings back errors - no valid User-agent
kloco is offline
Reply With Quote
View Public Profile
 
Old 11-15-2006, 01:13 AM Re: robots.text validator
Link.ezer.com's Avatar
Novice Talker

Posts: 5
Location: www.ezer.com
Trades: 0
Delete your robots.txt file.
and
Try to use note pad to create your new robots.txt
__________________
(
Please login or register to view this content. Registration is FREE
)
|
Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE

Last edited by Link.ezer.com; 11-15-2006 at 01:22 AM..
Link.ezer.com is offline
Reply With Quote
View Public Profile Visit Link.ezer.com's homepage!
 
Old 11-15-2006, 10:21 AM Re: robots.text validator
Novice Talker

Posts: 8
Name: Kenneth W. Lowry
Trades: 0
I used notepad That is what the file is on now.
kloco is offline
Reply With Quote
View Public Profile
 
Old 11-15-2006, 12:46 PM Re: robots.text validator
vangogh's Avatar
Post Impressionist

Latest Blog Post:
Why Responsive Design?
Posts: 10,815
Name: Steven Bradley
Location: Boulder, Colorado
Trades: 0
It looks like all the errors you're getting are coming from those extra characters at the beginning of the User-agent line. For whatever reason they somehow got into the file. It might be easiest to just delete the file and upload a new one. Don't use the one you have now and write a new robots.txt file in notepad.

If you're still seeing the same error those characters might be getting added when you're FTPing to the server. How are you uplaoding the file?
__________________
l Search Engine Friendly Web Design |
Please login or register to view this content. Registration is FREE

l Tips On Marketing, SEO, Design, and Development |
Please login or register to view this content. Registration is FREE

l
Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE
vangogh is offline
Reply With Quote
View Public Profile Visit vangogh's homepage!
 
Old 11-15-2006, 04:08 PM Re: robots.text validator
Novice Talker

Posts: 8
Name: Kenneth W. Lowry
Trades: 0
I did that. I still get those characters in the analysis. I have been using both Dreamweaver 8 and Core FTP Lite. This is very frustrating.
kloco is offline
Reply With Quote
View Public Profile
 
Old 11-15-2006, 08:13 PM Re: robots.text validator
vangogh's Avatar
Post Impressionist

Latest Blog Post:
Why Responsive Design?
Posts: 10,815
Name: Steven Bradley
Location: Boulder, Colorado
Trades: 0
It might be in the way the programs are uploading the file to the server. I'm not sure where it would be in either program, but look for the settings about the program sends file through FTP. The choices should be to upload in either binary or ascii format (and maybe an option to automaticlly decide). Make sure to upload the file with the binary option since the ascii option can sometimes add the extra characters.

Hopefully that will work, but if not I'll see if I can think of something else that might work.
__________________
l Search Engine Friendly Web Design |
Please login or register to view this content. Registration is FREE

l Tips On Marketing, SEO, Design, and Development |
Please login or register to view this content. Registration is FREE

l
Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE
vangogh is offline
Reply With Quote
View Public Profile Visit vangogh's homepage!
 
Old 11-16-2006, 07:26 AM Re: robots.text validator
Novice Talker

Posts: 8
Name: Kenneth W. Lowry
Trades: 0
It works now. The poblem was in the encoding. The notepad.txt was encoded UTF-8. When I changed it to ANSI it didn't add the extra characters.

Thanks for your help!

Last edited by kloco; 11-16-2006 at 07:27 AM.. Reason: dropped character
kloco is offline
Reply With Quote
View Public Profile
 
Old 11-16-2006, 01:47 PM Re: robots.text validator
vangogh's Avatar
Post Impressionist

Latest Blog Post:
Why Responsive Design?
Posts: 10,815
Name: Steven Bradley
Location: Boulder, Colorado
Trades: 0
Glad you got it all worked out. I hadn't even thought to check that, but it's good to know if anyone has the problem in the future.
__________________
l Search Engine Friendly Web Design |
Please login or register to view this content. Registration is FREE

l Tips On Marketing, SEO, Design, and Development |
Please login or register to view this content. Registration is FREE

l
Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE
vangogh is offline
Reply With Quote
View Public Profile Visit vangogh's homepage!
 
Reply     « Reply to robots.text validator
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off





   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML



Page generated in 0.47222 seconds with 12 queries