|
I don't know if Blogger will allow this, but I've been thinking it might be a good idea to add a robots.txt file to my blog, and make sure Google isn't crawling my RSS feed and seeing duplicate content. Have been doing homework on this, and I found something interesting that I didn't know.
robots.txt ( all lower case ) has to be in Unix format, with a LF or for VBScript people Chr$(10) for line endings. Not CR+LF - Chr$(13) + Chr$(10) like in Windows - and not simply CR or Chr$(13) like on the Mac, who just had to be different. Your html can be in whatever format you want, Unicode or ASCII, but robots.txt is expected to be a particular way, and while some programs will recognize other formats, a lot of spiders won't.
|
