robots.txt and all that
- 4 Comment
Google indexes all sites using robots, and these critters’ first port of call is the robots.txt file.
Have you got any idea what your blog’s robots.txt file has got written in it? If you don’t have your own domain, there is probably no need to know, but I suspect the team who put this revised Blogaboutyourblog.com theme together know very well what theirs says. And over at The Pisstakers, we do too, having recently gone through the site with a fine tooth comb after a mega theme and feature update this month.
So go check out the robots.txt deal on your server, in case Google catch you out like they did Mybloglog blogger, Jangro, whose search engine traffic died over the course of a couple of days due to a slightly mis-interpreted line of code. Google have plenty of info for webmasters explaining what the file should say, or more importantly what the code shouldn’t disallow.
Blogging in weird places
And before you fall asleep from geek overload, I found a high flying webmaster who actually uses the robots.txt file to blog in! The Bot Blog has the craziest location of them all, and the robots don’t mind a bit – they can’t see it right in front of their noses because he knows the simple code to hide the text.
Random Posts
4 Comments on this post
Trackbacks
-
Lee said:
Another very good reason to have a robots.txt file is simply that if you check your server logs and find you have a whole bunch of 404 errors it is a very good chance that’s bots looking for this file and not finding it!
And yes we have one here LOL – but we don’t disallow anything so it’s a fairly simple one.
January 14th, 2007 at 12:36 pm -
Ed said:
So I just wrote a nice list of links in my 404 page to help out robots. Aren’t I kind – and misguided!
January 14th, 2007 at 11:28 pm -
Lee said:
The 404 page has got to be the most underused bit of website real estate out there. I’m just as guilty too – I run a couple of sites that could easily have cool 404 pages but I just haven’t got around to them.
January 15th, 2007 at 2:12 am -
Mr. Apache said:
Great tips! I saw that same blogging in the robots.txt file and was stunned. AskApache has the best robots.txt for SEO though..
March 25th, 2007 at 2:30 pm