Comments on: robots.txt and all that http://blogaboutyourblog.com/2007/01/14/robotstxt-and-all-that/ Giving you your daily blog fix Thu, 11 Mar 2010 03:55:03 -0700 http://wordpress.org/?v=2.8.4 hourly 1 By: Mr. Apache http://blogaboutyourblog.com/2007/01/14/robotstxt-and-all-that/comment-page-1/#comment-1073 Mr. Apache Sun, 25 Mar 2007 21:30:16 +0000 http://blogaboutyourblog.com/2007/01/14/robotstxt-and-all-that/#comment-1073 Great tips! I saw that same blogging in the robots.txt file and was stunned. AskApache has the best robots.txt for SEO though.. Great tips! I saw that same blogging in the robots.txt file and was stunned. AskApache has the best robots.txt for SEO though..

]]> By: Lee http://blogaboutyourblog.com/2007/01/14/robotstxt-and-all-that/comment-page-1/#comment-9 Lee Mon, 15 Jan 2007 02:12:43 +0000 http://blogaboutyourblog.com/2007/01/14/robotstxt-and-all-that/#comment-9 The 404 page has got to be the most underused bit of website real estate out there. I'm just as guilty too - I run a couple of sites that could easily have cool 404 pages but I just haven't got around to them. The 404 page has got to be the most underused bit of website real estate out there. I’m just as guilty too – I run a couple of sites that could easily have cool 404 pages but I just haven’t got around to them.

]]> By: Ed http://blogaboutyourblog.com/2007/01/14/robotstxt-and-all-that/comment-page-1/#comment-8 Ed Sun, 14 Jan 2007 23:28:18 +0000 http://blogaboutyourblog.com/2007/01/14/robotstxt-and-all-that/#comment-8 So I just wrote a nice list of links in my 404 page to help out robots. Aren't I kind - and misguided! So I just wrote a nice list of links in my 404 page to help out robots. Aren’t I kind – and misguided!

]]> By: Lee http://blogaboutyourblog.com/2007/01/14/robotstxt-and-all-that/comment-page-1/#comment-7 Lee Sun, 14 Jan 2007 12:36:14 +0000 http://blogaboutyourblog.com/2007/01/14/robotstxt-and-all-that/#comment-7 Another very good reason to have a robots.txt file is simply that if you check your server logs and find you have a whole bunch of 404 errors it is a very good chance that's bots looking for this file and not finding it! And yes we have one here LOL - but we don't disallow anything so it's a fairly simple one. Another very good reason to have a robots.txt file is simply that if you check your server logs and find you have a whole bunch of 404 errors it is a very good chance that’s bots looking for this file and not finding it!

And yes we have one here LOL – but we don’t disallow anything so it’s a fairly simple one.

]]>