Comments on: How to write robot.txt to control search engine spider http://www.techiecorner.com/18/write-robot-txt-to-control-search-engine-spider/ The place for computer tips and tricks! microsoft windows, open source, database, programming, freeware and etc Wed, 17 Mar 2010 21:18:27 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Ozymandiashttp://www.techiecorner.com/18/write-robot-txt-to-control-search-engine-spider/comment-page-1/#comment-113663 Ozymandias Tue, 26 May 2009 08:37:26 +0000 http://www.techiecorner.com/18/how-to-write-robottxt-to-control-search-engine-spider/#comment-113663 How do I make spiders list my website? How do I make spiders list my website?

]]> By: Bradhttp://www.techiecorner.com/18/write-robot-txt-to-control-search-engine-spider/comment-page-1/#comment-83856 Brad Tue, 16 Sep 2008 17:12:59 +0000 http://www.techiecorner.com/18/how-to-write-robottxt-to-control-search-engine-spider/#comment-83856 Who do I prevent my admin controls from being spidered? Who do I prevent my admin controls from being spidered?

]]> By: how to write a robothttp://www.techiecorner.com/18/write-robot-txt-to-control-search-engine-spider/comment-page-1/#comment-63418 how to write a robot Thu, 29 May 2008 12:51:40 +0000 http://www.techiecorner.com/18/how-to-write-robottxt-to-control-search-engine-spider/#comment-63418 [...] should go thru your site and which spider is not allow to visit your site.Example of a robot.txthttp://www.techiecorner.com/18/write-robot-txt-to-control-search-engine-spider/Robot reporter &39to write news in future&39 UK news The GuardianIn a few years, newspapers could be [...] [...] should go thru your site and which spider is not allow to visit your site.Example of a robot.txthttp://www.techiecorner.com/18/write-robot-txt-to-control-search-engine-spider/Robot reporter &39to write news in future&39 UK news The GuardianIn a few years, newspapers could be [...]

]]>