踢掉google和baidu爬虫!

在网站根目录下建立robots.txt文件,写入:

User-agent: GoogleBot
Disallow: *
User-agent: googleBot
Disallow: *
User-agent: Baiduspider
Disallow: *
User-agent: baiduspider
Disallow: 

 

User-agent: GoogleBotDisallow: *
User-agent: googleBotDisallow: *

 

User-agent: BaiduspiderDisallow: *
User-agent: baiduspiderDisallow: *

posted @ 2010-05-04 16:05  jiangyao  阅读(413)  评论(0)    收藏  举报