• 欢迎访问db1024网站,推荐使用最新版火狐浏览器和Chrome浏览器访问本网站
  • 如果您觉得本站非常有看点,那么赶紧使用Ctrl+D 收藏db1024吧

WordPress博客网站常用Robots.txt规则

主机教程 Liam 1年前 (2019-05-01) 421次浏览 12个评论 扫描二维码

说明:我们有时候页面有些内容由于某些因素,都要防止被搜索引擎收录,而禁止搜索引擎收录比较简单的办法是在Robots.txt里设置规则,Robots协议(也称为爬虫协议、机器人协议等)的全称是“网络爬虫排除标准”(Robots Exclusion Protocol),网站通过Robots协议告诉搜索引擎哪些页面可以抓取,哪些页面不能抓取。这里说下方法。

这里以WordPress博客网站为例,在网站根目录Robots.txt里添加如下规则:

User-agent: * Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /*/comment-page-* Disallow: /*?replytocom=* Disallow: /category/*/page/ Disallow: /tag/*/page/ Disallow: /*/trackback Disallow: /feed Disallow: /*/feed Disallow: /comments/feed Disallow: /?s=* Disallow: /*/?s=* Disallow: /attachment/ 

下面分别解释一下各项的作用:

1、Disallow: /wp-admin/、Disallow: /wp-content/和Disallow: /wp-includes/ 用于告诉搜索引擎不要抓取后台程序文件页面。  2、Disallow: /*/comment-page-*和Disallow: /*?replytocom=* 禁止搜索引擎抓取评论分页等相关链接。  3、Disallow: /category/*/page/和Disallow: /tag/*/page/ 禁止搜索引擎抓取收录分类和标签的分页。  4、Disallow: /*/trackback 禁止搜索引擎抓取收录trackback等垃圾信息  5、Disallow: /feed、Disallow: /*/feed和Disallow: /comments/feed 禁止搜索引擎抓取收录feed链接,feed只用于订阅本站,与搜索引擎无关。  6、Disallow: /?s=*和Disallow: /*/?s=* 禁止搜索引擎抓取站内搜索结果  7、Disallow: /attachment/ 禁止搜索引擎抓取附件页面,比如毫无意义的图片附件页面。 

上面只是些基本的写法,当然还有很多,不过够用了。当然像TypechoEmlogZblog这些博客就自己看着办,方法一样。


db1024 , 版权所有丨如未注明 , 均为原创丨本网站采用BY-NC-SA协议进行授权
转载请注明原文链接:WordPress博客网站常用Robots.txt规则
喜欢 (1)
关于作者:
发表我的评论
取消评论
表情 贴图 加粗 删除线 居中 斜体 签到

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
(12)个小伙伴在吐槽
  1. Hello Dear, are you in fact visiting this website daily, if so afterward you will without doubt obtain pleasant knowledge.
    website hosting companies2020-09-05 18:51 回复 Mac OS X | Firefox浏览器 61.0
  2. Spot on with this write-up, I absolutely feel this web site needs a lot more attention. I'll probably be back again to see more, thanks for the advice!
    web hosting providers2020-08-31 20:25 回复 Windows 7 | Opera 9.80
  3. Appreciate the recommendation. Let me try it out.
    black mass2020-08-28 20:04 回复 Mac OS X | Chrome 67.0.3396.79
  4. Greetings! Very useful advice within this post! It is the little changes that produce the greatest changes. Many thanks for sharing!
    cheap flights2020-08-28 04:28 回复 Linux | Chrome 59.0.3071.86
  5. That is a great tip particularly to those fresh to the blogosphere. Simple but very accurate info… Thank you for sharing this one. A must read post!
    cheap flights2020-08-27 18:37 回复 Linux | Firefox浏览器 60.0.2
  6. I am curious to find out what blog system you have been using? I'm experiencing some minor security issues with my latest blog and I'd like to find something more safeguarded. Do you have any recommendations? cheap flights 3aN8IMa
    cheap flights2020-08-27 04:50 回复 Windows 8.1 | Firefox浏览器 60.0
  7. Hi there i am kavin, its my first occasion to commenting anywhere, when i read this article i thought i could also make comment due to this good post. y2yxvvfw cheap flights
    cheap flights2020-08-25 15:15 回复 Mac OS X | Chrome 65.0.3325.181
  8. I really like your blog.. very nice colors & theme. Did you design this website yourself or did you hire someone to do it for you? Plz respond as I'm looking to create my own blog and would like to find out where u got this from. appreciate it cheap flights 3aN8IMa
    cheap flights2020-08-25 07:28 回复 Mac OS X | Firefox浏览器 61.0
  9. My brother recommended I might like this blog. He was once entirely right. This put up actually made my day. You cann't imagine simply how much time I had spent for this information! Thank you!
    cheap flights2020-08-24 14:43 回复 Mac OS X | Chrome 51.0.2704.84
  10. Have you ever thought about writing an ebook or guest authoring on other blogs? I have a blog centered on the same ideas you discuss and would really like to have you share some stories/information. I know my readers would value your work. If you're even remotely interested, feel free to send me an email.
    best web hosting2020-08-14 06:27 回复 Linux | Firefox浏览器 2.0.0.6
  11. Outstanding quest there. What occurred after? Good luck!
    web hosting company2020-08-12 04:52 回复 Linux | Chrome 66.0.3359.106
  12. Have you ever considered about including a little bit more than just your articles? I mean, what you say is fundamental and everything. However just imagine if you added some great visuals or videos to give your posts more, "pop"! Your content is excellent but with images and videos, this website could definitely be one of the best in its niche. Great blog! adreamoftrains best hosting
    adreamoftrains best web hosting 20202020-08-11 11:10 回复 Linux | Firefox浏览器 52.9