deblan/journalduhacker
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

block all spiders by default, allow major ones

Browse source
This commit is contained in:
joshua stein 2013-06-25 13:56:31 -05:00
parent ed512cc065
commit b1e9e359a3
1 changed files with 14 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

17
public/robots.txt
View file

@ -1,5 +1,16 @@
# no caching
User-agent: ia_archiver
# block all spiders by default
User-agent: *
Disallow: /
User-agent: archive.org_bot
# but allow major ones
User-agent: Googlebot
Allow: /
User-agent: Slurp
Allow: /
User-Agent: msnbot
Disallow:
User-agent: Baiduspider
Disallow: /