Carl Chenet
|
29f0b6a944
|
let spiders crawl all over us
|
2017-05-16 19:50:36 +02:00 |
|
joshua stein
|
ce8eb45008
|
block robots from /search
|
2015-01-11 12:59:41 -06:00 |
|
|
joshua stein
|
b1e9e359a3
|
block all spiders by default, allow major ones
|
2013-06-30 00:54:02 -05:00 |
|
|
joshua stein
|
dc71bbbd98
|
block archivers
|
2012-10-24 11:53:43 -05:00 |
|
|
joshua stein
|
aef2ccefba
|
empty robots for now, let everyone in
|
2012-08-24 11:42:23 -05:00 |
|
|
joshua stein
|
093747b796
|
initial work on conversion from php tree
|
2012-06-16 20:15:46 -05:00 |
|