joshua stein
|
b1e9e359a3
|
block all spiders by default, allow major ones
|
2013-06-30 00:54:02 -05:00 |
|
joshua stein
|
dc71bbbd98
|
block archivers
|
2012-10-24 11:53:43 -05:00 |
|
joshua stein
|
aef2ccefba
|
empty robots for now, let everyone in
|
2012-08-24 11:42:23 -05:00 |
|
joshua stein
|
093747b796
|
initial work on conversion from php tree
|
2012-06-16 20:15:46 -05:00 |
|