deblan/journalduhacker
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

let spiders crawl all over us

Browse source
This commit is contained in:
Carl Chenet 2017-05-16 19:50:36 +02:00
parent 60ec1d6250
commit 29f0b6a944
2 changed files with 0 additions and 18 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
app/views/layouts/application.html.erb
View file

@ -8,7 +8,6 @@
<link rel="apple-touch-icon" sizes="72x72" href="/apple-touch-icon-144.png" />
<link rel="apple-touch-icon" sizes="144x144" href="/apple-touch-icon-144.png" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="robots" content="noarchive,noodp,noydir" />
<meta name="referrer" content="always" />
<meta name="theme-color" content="#AC130D" />
<% if @meta_tags %>

17
public/robots.txt
View file

@ -1,19 +1,2 @@
# block all spiders by default
User-agent: *
Disallow: /
# but allow major ones
User-agent: Googlebot
Allow: /
User-agent: Slurp
Allow: /
User-Agent: msnbot
Disallow:
User-agent: Baiduspider
Disallow: /
User-agent: *
Disallow: /search