← Back to branch summary

~widelands-dev/widelands-website/trunk

« back to all changes in this revision

Viewing changes to templates/robots.txt

  • Committer: kaputtnik
  • Date: 2019-07-15 05:30:43 UTC
  • Revision ID: kaputtnik-20190715053043-4m31r19ia98uqtts
adjusted robots.txt, see bug 1836505

Show diffs side-by-side

added added

removed removed

Lines of Context:
templates/robots.txt
1
 
# robots.txt for wl.widelands.org
 
1
# robots.txt for www.widelands.org
2
2
 
3
3
# These things should never be crawled
4
4
User-agent: *
5
 
Disallow: /profile
6
 
Disallow: /admin
7
 
Disallow: /accounts
 
5
Disallow: /profile/
 
6
Disallow: /admin/
 
7
Disallow: /accounts/
 
8
 
 
9
Crawl-delay: 10
8
10
 
9
11
# url to sitemap
10
 
Sitemap: https://wl.widelands.org/sitemap.xml/
 
12
Sitemap: https://www.widelands.org/sitemap.xml/