[GRASS-dev] [GRASS GIS] #3983: robots.txt unmaintained and lacks sitemaps

#3983: robots.txt unmaintained and lacks sitemaps
-------------------------+-------------------------
Reporter: jidanni | Owner: grass-dev@…
     Type: defect | Status: new
Priority: trivial | Milestone:
Component: Website | Version: unspecified
Keywords: | CPU: Unspecified
Platform: Unspecified |
-------------------------+-------------------------
Google is returning older versions of man pages instead of the current
(v78) ones.

This is due to robots.txt not having been updated since 2015,

{{{
$ HEAD https://grass.osgeo.org/robots.txt
Last-Modified: Tue, 20 Jan 2015 11:14:30 GMT

$ GET https://grass.osgeo.org/robots.txt
Disallow: /grass61/manuals/
Disallow: /grass62/manuals/
Disallow: /grass63/manuals/
Disallow: /grass65/manuals/
}}}

And most importantly, the lack of sitemap entries.
  https://support.google.com/webmasters/answer/156184

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3983&gt;
GRASS GIS <https://grass.osgeo.org>

#3983: robots.txt unmaintained and lacks sitemaps
--------------------------+-------------------------
  Reporter: jidanni | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: trivial | Milestone:
Component: Website | Version: unspecified
Resolution: | Keywords:
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by neteler):

I have added more entries to the existing robots.txt:

{{{
...
Disallow: /grass64/manuals/
Disallow: /grass65/manuals/
Disallow: /grass70/manuals/
Disallow: /grass71/manuals/
Disallow: /grass72/manuals/
Disallow: /grass73/manuals/
Disallow: /grass74/manuals/
Disallow: /grass75/manuals/
Disallow: /grass77/manuals/
}}}

Re: sitemap.xml: We are waiting for the initial draft of the new Web page
(see https://grasswiki.osgeo.org/wiki/Website_migration_plan_2019). With
the current oldish CMS is appears to be a "mission impossible"...

I hope that the SEO will then be way better than now.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3983#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>