#139: Apache - robots
---------------------+-----------------------
Reporter: sbarnes | Owner: sbarnes
Type: task | Status: assigned
Priority: normal | Milestone:
Component: General | Resolution:
Keywords: |
---------------------+-----------------------
Comment (by TemptorSent):
This ticket has been cold for 10 years.
It is not clear whether proper handling of spiderbots was ever actually
resolved properly.
It is unacceptable to limit the total number of connections to less than
needed for actual web users, and it is undesirable to completely stop
indexing of discoverable data.
Proper use of [http://www.robotstxt.org/robotstxt.html robots.txt] and
possibly connection/rate limiting for spiders is a more appropriate
solution.
The current configuration should be reviewed, corrected as necessary, and
documented on the SAC wiki before closing this ticket.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/139#comment:5>
OSGeo <http://www.osgeo.org/>
OSGeo committee and general foundation issue tracker.