Perhaps an "allow from" entry for that file would be better, set to only
allow from the machine that triggers the cron job?
robots.txt is only advisory
Jason
-----Original Message-----
From: sac-bounces@lists.osgeo.org [mailto:sac-bounces@lists.osgeo.org]
On Behalf Of Tyler Mitchell
Sent: Wednesday, January 31, 2007 13:05
To: System Administration Committee Discussion/OSGeo; Web committee
discussions
Subject: [SAC] Added robots.txt
A question about spiders grabbing onto our cron job link came up on IRC.
Is it preferred to add a robots.txt to disallow this?
I have put one up at:
I've added the /var/www/html/robots.txt
Make sense to keep it?
Tyler
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac