A question about spiders grabbing onto our cron job link came up on IRC. Is it preferred to add a robots.txt to disallow this?
I have put one up at:
I've added the /var/www/html/robots.txt
Make sense to keep it?
Tyler
A question about spiders grabbing onto our cron job link came up on IRC. Is it preferred to add a robots.txt to disallow this?
I have put one up at:
I've added the /var/www/html/robots.txt
Make sense to keep it?
Tyler