bots crawl calendar excessively

Project:BookRoom
Component:Miscellaneous
Category:task
Priority:normal
Assigned:Unassigned
Status:closed
Description

According to the logs, as discovered by Hoover, it appears that bots are crawling Bookroom (and Mediavision which uses the same sort of calendar) using the left and write arrows to move 1 month up from 'today'.
Dave discovered a robots.txt file in the directory...but was too low to be activated. He has since moved the robots.txt file (around Feb 14, 2010) up so it should start having some effects of blocking honest bots.
If a software solution was needed an approach ah might be to not generate a left arrow to allow indexing prior to 2005 (the start of Bookroom) or after about 3 years from 'today.'
the next step would be to block viewing the dark past or future pages if the url is manually adjusted.

Comments

#1

Status:active» closed

Back to top