Rate limiting is a restriction on the number of requests per unit of time during automated data collection. It has two reasons. The first is practical – aggressive scraping is immediately detected and blocked, so a slower approach is, paradoxically, faster to reach the goal. The second is ethical and legal, since thousands of requests per minute can slow down or crash someone else's server, which is already damage, not data collection. Decent practice includes respecting the rules in the robots.txt file, identifying the bot with contact details for the operator, collecting data outside peak hours, downloading only the pages needed instead of the whole website, and storing and reusing the results instead of downloading them repeatedly. If the operator offers an interface or a data export, use it in preference.
See also: Legal aspects of scraping, AI Crawler, Data audit.
