This library helps write robots.txt compliant web robots in Ruby.

Homepage:
https://rubygems.org/gems/webrobots/
