robots.cr

Simple robots.txt parsing for Crystal

Installation

1. Add the dependency to your shard.yml:

   dependencies:
     robots:
       github: grkek/robots
   

2. Run shards install

Usage

require "robots"

# From reddit
robotstxt = <<-TEXT
# 80legs
User-agent: 008
Disallow: /

User-Agent: *
Disallow: /goto
Disallow: /*after=
Disallow: /*before=
Disallow: /domain/*t=
Disallow: /login
Disallow: /reddits/search
Disallow: /search
Disallow: /r/*/search
Allow: /
TEXT

reddit = Robots.new(robotstxt, "Googlebot")

reddit.allowed?("/login")
# => false

reddit.allowed?("/r/SushiChain")
# => true

reddit.allowed?("/r/SushiChain/search")
# => false

Contributing

1. Fork it (https://github.com/grkek/robots/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Contributors

• Chris Watson - creator and maintainer
• Giorgi Kavrelishvili - maintainer