link_followability

A small Crystal shard that reads links the way search engines do: every rel token on every <a>, plus the page-level <meta name="robots"> and the X-Robots-Tag response header.

• nofollow, ugc and sponsored are reported as not-followed hints; an anchor can carry more than one of them.
• noopener and noreferrer are browser-security tokens and do not change whether a link is followed.
• none in a robots directive means noindex, nofollow.
• A crawler-scoped header such as X-Robots-Tag: googlebot: noindex applies only to that crawler.

Installation

Add the dependency to your shard.yml:

dependencies:
  link_followability:
    github: theluckystrike/link-followability

Then run shards install.

Usage

require "link_followability"

html = %(<meta name="robots" content="index, follow">
<a href="https://example.com/a" rel="noopener noreferrer">A</a>
<a href="https://example.com/b" rel="ugc nofollow">B</a>)

page = LinkFollowability.merge(
  LinkFollowability.parse_directives(LinkFollowability.meta_robots(html)),
  LinkFollowability.parse_x_robots_tag(nil),
)

LinkFollowability.anchors(html).each do |a|
  puts "#{a.href} followed=#{LinkFollowability.followed?(a, page)} hints=#{a.hints}"
end
# https://example.com/a followed=true hints=[]
# https://example.com/b followed=false hints=["ugc", "nofollow"]

API

The anchor scanner tokenizes start tags; it is not a full HTML5 parser, so feed it the served HTML, not a DOM produced by client-side JavaScript.

Related

For a no-install version, the paste-your-HTML rel token checker on handsofflinks.com prints the same per-anchor rel tokens in the browser.

Development

crystal spec

License

MIT