linkfollow ~main

linkfollow

Decide whether a link on a served page is actually followed.

Auditing a backlink comes down to one question asked of the response you just fetched: given this anchor's rel, this page's <meta name="robots"> and this response's X-Robots-Tag header, does the link pass? The three inputs interact, and getting the precedence wrong is how an audit ends up recording a nofollow link as a win.

This is the check behind handsofflinks, which re-fetches every page it publishes and reads the attribute off the served HTML before a row is allowed to say LIVE.

The rules it implements

• rel is a space-separated set of case-insensitive link types. Any of nofollow, ugc or sponsored marks that individual link as not passing. noopener and noreferrer are window and referrer hints — they do not.
• A page-level nofollow, from meta robots or X-Robots-Tag, applies to every link on the page and overrides a clean rel.
• none is shorthand for noindex, nofollow. all is the default and asserts nothing.
• X-Robots-Tag values may carry a user-agent prefix such as googlebot: nofollow. A directive addressed to a named agent applies only when it names the agent you are evaluating.
• noindex on its own does not stop the link from being followed, so followability and indexability are reported separately.

Usage

import linkfollow;

auto v = resolveTag(
    `<a href="https://example.com/" rel="noopener">example.com</a>`,
    metaRobotsContent,   // "" when the page has no meta robots
    xRobotsTagHeader);   // "" when the response has no such header

if (v.followed) { /* record it */ }
else            { /* v.reason says which of the five things blocked it */ }

Or drive it from parts you already extracted:

AnchorContext ctx;
ctx.rel = "ugc";
assert(resolve(ctx) == Verdict(false, Reason.relUgc));
assert(pageIndexable(ctx));

API

The attribute reader handles double-quoted, single-quoted and unquoted values, matches names case-insensitively, and will not match xrel= when you asked for rel — the substring bugs that make a hand-rolled audit lie to you.

Reason distinguishes relNofollow, relUgc, relSponsored, metaRobotsNofollow and headerNofollow, so a ledger can record why a link did not count rather than just that it did not.

Tests

dub test --compiler=ldc2

Eleven unittest blocks, including precedence (header beats meta beats rel), substring safety (nofollowish is not nofollow), user-agent targeting, and the quoting variants of the attribute reader.

License

MIT.