Link checker: HEAD-only requests cause false positives on some sites (e.g. crates.io)

[marc0olo]

Problem

The link checker in links/check.go uses HEAD requests exclusively. Some sites don't handle HEAD correctly and return 404 or 405 even though the page exists:

• crates.io: https://crates.io/crates/ic-vetkeys returns 404 for HEAD (SPA that requires Accept: text/html), but 200 for GET. The crate exists (v0.6.0, 93k downloads).
• This is a known issue across link checkers — tools like lychee and markdown-link-check fall back to GET when HEAD fails.

Reproduction

# HEAD returns 404
curl -sI "https://crates.io/crates/ic-vetkeys" | head -1
# HTTP/2 404

# GET returns 200
curl -s -o /dev/null -w "%{http_code}" "https://crates.io/crates/ic-vetkeys"
# 200

Question

Is the HEAD-only approach intentional (e.g. to avoid bandwidth/rate-limiting concerns), or would a GET fallback when HEAD returns 404/405 be welcome? Happy to submit a PR if the fallback approach seems reasonable.

Context

Discovered while running skill-validator check on dfinity/icskills. The false positive blocks CI deployment.

[dacharyc]

Hey @marc0olo - thanks for the issue. Not an intentional design decision - just something I hadn't encountered yet. Just tagged v1.5.2 with this fix, and tested it on your repo's vetkd skill which had the reference example. The link checker is passing now.

I did notice that your skill is still failing structural validation, but now it's due to the token/line limits for SKILL.md that are encoded in skill-validator. I based the limits on the spec's Progressive Disclosure recommendations. I think it's probably in the best interest of agent performance to keep those limits, so I'd recommend pruning if possible - moving conditional logic out to reference files or otherwise restructuring to be more token efficient. But if you're unable to do that, I'm open to configurable flags for those limits. I've just been avoiding it until someone actually needs it in the interest of trying to maintain the optimal agent experience.

[marc0olo]

Hey @dacharyc, thanks for having a look and fixing this.

I am aware that not all skills are in best shape possible yet, we are still focusing on the foundational things right now before improving individual skills.

Currently the only skill in our repo using reference files is the icp-cli skill. And there is an open PR for a canhelp skill which includes scripts.

Having the link-checker fixed is sufficient for us, thanks! 👍