Why Dev Teams Should Own an SEO Crawler

Published 2026-10-16 · 8 min read · by the isaeo team

The gap in your pipeline

Your pipeline is good at catching things that fail loudly. Tests fail on bad logic. Type checks fail on bad types. Uptime monitors fire on 500s. Error tracking catches exceptions.

Now consider this diff: someone adds <meta name="robots" content="noindex"> to a shared layout, for a legitimate reason on one page type. Every test passes. The page returns 200. It renders identically. Code review approves it, because it looks correct — and in isolation it is.

An entire section leaves the search index, and nothing in your stack has an opinion about it. That's not a monitoring oversight; it's a category your tools were never designed to cover. Nothing in a normal pipeline reads a robots meta tag and compares it to yesterday's.

What your tests can't see

The pattern in every case below: correct-looking code, passing checks, invisible consequence.

• A noindex in a shared template. Correct for the page it was added for. Catastrophic for the forty page types sharing the layout.
• A staging robots.txt in the deploy. Config copied along with everything else. Disallow: / is the right answer on staging.
• An H1 that became a styled div. Visually identical, so design review passes. Structurally, the page now has no heading.
• A canonical built from the wrong variable. Points at staging, or at a parent, or at the homepage. Renders nothing visible.
• Content moved behind hydration. A refactor makes copy client-only. Humans see a full page; crawlers get a shell. See JavaScript SEO and rendering.
• A nav change orphaning a section. The pages still work. Nothing links to them, so they fade out of the index over weeks.

Every one of these is a code change with a delayed, invisible cost — the exact profile of bug that automated checks exist for.

What to assert on

Keep the assertion set small and high-signal. A crawl produces hundreds of findings; a check that fails on any of them will be disabled within a fortnight. Fail the build — or fire the alert — on these only:

Note what's absent: meta description length, alt text, word count, title optimisation. Those are content quality, they change constantly for legitimate reasons, and blocking a deploy on them trains everyone to bypass the check. Put them in a report someone reads monthly.

The distinction worth internalising: assert on can engines see this page, report on is this page good. The first is binary and objective; the second needs judgement.

Where it goes in the pipeline

Three viable placements, and the middle one is where most teams should start.

Post-deploy crawl (start here). Deploy to production, crawl, compare against the previous crawl, alert on the assertions above. Doesn't block anything, needs no CI plumbing, catches everything within minutes of the change. The realistic answer for most teams — and it's the same mechanism as change tracking, just triggered by a deploy instead of a schedule.

Staging crawl in CI. Better, because it catches problems before users see them. Requires a staging environment that's crawlable — reachable, with real content, ideally behind HTTP auth your crawler can use. Fail the build on the narrow assertion set only.

Scheduled crawl, independent of deploys. Still necessary even with the above, because not everything that changes your pages is a deploy: a CMS edit, an expired certificate, a CDN rule, a third-party script. See how often to crawl.

Most teams want the first and third. The second is worth adding when SEO regressions have already cost you something.

Who should own it

The failure mode here is organisational rather than technical, and it's worth naming plainly: the marketing team owns SEO outcomes, the engineering team ships the regressions, and the alert goes to whoever configured the tool.

What works:

• The indexability alert goes to engineering. Same channel as your other production alerts, not a marketing inbox. It's a deploy regression and it needs the person who can revert.
• Content-quality reporting goes to marketing. Monthly, not as an alert.
• One named owner for the assertion list. Someone decides what's worth failing on, or the list rots into noise and gets muted.
• Treat a fired alert like any other incident. Fix, then ask why no check caught it earlier. Usually the answer is a missing assertion, and that's a cheap fix.

The cultural shift that makes this stick: a page silently dropping out of the index is a bug, not a marketing problem. Once a team accepts that framing, the tooling question answers itself — and the cost of not doing it is the argument for the framing.