A crawler observes fetchable URLs and technical signals under its own settings. Search Console shows Google’s property data. Server logs, rendered tests, analytics, and CMS/configuration provide other evidence. AI may summarize patterns, but it cannot convert a crawl export into Google’s ranking diagnosis.

// concept

Define Audit Scope

Record property/hosts, environment, date, crawler/user agent, start URLs, robots behavior, rendering mode, limits, authentication, included/excluded paths, and permission. Never crawl a site you are not authorized to test or overload production.

Export only necessary fields:

// prompt — copy me4 lines

URL, status, content type, canonical, index directive,
title/H1, internal inlinks/outlinks, depth,
rendered/HTML differences, hreflang, structured data,
size/timing where measured

Remove query parameters or data that may contain personal/session information before AI processing.

// concept

Use AI for Triage, Not Verdict

Ask for clusters under a fixed taxonomy: broken internal link, redirect chain, conflicting canonical, unintended noindex, orphan candidate, duplicate title, thin/unknown, render failure, or needs manual review. Require example URLs and counts from the provided data. Reject invented counts/causes.

Validate high-impact findings manually with HTTP fetch, rendered browser, CMS rules, Search Console URL Inspection, and server configuration.

// worked_example

Worked Example

A 2,000-URL crawl reports 180 canonical mismatches. AI groups them by template, but manual inspection finds 160 are intentional filtered URLs and 20 product pages incorrectly canonicalize to category. The fix targets the product template only.

The audit report separates evidence, interpretation, business impact, confidence, recommended test, owner, and rollback. It does not state that fixing canonicals will recover a specific ranking.

// failure_cases

Failure Cases to Diagnose

6 cases to diagnose

Crawl equals Google index
compare Search Console and logs.
AI counts without deterministic aggregation
compute first.
Every warning becomes ticket
prioritize material, verified issues.
Crawler ignores JavaScript when site needs rendering
compare modes.
Private URLs exported
sanitize scope/data.
Fix deployed sitewide without sample test
stage and rollback.

// pakistan_angle

Pakistan Angle

Audit low-bandwidth mobile paths and server reliability from relevant regions where possible. A page technically available from the office may fail through a local network/provider condition; distinguish anecdote from repeatable monitoring.

Check localized city/language templates for doorway duplication, wrong PKR details, and cross-canonical mistakes. Do not infer user quality from regional traffic.

Retain the raw export, crawler configuration, and transformation steps so another reviewer can reproduce each count. Screenshots alone are weak audit evidence because filters and excluded rows remain invisible.

// hands_on

Hands-On Exercise

5 steps

authorize and configure a bounded crawl.
sanitize/export the schema.
compute issue counts deterministically.
use AI to cluster examples.
manually verify top five findings and create staged tests.

// completion_rubric

Completion Rubric

6 checks — tick as you verify

0/6

// sources

Sources

3 official sources — check every claim yourself

// check_yourself

Check yourself

4 questions · answers and options are taken word-for-word from this course

0/4

1 / 4 · diagnose
Your work shows this failure mode: “Crawl equals Google index.” What does the lesson tell you to do about it?