Most small sites need clean discovery and intentional indexation more than “crawl budget optimization.” Google says crawl-budget management is mainly relevant to very large or rapidly changing sites. Start by deciding which URLs should exist and be indexed.

// concept

Build the Indexation Matrix

For each template/path:

// prompt — copy me10 lines

purpose
should exist?
should be crawlable?
should be indexable?
canonical target
internal-link source
sitemap inclusion
expected status
Search Console evidence
owner

Common controls differ:

robots.txt controls crawling, not guaranteed removal from index;
noindex requires Google to crawl and see it;
canonical is a signal for duplicate consolidation, not a deletion mechanism;
301/308 redirects map moved equivalents;
404/410 removes unavailable non-equivalent content;
sitemap lists preferred canonical indexable URLs but does not guarantee indexing.

// concept

Reduce URL Waste

Fix faceted/filter parameters, calendars, search results, session IDs, duplicate print/tracking URLs, infinite spaces, broken pagination, soft 404s, and generated thin pages at the application/CMS source. Avoid blocking before Google can see a needed noindex during a transition.

Use Page Indexing report and URL Inspection for owned evidence. Remember Search Console reports have limits/sampling depending on report.

// worked_example

Worked Example

An ecommerce site creates crawlable URLs for every sort/filter combination. The team identifies which category filters have distinct user value, gives those stable canonical pages, and prevents/controls meaningless combinations through application links, parameters, and indexing directives.

Sitemap contains only canonical sellable categories/products. Discontinued product URLs with useful replacements redirect only when genuinely equivalent; others return proper unavailable status. The team monitors indexing groups rather than repeatedly requesting indexing.

// failure_cases

Failure Cases to Diagnose

6 cases to diagnose

Robots used to deindex quickly
choose correct removal/noindex path.
All excluded URLs called errors
many exclusions are intentional.
Canonical points to irrelevant hub
map true duplicates only.
Every 404 redirects home
creates confusing soft-404 behavior.
Sitemap includes redirects/noindex
list canonical indexable URLs.
“Crawl budget” project for 100 pages
fix architecture/content first.

// pakistan_angle

Pakistan Angle

Marketplace and directory sites may generate city/category/filter combinations quickly. Publish combinations only when inventory, distinct task, and maintainable content exist. Empty “best X in city” pages create poor user value.

For bilingual URLs, use deliberate language URLs/hreflang and consistent canonicals. Do not canonicalize genuine Urdu content to English merely to simplify reporting.

// hands_on

Hands-On Exercise

5 steps

Create the path/template matrix.
sample Search Console indexing groups.
find one URL explosion source.
select correct status/canonical/noindex/robots/sitemap changes.
stage, inspect, and monitor.

// completion_rubric

Completion Rubric

6 checks — tick as you verify

0/6

// sources

Sources

4 official sources — check every claim yourself

// check_yourself

Check yourself

4 questions · answers and options are taken word-for-word from this course

0/4

1 / 4 · diagnose
Your work shows this failure mode: “Robots used to deindex quickly.” What does the lesson tell you to do about it?