Information Storage and Retrieval: Web Crawlers, Spiders and Collection Building

Lesson overview

Explain how crawlers and spiders discover web content, manage frontiers, respect robots.txt and politeness, canonicalise URLs and build searchable collections.

CourseInformation Storage and Retrieval

Topic strandWeb acquisition

Assessment styleWorked scenario, applied task and digital exam practice

EvidenceDefinitions, representation, method, result and interpretation

Starter

Write down the user need, the data being stored or searched, and the decision the system has to support. Then predict which representation, index, query method or governance control will matter most in this lesson.

Learning objectives

Describe how a crawler discovers and fetches pages using links and sitemaps.
Explain crawler frontier, politeness and robots.txt constraints.
Identify duplicate and canonical URL problems in collection building.

Learning outcomes

Students can describe how a crawler discovers and fetches pages using links and sitemaps.
Students can explain crawler frontier, politeness and robots.txt constraints.
Students can identify duplicate and canonical URL problems in collection building.

Key vocabulary, acronyms and terminology

spider: Another term for a web crawler.
frontier: The queue or schedule of URLs waiting to be crawled.
URL: Uniform Resource Locator: the address used to locate and fetch a web resource.
robots.txt: A site file that declares crawler access rules.
politeness: Crawler behaviour that avoids overloading sites.
canonical URL: The preferred Uniform Resource Locator representing a page when duplicates exist.
sitemap: A file listing URLs a site wants crawlers to discover.

Detailed teaching notes

Core concept

A crawler builds a collection by repeatedly selecting URLs, fetching pages, extracting links and scheduling new URLs. The hard part is not fetching one page; it is doing this ethically, efficiently and without filling the index with duplicates.

Representation choice

The crawler frontier stores pending URLs with priorities, depths, host rules and timestamps. Fetched pages produce content for indexing and new links for the frontier.

Method and reasoning

Start from seed URLs and sitemaps, check robots.txt, normalise and canonicalise URLs, fetch politely, extract links and record crawl status. Revisit schedules should reflect update frequency and importance.

Risk and limitation

A crawler that ignores politeness or canonical URLs can harm sites and damage its own collection. It may fetch infinite calendar pages, duplicate tracking URLs or pages explicitly disallowed by robots.txt.

Degree-level deep dive

Crawling is governed collection building

A crawler is not simply a downloader. It chooses a frontier, respects robots.txt, applies politeness, canonicalises URLs, avoids duplicates and decides when content should be revisited.

URL variants can damage evidence

Tracking parameters, print pages and calendar links can create many addresses for equivalent or low-value content. Canonical URL handling prevents ranking signals and analytics from being split across duplicates.

What excellent work shows

A strong answer includes seed selection, frontier policy, robots.txt checks, politeness delays, URL normalisation, duplicate detection and revisit rules. Ethical and technical collection quality should both be visible.

Concrete example to study

Duplicate URL set

https://example.ac.uk/article?id=42
https://example.ac.uk/article?id=42&utm_source=newsletter
https://example.ac.uk/print/article?id=42

Reasoning

All three URLs may show the same article content.
The crawler should remove tracking parameters where safe and identify the canonical URL.
Only the canonical version should receive the main index record, with duplicates recorded or folded into it.

Reveal takeaway

Collection quality depends on controlling duplicates before they pollute ranking, analytics and result diversity.

Worked example

Scenario

A crawler keeps indexing the same product page under URLs with tracking parameters.

Worked solution

Normalise URLs by removing irrelevant tracking parameters where policy allows.
Use canonical link metadata if the site provides it.
Detect near-duplicate content during ingestion.
Store the canonical URL in the index so ranking and analytics are not split across duplicates.

Reveal model result

Collection building must control duplicates before ranking. Otherwise the index becomes noisy and popularity signals are divided across equivalent URLs.

\text{Define the data, choose the representation, apply the method, interpret the result.}

Applied retrieval task

Write a crawl policy for a public course website.

Deliverables

Choose seed URLs and identify whether a sitemap exists.
State robots.txt and politeness checks.
Describe how duplicate URLs will be canonicalised.

Success checks

The policy respects external-site load.
The duplicate-handling rule is explicit.

Common misconception

More crawling is not automatically better. A good collection is selective, respectful and deduplicated.

Quick checks

1. In this lesson, why does spider matter?

It is only a presentational label and has no effect on retrieval quality.It replaces the need to understand user tasks or data representation.Another term for a web crawler.

2. Which answer best shows degree-level understanding of web crawlers, spiders and collection building?

Define frontier, apply it to the scenario, then explain the trade-off or risk.Give a final recommendation without showing how the data or query was represented.Name as many technologies as possible without explaining the scenario.

Digital exam practice

Example exam task

A small search engine is crawling an online archive but creates thousands of duplicate pages because each article appears with tracking parameters and print-view URLs. In your answer, define the relevant objects or data structures, use course-specific vocabulary, show the method rather than only the result, and finish with a decision about the storage or retrieval system.

Notation and technical toolkit

frontier = {u_1,u_2,...}set or queue of candidate URLs Use to reason about crawler scheduling.

robots.txtcrawler access policy Use before fetching pages from a host.

canonical(u)preferred representation of a URL Use to avoid duplicate indexing.

What a good answer is expected to show

A strong answer for Web Crawlers, Spiders and Collection Building the answer should describe crawler frontier management, robots.txt, politeness, canonical URL handling and duplicate control. It should connect the formal or technical representation to the user's information need instead of listing terms without using them.

How to solve it

Explain how the spider discovers and schedules URLs in the frontier.
Check robots.txt and apply politeness delays for each host.
Normalise URLs and identify canonical versions.
Prevent duplicate pages from polluting the search index.

Model answer

Reveal model answer

The spider should maintain a frontier of URLs discovered from seed pages and the archive sitemap. Before fetching, it must respect robots.txt and use politeness delays so the archive is not overloaded. The crawler should normalise tracking parameters, recognise print-view duplicates and prefer the canonical URL declared by the page or chosen by policy. Only the canonical version should receive the main index entry, while duplicates can redirect signals to it.

Practise next

Explain why a crawler should not simply fetch every URL as fast as possible.
Give two URL variants that might point to the same underlying document.

Self-marking criteria

Defines the scenario objects, data forms or system components before solving.
Uses the lesson vocabulary accurately and in context.
Shows a clear method with enough working for a marker to follow.
Connects the result back to retrieval, storage, analytics or governance.
States a limitation, trade-off or quality risk rather than presenting the answer as absolute.

Extension

Design a revisit policy for news pages, archived pages and course timetable pages.

Study route

Save a short worked answer from this lesson using this pattern: define the need or data, choose the representation, show the method, state the result and interpret the implication for the system.

Next lesson: Tokenisation, Stopwords and Text Normalisation.