Lesson overview
Introduce retrieval models that decide which items match a query and how strongly, including Boolean retrieval, vector space thinking, TF-IDF and BM25.
Starter
Write down the user need, the data being stored or searched, and the decision the system has to support. Then predict which representation, index, query method or governance control will matter most in this lesson.
Learning objectives
- Explain relevance as a relationship between a result and an information need.
- Compare Boolean retrieval with ranked retrieval.
- Describe how TF-IDF and BM25 reward informative terms.
Learning outcomes
- Students can explain relevance as a relationship between a result and an information need.
- Students can compare Boolean retrieval with ranked retrieval.
- Students can describe how TF-IDF and BM25 reward informative terms.
Key vocabulary, acronyms and terminology
- Boolean retrieval
- Retrieval using logical conditions such as AND, OR and NOT.
- ranked retrieval
- Retrieval that orders results by estimated usefulness.
- TF-IDF
- A weighting idea using term frequency and inverse document frequency.
- BM25
- A common probabilistic ranking function used in search systems.
- term frequency
- How often a term occurs in a document.
- document frequency
- How many documents contain a term.
Detailed teaching notes
Core concept
A retrieval model defines what it means for a document to match a query. Boolean retrieval gives precise inclusion rules; ranked models estimate degrees of relevance so the best candidates appear first.
Representation choice
Text can be represented as term sets, weighted vectors or probabilistic evidence. TF-IDF and BM25 both treat rare informative terms differently from common terms, although BM25 also controls saturation and document length effects.
Method and reasoning
Choose Boolean retrieval when the user needs transparent constraints, such as legal or database filtering. Choose ranked retrieval when many partial matches are possible and result ordering matters.
Risk and limitation
Ranking scores are not truth. They are model outputs shaped by tokenisation, term weighting, field boosts, freshness, popularity and evaluation data.
Degree-level deep dive
Ranking is an estimate
Boolean retrieval decides whether a document satisfies conditions. Ranked retrieval estimates usefulness, often using term evidence such as term frequency, document frequency, field weighting and document length.
TF-IDF and BM25 need interpretation
TF-IDF gives more weight to terms that are frequent in a document but rare across the collection. BM25 refines this with saturation and length normalisation, so repeated terms help only up to a point.
What excellent work shows
A strong answer explains why exact matching may not be enough, names the evidence used for ranking, and states a limitation. Ranking evidence is not the same as relevance itself, because user intent and context still matter.
Concrete example to study
Tiny ranking evidence
Query: climate policy
D1: 'climate policy climate law'
D2: 'policy archive minutes'
D3: 'gardening climate tips'Reasoning
- D1 has both query terms and repeats climate, so lexical evidence is strong.
- D2 has policy but lacks climate, so it may be partly relevant or off-topic.
- D3 has climate but not policy, so ranking should not treat it as equal to D1.
Reveal takeaway
Term evidence helps ranking, but final relevance still depends on whether the user wanted law, archives, science or everyday advice.
Worked example
Scenario
A collection contains many pages with the word 'the' but fewer pages with 'crawler'.
Worked solution
- Treat 'the' as low information because it appears in many documents.
- Treat 'crawler' as more informative because its document frequency is lower.
- Combine term evidence into a score for each candidate document.
- Rank documents by score, then show snippets so users can judge relevance.
Reveal model result
TF-IDF and BM25 help prevent common terms from dominating the ranking. They do not understand meaning by themselves, but they provide useful statistical evidence.
Applied retrieval task
Compare Boolean and ranked retrieval for a help-desk knowledge base.
Deliverables
- Write one Boolean query with AND, OR and NOT.
- Write one ranked-search scenario where partial matches are useful.
- Explain one situation where the highest score might still be irrelevant.
Success checks
- The comparison states user need, not only syntax.
- The limitation of ranking is explicit.
Common misconception
A high retrieval score is not the same as truth or quality. It is the model's estimate under chosen representation and weighting assumptions.
Quick checks
1. In this lesson, why does Boolean retrieval matter?
2. Which answer best shows degree-level understanding of relevance, ranking and retrieval models?
Digital exam practice
Example exam task
A search system ranks a short page about crawlers above a long general page about web history for the query 'web crawler indexing'. In your answer, define the relevant objects or data structures, use course-specific vocabulary, show the method rather than only the result, and finish with a decision about the storage or retrieval system.
Notation and technical toolkit
What a good answer is expected to show
A strong answer for Relevance, Ranking and Retrieval Models the answer should explain why informative terms affect ranking, compare Boolean and ranked retrieval, and avoid claiming that a score proves relevance absolutely. It should connect the formal or technical representation to the user's information need instead of listing terms without using them.
How to solve it
- Define relevance in relation to the user's information need.
- Explain how term frequency and document frequency affect weighting.
- Contrast exact Boolean matching with ranked partial evidence.
- State why the final user-facing result still needs snippets and evaluation.
Model answer
Reveal model answer
The short crawler page may rank higher because it contains several query terms that are informative in the collection. In a TF-IDF-style model, a term such as crawler can carry more weight than a common term, while BM25 also limits the effect of repeated terms and adjusts for document length. Boolean retrieval would only decide whether documents satisfy stated conditions, but ranked retrieval orders candidates by estimated usefulness. The score is evidence, not a guarantee of relevance.
Practise next
- Explain why a stopword usually has low ranking value.
- Create a two-document example where a rarer term changes the ranking.
Self-marking criteria
- Defines the scenario objects, data forms or system components before solving.
- Uses the lesson vocabulary accurately and in context.
- Shows a clear method with enough working for a marker to follow.
- Connects the result back to retrieval, storage, analytics or governance.
- States a limitation, trade-off or quality risk rather than presenting the answer as absolute.
Extension
Research one BM25 parameter and explain how it changes ranking behaviour.
Study route
Save a short worked answer from this lesson using this pattern: define the need or data, choose the representation, show the method, state the result and interpret the implication for the system.
Next lesson: Generic Information Retrieval Architecture.