Lesson overview
Teach the text-processing steps that turn documents and queries into comparable terms, including tokenisation, stopword handling, case folding and normalisation.
Starter
Write down the user need, the data being stored or searched, and the decision the system has to support. Then predict which representation, index, query method or governance control will matter most in this lesson.
Learning objectives
- Explain tokenisation and why token boundaries are language and domain dependent.
- Evaluate when stopword removal helps or harms retrieval.
- Apply text normalisation policies before indexing and querying.
Learning outcomes
- Students can explain tokenisation and why token boundaries are language and domain dependent.
- Students can evaluate when stopword removal helps or harms retrieval.
- Students can apply text normalisation policies before indexing and querying.
Key vocabulary, acronyms and terminology
- tokenisation
- Splitting text into retrievable units called tokens.
- stopword
- A very common word often removed or downweighted in retrieval.
- case folding
- Converting case variants into a common form.
- normalisation
- Standardising text forms before comparison.
- token
- A unit such as a word, number, symbol or phrase used by an index.
- phrase query
- A query that preserves adjacency and order.
Detailed teaching notes
Core concept
Tokenisation decides what the index can match. If the system splits text badly, later ranking cannot fully repair the damage.
Representation choice
The phrase 'Newcastle-under-Lyme' might be one token, several tokens or a mixture depending on policy. Numbers, hashtags, email addresses, code identifiers and apostrophes all need domain decisions.
Method and reasoning
Apply the same compatible processing to documents and queries: decode text, normalise Unicode, case-fold where appropriate, tokenise, handle stopwords and preserve positions if phrase retrieval matters.
Risk and limitation
Stopword removal can break meaning. Removing 'to be or not to be' or 'The Who' can make searches worse, so modern systems often downweight common words rather than always deleting them.
Degree-level deep dive
Tokenisation defines the index vocabulary
The system can only retrieve what it has represented. Decisions about hyphens, apostrophes, numbers, code identifiers, hashtags and phrases determine the units that later ranking and phrase search can use.
Stopwords are not always noise
Removing common words can improve efficiency or reduce weak evidence, but it can also destroy meaning in phrases such as 'The Who' or 'to be or not to be'. Modern systems often downweight rather than delete.
What excellent work shows
A strong answer describes document and query processing as compatible pipelines. It should justify tokenisation and normalisation decisions using the collection domain, user tasks and risks to phrase, name or code search.
Concrete example to study
Tokenisation policy comparison
Text: Newcastle-under-Lyme's archive opened in 2026.
Policy A: [newcastle, under, lyme, archive, opened, 2026]
Policy B: [newcastle-under-lyme, archive, opened, 2026]Reasoning
- Policy A may improve broad matching for Newcastle and Lyme separately.
- Policy B preserves the place name as a meaningful unit.
- A strong system might index both forms, while preserving positions for phrase matching.
Reveal takeaway
Tokenisation is a design decision that changes what the index can retrieve, so students should justify token boundaries using the domain and the searches the system must support.
Worked example
Scenario
A search for 'The Who' returns poor results because both words are removed as stopwords.
Worked solution
- Identify that the query is a named entity, not ordinary prose.
- Avoid blindly removing stopwords from short or quoted queries.
- Preserve phrase positions so the exact phrase can be matched.
- Use ranking features that recognise artist names or authority metadata.
Reveal model result
Stopword policy must be sensitive to context. A word that is usually low-value may be essential inside a name or phrase query.
Applied retrieval task
Create a tokenisation policy for a university course search.
Deliverables
- Decide how to handle hyphenated module titles.
- State whether course codes such as CS101 are preserved.
- Explain how stopwords are handled in quoted phrases.
Success checks
- The policy covers both documents and queries.
- The examples match a real course-search domain.
Common misconception
Tokenisation is not a harmless preprocessing detail. It defines the vocabulary of the index and therefore shapes every result.
Quick checks
1. In this lesson, why does tokenisation matter?
2. Which answer best shows degree-level understanding of tokenisation, stopwords and text normalisation?
Digital exam practice
Example exam task
A retrieval system indexes course titles and descriptions, but searches for 'AI for business' and 'The Web' return inconsistent results. In your answer, define the relevant objects or data structures, use course-specific vocabulary, show the method rather than only the result, and finish with a decision about the storage or retrieval system.
Notation and technical toolkit
What a good answer is expected to show
A strong answer for Tokenisation, Stopwords and Text Normalisation the answer should discuss tokenisation, stopwords, case folding and normalisation, and explain how to apply the same policy during indexing and querying. It should connect the formal or technical representation to the user's information need instead of listing terms without using them.
How to solve it
- Identify tokenisation decisions for titles, acronyms and short phrases.
- Explain why stopword removal may harm short named queries.
- Apply case folding and Unicode normalisation consistently.
- Preserve token positions where phrase search is needed.
Model answer
Reveal model answer
The index should use a documented tokenisation policy for both stored course descriptions and user queries. Case folding can make Web and web comparable, while Unicode normalisation prevents hidden representation mismatches. Stopwords should not be blindly removed from short titles such as 'The Web' because the removed word may be part of the name. For phrases, the index should keep token positions so quoted or title-like queries can match adjacent terms.
Practise next
- Tokenise three course titles and justify each boundary.
- Explain when stopword removal improves efficiency but reduces precision.
Self-marking criteria
- Defines the scenario objects, data forms or system components before solving.
- Uses the lesson vocabulary accurately and in context.
- Shows a clear method with enough working for a marker to follow.
- Connects the result back to retrieval, storage, analytics or governance.
- States a limitation, trade-off or quality risk rather than presenting the answer as absolute.
Extension
Compare whitespace tokenisation with a language-aware tokenizer for social media posts.
Study route
Save a short worked answer from this lesson using this pattern: define the need or data, choose the representation, show the method, state the result and interpret the implication for the system.
Next lesson: Keywords, Stemming and Lemmatisation.