Information Storage and Retrieval: Keywords, Stemming and Lemmatisation

Lesson overview

Explain how keyword selection, stemming and lemmatisation affect recall and precision, including common stemming errors and domain vocabulary issues.

CourseInformation Storage and Retrieval

Topic strandText processing

Assessment styleWorked scenario, applied task and digital exam practice

EvidenceDefinitions, representation, method, result and interpretation

Starter

Write down the user need, the data being stored or searched, and the decision the system has to support. Then predict which representation, index, query method or governance control will matter most in this lesson.

Learning objectives

Explain how keywords represent the important searchable terms in a document.
Compare stemming with lemmatisation.
Evaluate over-stemming, under-stemming and domain-specific vocabulary risks.

Learning outcomes

Students can explain how keywords represent the important searchable terms in a document.
Students can compare stemming with lemmatisation.
Students can evaluate over-stemming, under-stemming and domain-specific vocabulary risks.

Key vocabulary, acronyms and terminology

keyword: A significant term used to represent content or a user need.
stemming: Heuristic reduction of word forms to a stem.
Porter stemmer: A well-known English stemming algorithm.
lemmatisation: Reduction of inflected forms to dictionary lemmas using vocabulary knowledge.
over-stemming: Merging terms that should remain distinct.
under-stemming: Failing to merge terms that users expect to match.

Detailed teaching notes

Core concept

Stemming and lemmatisation try to connect related word forms so users do not need to guess exact endings. The trade-off is between broader recall and accidental merging.

Representation choice

A Porter stemmer may reduce connected, connecting and connection to related stems, while lemmatisation aims for dictionary forms such as run from running. Domain terms and names may need exceptions.

Method and reasoning

Decide whether the collection benefits from morphological matching. Use stemming for simple speed and recall, lemmatisation for more linguistically informed matching, and controlled vocabulary for specialised domains.

Risk and limitation

Over-stemming can merge policy and police-like forms in some algorithms or domains; under-stemming can leave analyse and analysis apart when users expect them together.

Degree-level deep dive

Morphology affects recall and precision

Keyword retrieval improves when related word forms can match, but aggressive conflation can merge different meanings. Stemming and lemmatisation are therefore retrieval trade-offs, not automatic improvements.

Algorithm choice matters

The Porter stemmer applies rule-based suffix stripping; lemmatisation uses vocabulary and grammatical analysis to return dictionary forms. The second can be more precise but usually needs language resources.

What excellent work shows

A strong answer gives examples of under-stemming and over-stemming, then chooses a method for the domain. Legal, medical and code collections often need more cautious term handling than general web search.

Concrete example to study

Stem versus lemma

Words: connect, connected, connecting, connection
Possible stem: connect
Possible lemma examples: connect, connection

Reasoning

A stemmer may conflate related surface forms quickly.
A lemmatiser tries to return dictionary forms using vocabulary and grammar.
Over-stemming can merge terms that users expect to stay distinct; under-stemming can miss useful matches.

Reveal takeaway

The method should be chosen from the domain and the retrieval risk, not because one algorithm is always superior.

Worked example

Scenario

A search for 'compute' should also find pages about computing and computation.

Worked solution

Identify related morphological forms in the domain vocabulary.
Apply stemming or lemmatisation to connect some forms.
Check whether the merged terms still share meaning in this collection.
Evaluate whether recall improves without damaging precision too much.

Reveal model result

The correct choice depends on the collection. Stemming may be enough for broad web search, but a technical archive may need careful vocabulary control.

\text{Define the data, choose the representation, apply the method, interpret the result.}

Applied retrieval task

Build a keyword policy for a research repository.

Deliverables

List five keywords for one abstract.
Decide which terms should be stemmed or lemmatised.
Identify one term that should not be merged because it is a proper name or distinct concept.

Success checks

The policy explains precision and recall trade-offs.
The examples come from the same domain.

Common misconception

Stemming is not the same as understanding meaning. It is a surface-form technique that can help matching but can also create false matches.

Quick checks

1. In this lesson, why does keyword matter?

It replaces the need to understand user tasks or data representation.A significant term used to represent content or a user need.It is only a presentational label and has no effect on retrieval quality.

2. Which answer best shows degree-level understanding of keywords, stemming and lemmatisation?

Give a final recommendation without showing how the data or query was represented.Name as many technologies as possible without explaining the scenario.Define stemming, apply it to the scenario, then explain the trade-off or risk.

Digital exam practice

Example exam task

A repository search for 'analysing retrieval systems' does not find papers tagged 'analysis of information retrieval'. In your answer, define the relevant objects or data structures, use course-specific vocabulary, show the method rather than only the result, and finish with a decision about the storage or retrieval system.

Notation and technical toolkit

term -> stemstemming transformation Use to show heuristic term reduction.

word -> lemmalemmatisation transformation Use when dictionary form matters.

recall / precisioncoverage versus exactness Use to judge whether expansion has helped.

What a good answer is expected to show

A strong answer for Keywords, Stemming and Lemmatisation the answer should compare keyword matching, the Porter stemmer, lemmatisation and the precision-recall risks of merging word forms. It should connect the formal or technical representation to the user's information need instead of listing terms without using them.

How to solve it

Identify the mismatched word forms and keywords.
Explain how stemming might reduce forms to comparable stems.
Explain how lemmatisation differs from heuristic stemming.
Evaluate over-stemming and under-stemming risks.

Model answer

Reveal model answer

The system is missing related word forms: analysing and analysis, retrieval and possibly retrieve. A Porter stemmer could reduce some forms heuristically, increasing recall, while lemmatisation would use vocabulary knowledge to map inflected forms to lemmas. I would test the effect on this repository because over-stemming can merge unrelated terms and reduce precision. The answer should connect the processing choice to measurable retrieval behaviour rather than simply saying stemming is always good.

Practise next

Give an example of over-stemming in a domain you know.
Explain why lemmatisation can be more accurate but more expensive than stemming.

Self-marking criteria

Defines the scenario objects, data forms or system components before solving.
Uses the lesson vocabulary accurately and in context.
Shows a clear method with enough working for a marker to follow.
Connects the result back to retrieval, storage, analytics or governance.
States a limitation, trade-off or quality risk rather than presenting the answer as absolute.

Extension

Compare a stemmed search and an exact search on a small set of article titles.

Study route

Save a short worked answer from this lesson using this pattern: define the need or data, choose the representation, show the method, state the result and interpret the implication for the system.

Next lesson: Inverted Indexes and Posting Lists.