Information Storage and Retrieval: Precision, Recall, F-Measure and Evaluation

Lesson overview

Teach how retrieval systems are evaluated using relevance judgements, precision, recall, F1 and confusion-matrix thinking, while recognising limits of test collections.

CourseInformation Storage and Retrieval

Topic strandEvaluation

Assessment styleWorked scenario, applied task and digital exam practice

EvidenceDefinitions, representation, method, result and interpretation

Starter

Write down the user need, the data being stored or searched, and the decision the system has to support. Then predict which representation, index, query method or governance control will matter most in this lesson.

Learning objectives

Calculate precision, recall and F1 for a small retrieval result set.
Use a confusion matrix to distinguish relevant and retrieved items.
Explain why evaluation depends on relevance judgements and user tasks.

Learning outcomes

Students can calculate precision, recall and F1 for a small retrieval result set.
Students can use a confusion matrix to distinguish relevant and retrieved items.
Students can explain why evaluation depends on relevance judgements and user tasks.

Key vocabulary, acronyms and terminology

precision: The proportion of retrieved items that are relevant.
recall: The proportion of relevant items that were retrieved.
F1: The harmonic mean of precision and recall.
confusion matrix: A table of true positives, false positives, false negatives and true negatives.
relevance judgement: A decision about whether an item satisfies an information need.
test collection: A fixed collection, query set and relevance judgements used for evaluation.

Detailed teaching notes

Core concept

Evaluation asks whether retrieval results are useful, not just whether the system returns something. Precision focuses on result quality; recall focuses on coverage of relevant items.

Representation choice

For a query, retrieved relevant items are true positives, retrieved non-relevant items are false positives, and missed relevant items are false negatives. True negatives are usually huge in retrieval and less informative.

Method and reasoning

Calculate precision as TP/(TP+FP), recall as TP/(TP+FN), and F1 as 2PR/(P+R). Then interpret the numbers in relation to the user task.

Risk and limitation

A single metric can mislead. Medical or legal search may value recall; a top-ten web result may value early precision. Evaluation must match the task.

Degree-level deep dive

Evaluation needs a judgement set

Precision and recall require a definition of relevance for a set of queries. Without relevance judgements, the metrics are just arithmetic with no grounding in whether results satisfy users.

Metrics express trade-offs

Precision asks how many retrieved items are relevant. Recall asks how many relevant items were retrieved. F1 balances them, but a legal search, medical search or casual web search may prefer different trade-offs.

What excellent work shows

A strong answer constructs the confusion matrix carefully, calculates precision, recall and F1, and interprets the consequence for the user task. The conclusion should say whether the metric fits the scenario.

Concrete example to study

Metric calculation

Relevant documents: 8
Retrieved documents: 10
Relevant retrieved: 6
Precision = 6/10 = 0.60
Recall = 6/8 = 0.75
F1 = 2PR/(P+R) = 0.67

Reasoning

Precision asks how much of the result list was useful.
Recall asks how much of the relevant material was found.
F1 summarises the balance, but the scenario decides whether precision or recall matters more.

Reveal takeaway

The numbers only mean something when tied to a user task and relevance judgement set; a high score on the wrong metric can still produce a poor retrieval system.

Worked example

Scenario

A search retrieves 10 documents. Seven are relevant, and the assessor says there are 14 relevant documents in the collection.

Worked solution

Identify TP = 7 because seven retrieved documents are relevant.
Identify FP = 3 because ten were retrieved and three were not relevant.
Identify FN = 7 because fourteen relevant documents exist and seven were missed.
Calculate precision = 7/10 = 0.70, recall = 7/14 = 0.50 and F1 = 2(0.70)(0.50)/(1.20) = 0.583.

Reveal model result

The result has reasonable precision but weak recall. Whether that is acceptable depends on whether the user needs a few good hits or comprehensive coverage.

\text{Define the data, choose the representation, apply the method, interpret the result.}

Applied retrieval task

Evaluate two search configurations for a policy archive.

Deliverables

Create a small relevance table for one query.
Calculate precision, recall and F1 for two result sets.
Recommend one configuration for exploratory search and justify it.

Success checks

The recommendation follows from the task.
The calculation labels TP, FP and FN clearly.

Common misconception

Accuracy is rarely the main retrieval metric because true negatives dominate large collections. Precision and recall focus on the useful part of the result set.

Quick checks

1. In this lesson, why does precision matter?

The proportion of retrieved items that are relevant.It is only a presentational label and has no effect on retrieval quality.It replaces the need to understand user tasks or data representation.

2. Which answer best shows degree-level understanding of precision, recall, f-measure and evaluation?

Give a final recommendation without showing how the data or query was represented.Define recall, apply it to the scenario, then explain the trade-off or risk.Name as many technologies as possible without explaining the scenario.

Digital exam practice

Example exam task

A query has 20 relevant documents in the collection. A system retrieves 12 documents, of which 9 are relevant. In your answer, define the relevant objects or data structures, use course-specific vocabulary, show the method rather than only the result, and finish with a decision about the storage or retrieval system.

Notation and technical toolkit

P = TP/(TP+FP)precision Use to measure how many retrieved items were relevant.

R = TP/(TP+FN)recall Use to measure how many relevant items were found.

F1 = 2PR/(P+R)F-measure Use to balance precision and recall.

What a good answer is expected to show

A strong answer for Precision, Recall, F-Measure and Evaluation the answer should calculate precision, recall and F1, identify false positives and false negatives, and interpret the result for a retrieval task. It should connect the formal or technical representation to the user's information need instead of listing terms without using them.

How to solve it

Extract TP, FP and FN from the question.
Calculate precision as TP divided by retrieved items.
Calculate recall as TP divided by relevant items.
Calculate F1 and explain the trade-off.

Model answer

Reveal model answer

Here TP = 9, FP = 3 and FN = 11. Precision is 9/12 = 0.75, so most retrieved documents are useful. Recall is 9/20 = 0.45, so the system misses more than half of the relevant collection. F1 = 2(0.75)(0.45)/(0.75 + 0.45) = 0.5625. The system may be acceptable for quick browsing, but it is not strong for a task needing comprehensive retrieval.

Practise next

Create one result set with high precision but low recall.
Explain why relevance judgements can be difficult for exploratory queries.

Self-marking criteria

Defines the scenario objects, data forms or system components before solving.
Uses the lesson vocabulary accurately and in context.
Shows a clear method with enough working for a marker to follow.
Connects the result back to retrieval, storage, analytics or governance.
States a limitation, trade-off or quality risk rather than presenting the answer as absolute.

Extension

Research mean average precision and explain how it adds ranking position to evaluation.

Study route

Save a short worked answer from this lesson using this pattern: define the need or data, choose the representation, show the method, state the result and interpret the implication for the system.

Next lesson: Query Processing and Query Expansion.