A positional index is a search data structure that records not only which documents contain a word but also the positions where it occurs. A conventional inverted index may find all documents containing evidence; a positional index can help determine whether two or more words occur in a particular order or near one another. That makes exact phrase and proximity searches possible under clearly defined tokenisation rules.
Did you know that searching for the words ‘the child chased the dog’ is not the same as searching for those five words in any order? A document can contain all the same words while reversing the action. By retaining positions, search technology helps locate the actual expression a reader wants. But locating an expression is only one part of understanding it: the complete sentence, speaker, evidence and purpose still matter.
This complete guide explains what positional indexes are, why posting lists and token offsets matter, how exact phrase and proximity searches work, and where indexing can mislead through lost context, outdated text or ambiguous words. It connects the mechanism to Primary and Secondary English, Mathematics, Science, History, General Paper, translation and workplace language, with original examples and Stanford’s foundational information-retrieval documentation.
Editorial note: The indexing examples below use invented sentences and a one-based position convention expressly defined for teaching. No production search system, hit count or model performance result is claimed to have been tested.
Contents: how search remembers word order
Open 42 chapters and worked indexing exercises
- 1. An index can remember where a word appears
- 2. Why this is vocabulary technology
- 3. Begin with a clearly defined token sequence
- 4. An inverted index begins with terms and documents
- 5. A positional posting adds occurrence locations
- 6. Why document frequency is not enough
- 7. Phrase queries specify order and adjacency
- 8. Exact strings differ from loosely related concepts
- 9. Positions are counted after tokenisation
- 10. A two-word phrase needs an offset of one
- 11. A three-word phrase adds another condition
- 12. Order is different from co-occurrence
- 13. Proximity search relaxes exact adjacency
- 14. Ordered proximity and unordered proximity
- 15. Near in a document is not near in an argument
- 16. Sentence boundaries can matter
- 17. Stopwords may be essential to a phrase
- 18. Negation can reverse an indexed expression
- 19. Punctuation changes textual boundaries
- 20. Case handling creates ambiguity
- 21. Multiword names are ordered sequences
- 22. Words with several senses share positions
- 23. A concordance uses positional information
- 24. KWIC sorting asks about neighbours
- 25. Indexing may omit or normalise forms
- 26. Positional lists can use much more storage
- 27. A posting list may include counts too
- 28. Updating a document requires updating the index
- 29. The first match is not the best answer
- 30. A literal phrase can be quoted critically
- 31. Primary English: a five-token game
- 32. Secondary English: trace a quotation
- 33. Mathematics: phrase order sets conditions
- 34. Science: retrieve a specific procedure
- 35. History: separate a phrase from its source
- 36. General Paper: locate evidence without overclaiming
- 37. Translation: words may move to new positions
- 38. Public notices: exact wording can be consequential
- 39. Privacy and access control still apply
- 40. A reproducible six-column worksheet
- 41. A practical five-minute retrieval test
- 42. The final test: location, order and meaning
An original positional index example
Invented document A: ‘The student read the notice.’ After lowercasing, removing the final full stop and splitting on spaces, the tokens are numbered from 1. This convention belongs to this example, not every search engine.
| Term | Document ID | Positions | What it means |
|---|---|---|---|
| the | A | 1, 4 | The word appears twice |
| student | A | 2 | The word occurs after the first ‘the’ |
| read | A | 3 | The verb occurs at the third token |
| notice | A | 5 | The noun follows the second ‘the’ |
The exact phrase ‘the notice’ matches positions 4 and 5. The phrase ‘read the notice’ matches positions 3, 4 and 5. The phrase ‘notice the’ has no match in this sentence, even though both words are present. This is the practical difference between word presence and ordered phrase retrieval.
1. An index can remember where a word appears
A positional index is a data structure used in information retrieval to record the positions of terms within documents. A basic inverted index might tell us that evidence appears in documents A, C and F. A positional index goes further, recording locations such as token positions 4, 17 and 29 in each relevant document.
This extra information makes it possible to answer questions about phrase order and closeness. It can help a search engine find ‘strong evidence’ as a sequence rather than merely returning documents containing the two words somewhere far apart. The index locates strings under defined rules; readers still interpret meaning.
2. Why this is vocabulary technology
Understanding a dictionary word is not the same as locating the expression that solves a problem. A pupil looking for ‘at least’ should not be given every document that contains at and least in unrelated places. A positional index supports more precise retrieval.
That is vocabulary functioning as an interface between a question and a body of knowledge. Words become signals the machine can find in the right order, while the human decides what the phrase means and whether the surrounding passage answers the question.
3. Begin with a clearly defined token sequence
Consider the invented sentence ‘The student read the notice.’ For a simple demonstration, remove the final punctuation, lowercase the tokens and number them from 1. The resulting positions are: 1 the, 2 student, 3 read, 4 the, 5 notice.
These positions belong to the chosen tokenisation and counting convention. Another index may start at zero or split punctuation differently. The essential requirement is consistency, not the belief that every system must number words the same way.
4. An inverted index begins with terms and documents
An inverted index maps terms to the documents in which they appear. Its lists are often called postings. For student, a postings list may include document A and document C; for notice, document A and document B.
This makes word lookup faster than scanning every document from the beginning for each query. The first question is which documents contain a term. The next challenge is finding phrases, where the relative positions of terms matter.
5. A positional posting adds occurrence locations
A positional posting can associate a term with a document identifier and a list of token positions. For the invented five-token sentence, the appears at positions 1 and 4, while student appears at 2 and notice at 5.
The data are not a grammar analysis. The positions indicate where matched tokens occur in the indexed sequence. To know whether the introduces the student or the notice, a reader considers the surrounding words and sentence structure.
6. Why document frequency is not enough
Document frequency counts how many documents contain a term. A document containing both read and notice may or may not contain the phrase read the notice. Presence alone cannot establish a sequence.
A positional index preserves enough information to test whether positions match the requested offsets. This distinction is crucial for retrieving legal clauses, technical expressions and quotation fragments that must appear in a particular order.
7. Phrase queries specify order and adjacency
A phrase query such as ‘the notice’ asks for occurrences where the is immediately followed by notice under the index’s tokenisation rules. In the invented sentence, the matching positions are 4 and 5. The earlier the at position 1 does not start that phrase.
The index tests token offsets rather than guessing grammatical meaning. Finding the phrase is useful, but the complete sentence is still required to interpret what the notice says. Phrase matching is evidence of text occurrence, not of truth.
8. Exact strings differ from loosely related concepts
An exact phrase search for strong evidence need not retrieve compelling evidence or evidence that is strong. The latter expressions might convey related ideas, but they contain different token sequences.
This makes phrase search precise but potentially narrow. If the learner’s goal is a concept rather than a fixed quotation, query expansion or semantic resources may be needed. Expanding a search should still preserve differences in force and grammatical construction.
9. Positions are counted after tokenisation
A search index usually stores positions assigned during preprocessing. If punctuation, contractions, hyphens or symbols were split differently, token offsets can differ from what a reader expects by counting visible spaces.
Therefore, every serious technical description includes the tokenizer or relevant rules. A phrase might match in one index but not another because its token boundaries were constructed differently. The displayed words alone do not explain every search result.
10. A two-word phrase needs an offset of one
Suppose read occurs at position 3 and the at position 4 in the same document. That pair satisfies the adjacency condition for the phrase ‘read the’. If the second term occurs at position 6, there is an intervening token and the exact two-word phrase does not match.
This offset idea scales to longer phrases. The index compares positional lists for successive terms instead of reading each possible sentence as a human would. It can find an exact sequence efficiently, provided the indexing and query rules agree.
11. A three-word phrase adds another condition
In ‘The student read the notice’, the phrase ‘read the notice’ starts at position 3. The next words must occur at positions 4 and 5. The index can test all three offsets together.
This demonstrates why a position list is more useful for phrase queries than a basic document list. The phrase is a chain of conditions. A missing or reordered word breaks the exact match even when all three terms appear elsewhere in the document.
12. Order is different from co-occurrence
The expressions ‘the dog chased the child’ and ‘the child chased the dog’ contain the same ordinary content words. A bag-of-words matrix cannot distinguish those sequences when it records only counts. A positional representation can distinguish their arrangements.
That does not mean a positional index understands animal behaviour or assigns semantic roles. It simply retains the sequence needed for a later language analysis. Knowing where tokens occur is a necessary but not sufficient step toward understanding the event.
13. Proximity search relaxes exact adjacency
A proximity query asks whether two terms occur within a stated distance or window. Unlike an exact phrase, it may allow intervening tokens. Some systems also distinguish ordered proximity from proximity in either order.
The exact meaning of ‘within five words’ depends on the engine’s distance convention and tokenisation. A student should not assume every NEAR operator has identical syntax or inclusivity. Tool documentation governs the technical result.
14. Ordered proximity and unordered proximity
Ordered proximity requires that terms appear in a specified sequence, possibly with allowed gaps. Unordered proximity asks about closeness regardless of their order. These represent different retrieval questions even when the same two terms are supplied.
For a phrase whose order carries meaning, ordered matching may be essential. For exploratory searches about two related concepts, unordered proximity may help locate relevant passages. In either case, closeness alone does not prove a causal or logical relationship between the terms.
15. Near in a document is not near in an argument
Two terms can appear one token apart yet belong to a quotation, a negated clause or a list of unrelated topics. Positional proximity captures a relationship in the text sequence, not necessarily a semantic dependency.
If a query returns a surprising pairing, read the complete sentence and source context. A computationally valid hit can be an inappropriate example for a student’s composition. Search precision must be followed by interpretive precision.
16. Sentence boundaries can matter
An index that ignores sentence boundaries may find nearby terms separated by a full stop. Depending on the search implementation, a proximity match could cross from one sentence to the next, where the terms do not form one phrase.
A suitable query may need sentence restrictions or a larger context check. Never teach a collocation from two adjacent tokens without confirming they belong to the intended grammatical construction.
17. Stopwords may be essential to a phrase
Very common words such as the, not and only can be essential for exact phrase queries and the meaning of an instruction. Removing them indiscriminately from indexing can make certain phrases impossible to reconstruct accurately.
Some retrieval systems handle frequent words specially rather than simply discarding them. The correct design depends on requirements and implementation. The broader vocabulary lesson is that frequency does not determine semantic importance.
18. Negation can reverse an indexed expression
Consider ‘The results supported the proposal’ versus ‘The results did not support the proposal.’ A keyword search for support and proposal might retrieve both, but the propositions differ sharply.
A positional index can help locate tokens and possibly phrases involving not; it cannot by itself judge the evidence. The full clause and its context must be read before claiming that the proposal was supported.
19. Punctuation changes textual boundaries
Quotation marks, commas, dashes and full stops can influence how text is tokenised and which positions are assigned. In structured or legal documents, punctuation may also change meaning or divide clauses.
An exact phrase match may depend on whether punctuation is included, stripped or treated as separate tokens. A search workflow should preserve a link to the original text so that readers can verify its meaning and formatting.
20. Case handling creates ambiguity
A case-insensitive index may combine May the month with may the modal verb. A case-sensitive index may treat them differently but still need grammatical disambiguation in sentence-initial positions.
When the aim is a specific sense or part of speech, positional data alone is not enough. Use appropriate tags, surrounding context or manual reading. A good retrieval system distinguishes its string-matching features from actual semantic understanding.
21. Multiword names are ordered sequences
A location such as New York contains two visible words in a common English tokenisation. A positional index can find that ordered phrase without declaring that the two tokens are conceptually separate entities.
A named-entity layer could add further meaning. The index answers where the words occur; entity recognition attempts to identify what the phrase refers to. The two technologies complement one another.
22. Words with several senses share positions
A surface spelling such as charge can refer to a fee, a battery or a legal allegation. A positional index stores locations of matching tokens regardless of sense unless other processing explicitly separates them.
If a student needs electrical usage, the search may be refined with relevant neighbours. Yet the context still needs checking: a finance article could mention charging a device. The index is a locating technology, not a substitute for dictionary and subject knowledge.
23. A concordance uses positional information
A concordance displays a target with surrounding words. Position information helps software retrieve the appropriate local context and align occurrences for comparison.
This gives the learner a bridge between fast search and close reading: first locate a match, then inspect the full sentence, classify meaning and use the result when composing a new sentence. Neither speed nor alignment should replace understanding.
24. KWIC sorting asks about neighbours
KWIC sorting can order concordance lines by the token immediately before or after the target. A positional index can provide or support access to the positions of those neighbours, although actual tool architectures differ.
The resulting sequence may reveal candidate collocations. But the neighbour is not automatically the target’s grammatical object or modifier. The learner must read the clause before turning a nearby word into a rule.
25. Indexing may omit or normalise forms
Some systems lowercase, stem or lemmatise terms when indexing. A search for an exact visible word might then need additional information to recover the original form, depending on implementation.
For citations and precise technical writing, the original wording matters. The system should retain appropriate source references and, when required, stored text. Index normalisation is a practical choice with consequences, not a universal linguistic rule.
26. Positional lists can use much more storage
Recording every position consumes more storage than recording document IDs alone. Compression techniques can reduce the burden, and search systems choose trade-offs among storage, speed, supported queries and update complexity.
The concept matters to learners because richer search capabilities have engineering costs. A good system is designed for the questions readers actually ask. A simple class exercise can demonstrate positions without needing to implement a production-grade compressed index.
27. A posting list may include counts too
A positional posting can contain a document ID, term frequency and the occurrence positions. Since the position list already identifies occurrences, the count may be derived from its length or stored for efficiency under a particular implementation.
Do not assume that the same structure or format is used by every modern search engine. The Stanford information-retrieval text provides a foundational model; actual systems may partition, compress or optimise their indexes differently.
28. Updating a document requires updating the index
If a school notice changes from ‘may attend’ to ‘may not attend’, the indexed terms and positions must reflect the revised version. A stale index can direct readers to obsolete wording or produce misleading search snippets.
This is why version, timestamp and authoritative source matter. A positional index can find words accurately in the indexed copy while the real policy has already changed. Search results must be checked against the current controlling document.
29. The first match is not the best answer
An index may retrieve many passages that satisfy a phrase query. Ranking requires additional signals about relevance, document quality and user intent. Phrase presence alone cannot tell whether the result answers the student’s actual question.
A learner should open the most relevant source, check context and compare alternatives rather than copy the first sentence that looks similar. The index helps locate candidates; the reader evaluates their usefulness.
30. A literal phrase can be quoted critically
A document may contain the exact words of an accusation while arguing that the accusation is false. The index correctly retrieves the phrase, but a reader who ignores quotation and denial may attribute the wrong position to the author.
For History and media literacy, search is only the first stage. Identify who spoke, what was claimed and whether independent evidence supports it. Exact retrieval is not identical to factual endorsement.
31. Primary English: a five-token game
Write the original sentence ‘The child found the key’ on five cards, number them 1 through 5 and ask the pupil to locate ‘the key’. Then rearrange the cards and ask which phrases still appear exactly.
This makes adjacency and order visible without software. The child learns why the same words can form different sentences and why search for an exact phrase differs from search for separate words. Finish with a new meaningful sentence.
32. Secondary English: trace a quotation
A Secondary learner may search for a short quotation in a long text to examine its speaker and surrounding passage. An exact phrase search can locate the wording efficiently if the edition and tokenisation match.
Students should then read enough context to understand irony, contrast or attribution. Finding the quotation is not literary analysis; explaining its role in the source is. Search technology supports close reading rather than replacing it.
33. Mathematics: phrase order sets conditions
The expressions at least five, at most five and less than five differ in their mathematical implications. A system that recognises phrases can help find appropriate example problems.
Students must still translate the actual words into inequalities. The index can show where an expression occurs, but it cannot certify the answer to a Mathematics question. The success test is an accurately interpreted boundary.
34. Science: retrieve a specific procedure
Searching for an exact procedure phrase may help a student locate the relevant section of a lab manual. Technical instructions often rely on sequence markers such as before, after and only if.
The current authorised manual remains the reference for performing the work. A retrieved fragment could omit a critical safety warning. Positional matching should lead the user to the complete procedure, not replace it.
35. History: separate a phrase from its source
A historical phrase may be repeated by different writers, quoted in debate or reused decades later. A positional index can locate the string across documents, helping researchers inspect variation in context.
It cannot decide whether those documents represent independent sources or whether the phrase carries the same meaning over time. Date, author, audience and corroborating evidence remain essential.
36. General Paper: locate evidence without overclaiming
A GP student may search for a phrase that appears across several articles. An exact match makes the passages easy to find, but several occurrences could all repeat one original claim.
Check provenance and source independence before using the material as corroboration. A large number of search hits is not a large number of verified facts. Retrieval and evidence evaluation are separate intellectual tasks.
37. Translation: words may move to new positions
Languages differ in typical word order, morphology and segmentation. A phrase appearing as three tokens in one language may be expressed with one or several differently ordered elements in another.
A positional index is useful within each appropriately processed language, but direct comparison of token offsets is not a test of translation accuracy. The communicative action and full meaning must be preserved.
38. Public notices: exact wording can be consequential
Consider the difference between ‘staff may enter’ and ‘staff may not enter’. A search system that retrieves both under the broad word enter has done a useful locating job but has not resolved the permission question.
Before acting, readers should open the current notice and identify the complete condition, responsible authority and effective date. The practical value of precise vocabulary is preventing a wrong action.
39. Privacy and access control still apply
An index built from student writing or workplace documents can reveal sensitive terms, locations or identifiers even if it does not display the full documents. Search access should respect the permissions of the underlying material.
Use invented sentences for public demonstrations and authorised tools for private records. An accurate positional index does not automatically make indexed content safe to expose. Data governance is part of reliable knowledge technology.
40. A reproducible six-column worksheet
Record the original sentence, tokenisation rule, chosen term, document identifier, token positions and intended phrase query. An additional column can note whether the result was a genuine match and whether context altered the interpretation.
This turns a seemingly invisible indexing mechanism into something a student can audit. The worksheet helps diagnose errors in case handling, token boundaries or phrase offsets without pretending to model a whole production search engine.
41. A practical five-minute retrieval test
Take two invented sentences with identical content words but different order. Construct their token positions, then check whether the phrase query matches each. Explain what the index found and what information it did not determine.
The test is simple enough for school and powerful enough to expose a core limitation of bag-of-words representations. The final reflection asks what additional context a human would need to understand the sentences completely.
42. The final test: location, order and meaning
A student who understands positional indexes can distinguish a word’s presence from its exact sequence, recognise how a phrase query is evaluated and explain why the original source still matters.
That is vocabulary as a technology: language can be organised so we can find knowledge quickly, but words remain meaningful only in the relationships they express. The right result is not merely a match; it is a match interpreted accurately.
A seven-step paper index exercise
- Write two original sentences using the same small vocabulary in different orders.
- Choose and state a tokenisation convention, including how punctuation is handled.
- Number the tokens consistently within each sentence.
- Create one list of document IDs and token positions for each target word.
- Test one exact phrase by checking successive positional offsets.
- Test a wider proximity question and state its distance rule.
- Explain how the same vocabulary can express different meanings when order or negation changes.
Ten frequently asked questions
1. What is a positional index?
A mapping from terms to documents and occurrence positions within those documents.
2. Why not store only document IDs?
Document IDs show presence but cannot by themselves verify phrase order and adjacency.
3. Does the first word always have position one?
No. This guide uses one-based positions as an explicit teaching convention; systems can use other conventions.
4. What is a phrase query?
A search for terms occurring in a specified order and usually consecutively under the engine’s matching rules.
5. What does a proximity query test?
Whether terms occur within a defined distance, sometimes with an order restriction.
6. Does a valid match prove a statement is true?
No. It proves that the indexed text satisfies the search rule, not that its claim is correct.
7. Can punctuation change a match?
Yes. Tokenisation and indexing choices can affect positions and phrase boundaries.
8. Can a positional index distinguish every word sense?
No. Sense disambiguation requires further contextual or semantic analysis.
9. Is it the same as a document–term matrix?
No. A simple matrix stores term features by document; a positional index retains locations of term occurrences.
10. What is the student learning goal?
Understand word order as searchable evidence, then interpret the complete expression accurately.
Authoritative sources and linked vocabulary lessons
- Manning, Raghavan and Schütze: Introduction to Information Retrieval — foundational discussion of positional postings, phrase queries and indexes.
- Introduction to Information Retrieval, Chapter 2: The Term Vocabulary and Postings Lists — exact technical source for positional indexes and positional merges.
- Introduction to Information Retrieval, Chapter 1: Boolean Retrieval — document occurrence and term-document indexing foundations.
- Sketch Engine: Concordance — using searches to inspect words in context.
- Vocabulary | What Is a Document–Term Matrix? — compare search features with preserved token positions.
- Vocabulary | What Is a Corpus Query? — build reliable search conditions.
- Vocabulary | What Is a Word Token? — understand indexed units.
- Vocabulary | What Is KWIC Sorting? — use neighbouring-word positions to inspect patterns.
- Vocabulary Learning Hub — vocabulary as a school-to-life system.
A positional index is one of the quiet technologies that make precise language retrieval possible. It helps us find where words stand together. Understanding them requires the next step: reading what those words actually say, in their authentic context.