How artificial intelligence works when finding highlighted and underlined sentences in study notes begins with a familiar revision habit. A student opens a textbook and marks one definition yellow, underlines a mathematical rule and circles a tricky word. When the pages are photographed, the marks become coloured pixels and pencil strokes. Document AI can attempt to identify them and reconnect them to the exact words they emphasise.
Did you know that detecting a yellow region is not the same as deciding which sentence was highlighted? A yellow diagram, a printed title bar and a translucent highlighter mark may share similar colours. The model needs to decide which regions are user-added annotations, which printed words lie underneath them and whether a thin horizontal line is an underline, a table border or a fraction bar.
The useful chain is source page → locate text and its reading order → analyse colour and hand-drawn marks → classify highlight or underline regions → associate each mark with specific words → preserve its position and colour → export a reviewable study-note index. The purpose is to help a learner return to actual marked content, not to declare that every highlighted phrase is scientifically correct or more important than the surrounding explanation.
Quick summary
- Detect colour or hand-drawn marks
- Read the text beneath them
- Link each annotation to its words
- Preserve original position and context
- Review uncertainty and personal colour conventions
1. What is being extracted from the page?
A photographed textbook contains at least two layers of meaning: the printed material and the reader’s annotations. A highlight often lies behind text, while an underline sits near its baseline. An arrow or a marginal note may refer to a phrase in another region.
The extraction task is to record what was marked and where. It should preserve the exact wording and page reference, perhaps together with the annotation type and colour. The system is not automatically producing a new summary or deciding which facts the reader should memorise.
2. Why annotation detection differs from OCR
Optical character recognition estimates printed or handwritten characters. It can read a definition accurately even when it has no idea that the student highlighted it. Conversely, it can detect a coloured stroke while misreading the text underneath.
The full workflow needs both outputs: word boxes with their coordinates and annotation regions. An incorrect overlap rule can connect the right colour to the wrong sentence. A good product exposes the page crop so the student can inspect exactly what the computer selected.
3. How colour thresholding can find marker ink
Many highlighters produce relatively saturated yellow, green, pink or orange regions that differ from ordinary black text on white paper. A classical image-processing technique can convert the page to a hue–saturation–value representation and select pixels within a chosen colour range.
OpenCV’s official inRange tutorial explains HSV colour-based segmentation. It is a useful baseline for detecting coloured areas but does not understand why they are coloured. Printed page decorations, photographs and marker ink can occupy overlapping ranges and create false positives.
4. Transparent marker ink changes the underlying text
A translucent highlighter typically allows text to remain visible while tinting the surrounding paper and ink. The observed pixel colour is a combination of printed content, marker pigment, paper, lighting and camera response.
A detector cannot assume every yellow pixel represents a uniform pure yellow marker. Shaded letters may appear darker and less saturated than blank areas of the same stroke. Region grouping and context can help reconnect broken portions of one highlight, but this is an estimate rather than a direct record of the marker’s chemistry.
5. Why printed yellow headings cause mistakes
A textbook may use yellow title bands or coloured tables as part of the published design. A student may also draw a marker stroke across a paragraph. Both can appear as bright yellow blocks in a photograph.
The model can inspect irregular hand-drawn boundaries, line height, overlap with text and repetition of the same printed design across pages. It can compare colours and shapes, but not every case has a unique answer. A clean rectangular annotation is especially difficult to distinguish from a printed design without source information.
6. Underlining is a geometrical task
An underline usually appears immediately below a word or phrase. Its visual clue is an elongated stroke that roughly follows the text baseline. A detector can find line candidates and compare them with OCR word boxes.
However, the page may contain fraction bars, table rules, arrows and decorative dividers. A horizontal stroke is not automatically a reader’s underline. The model needs local text geometry and wider layout context before deciding what the line is marking.
7. Recognise a whole phrase rather than a few pixels
Suppose a student highlights the sentence ‘Energy is transferred, not created’. The highlighter may stop slightly short of the first word or extend past the full stop. A pixel-level mask will not necessarily align exactly with the word boundaries.
Software can estimate which text boxes overlap the marker region sufficiently to belong to the selection. But an aggressive rule may include part of the next sentence; a strict one may drop a short final word. This is why line-level and word-level review remain important.
8. A fictional science-notes example
Imagine a student highlighting a plant-transpiration definition yellow, drawing a green underline beneath a key term and circling one unfamiliar word. A page analyser finds the text, produces two coloured masks and identifies the separate circular annotation.
A note index might show the exact highlighted definition and the underlined term, each linked to its source page. The circle may be treated as a mark requesting extra vocabulary review, but that interpretation depends on the student’s own study convention.
Now imagine a green arrow printed in the diagram beside the underline. The model must keep the diagram arrow separate. The student and annotations are invented teaching examples, not reported processing results.
9. How a model learns user-added marks
IBM researchers described a system for detecting highlights and notes on printed text in 2008, combining colour-space analysis with anomaly detection in text-dominant document images. This illustrates that user-added annotations are a recognised, distinct document-analysis task.
A learned segmentation model can extend the approach by training on photographed pages with marked annotation masks. It may handle unusual colours, uneven paper or marker strokes better in some conditions, but it requires examples of legitimate printed design as well as genuine annotations.
| Mark | What AI can observe | What it cannot assume |
|---|---|---|
| Yellow highlight | Colour region across text | The sentence is a proven fact |
| Pencil underline | Stroke near text baseline | A nearby fraction bar is also a mark |
| Circle around word | Closed hand-drawn outline | The student’s precise reason |
| Margin arrow | Line and arrowhead | Its target without geometry |
| Printed coloured title | Design colour | A student added it |
10. Why weak pencil underlines are hard
An underline made in pale pencil may have nearly the same intensity as the paper texture. A threshold that cleans the background too aggressively can remove it before the annotation classifier sees it.
A rich original colour scan and cautious local contrast processing may help retain weak marks. Even then, a near-invisible stroke should remain uncertain rather than be invented because a language model expects an important word to be underlined.
11. Marginal notes and arrows add new relationships
A student’s note in the margin may point with a hand-drawn arrow to a sentence. That link can be more useful than simply extracting the note text. The system needs to identify the note region, the arrow direction and its intended endpoint.
Not all margin marks are annotations: page numbers, publisher labels and diagrams can occupy the same space. Document structure is essential. A program should label an uncertain association rather than connect a note to the nearest convenient paragraph.
12. Multiple highlighter colours may encode a private code
Some students use yellow for definitions, pink for mistakes and green for examples. Another learner may use colours with completely different meanings, or choose them randomly. The model can observe pigment colours without knowing the student’s convention.
A useful app can let the reader define those meanings. It should not automatically claim pink means urgent or green means mastered. Colour is recorded evidence; importance is a separate interpretation that should come from the user or a documented study scheme.
13. What if every sentence is highlighted?
A page with many marked lines provides many candidates, but their sheer number does not prove the contents are equally important or well understood. AI should not assign educational mastery to a highlight simply because it is prominent.
Research on highlighting as a study strategy has examined whether different colours and marking patterns influence later recall, with findings depending on the task and study design. Extraction software can help organise annotations, but it does not replace actual retrieval practice and understanding checks.
14. Why retaining the unmarked text matters
An excerpt without context can be misleading. A highlighted sentence may be followed by a qualification or exception that the student did not mark. A flashcard generator that captures only the highlight might therefore oversimplify the subject.
A responsible tool can show the sentence in its paragraph and page, letting readers open surrounding text. It should distinguish an author’s source words from the student’s annotation and from any AI-generated revision suggestion.
| Situation | Failure | Repair |
|---|---|---|
| Yellow title bar | False highlight | Check printed design |
| Faint pencil mark | Lost underline | Preserve original contrast |
| Highlight extends across lines | Wrong word selection | Inspect line and word boxes |
| Multiple colours | Invented study priorities | Use student-defined conventions |
| Arrow beside diagram | Wrong linked note | Check endpoint and context |
15. How to evaluate annotation extraction
Build a verified test collection with real printed colour, handwritten underlines, multiple marker shades, shadowed pages, faded annotations and overlapping text. Compare predicted annotation regions with reference masks and assess the linked words.
Measure region detection and text association separately. A system can correctly identify a yellow area while selecting the wrong line, or identify an underline while misreading a small mathematical symbol. Evaluation should include both image and semantic fidelity.
16. Privacy and copyright considerations
Study notes may contain student names, personal comments or copyrighted textbook pages. A scanning app must use appropriate permissions, access controls and retention practices. Producing a private study index is not automatically permission to republish a full textbook.
Keep only the information necessary for the intended revision workflow, and distinguish private annotations from public learning resources. The tool’s ability to extract coloured text does not decide the relevant rights or sharing policies.
17. A short classroom experiment
Write three original sentences. Highlight one yellow, underline the second in pencil and place a coloured printed rectangle beneath the third. Photograph the page and ask a partner which marks were added and which belong to the document design.
Now change the lighting and examine the colours. Add a fraction bar or a table rule near the underline. The exercise reveals why mark detection, text recognition and annotation interpretation are separate stages.
18. The bigger lesson: a mark must be linked to its context
Highlighted-note extraction is a small example of multimodal document understanding. The model observes both text and annotation geometry, then estimates which parts of the document the reader chose to mark.
The technology becomes trustworthy when it preserves exact source wording, shows uncertain regions and lets the student define personal colour conventions. It should help readers return to their own marked evidence rather than mistake colourful pixels for complete learning or verified importance.
19. A complete annotation index needs source coordinates
A learner’s note index can store the text, page, line and annotation colour, but should also preserve the precise location of the marker in the original image. Otherwise, two identical sentences on different pages may collapse into one entry and the student may lose the context.
Make each extraction reversible. Clicking the saved note should return to the full paragraph and show which pixels or underline motivated the selection. An AI-written paraphrase may be useful later, but should remain separately labelled as a new explanation rather than the original student’s marked text.
Text fidelity is especially important for formulas, dates and quoted definitions. A one-character OCR mistake can change what the highlighted sentence actually says.
20. Colour lighting, saturation and camera exposure
A neon marker can look yellow under one lamp and almost white under strong overexposure. A phone’s automatic white balance may change a green pen towards blue. Colour thresholding therefore needs testing on actual document photos rather than assuming one rigid HSV interval matches every capture.
A model may use nearby paper colour as a local reference and group strokes with similar boundaries. Yet it should still flag cases where a printed title background resembles a highlighter mark. A numeric colour difference does not establish that a person added the mark.
Retaining a colour source scan makes it possible to refine thresholds later without throwing away the original information during premature conversion to black and white.
21. Underline endpoints and adjacent sentences
Suppose a student underlines ‘Check the units’ but the line runs a little farther than the words. A strict pixel overlap threshold may drop the first or last word; a loose threshold may include the following phrase. The system needs a defined policy for how geometry becomes a selected text span.
Word boxes, baseline orientation and punctuation help. For multi-line highlights, the reader might have used two disconnected strokes for one idea. A grouping stage can propose one selection, with the original page available to verify the boundaries.
Do not automatically create a complete grammatical sentence from fragments. The exact marks may cover only a formula or keyword, and that partial selection can be the user’s intention.
22. What a good study-note review panel shows
A useful interface can display the source page, a coloured overlay of detected marks, extracted words and an uncertain-region indicator. It should let learners correct a mark type or text association without redrawing the whole page.
Evaluate both annotation-mask accuracy and whether the final extracted phrase is correct. A mathematically good mask can still include the wrong line of text, and a fully readable OCR result can miss the highlight entirely.
The final educational test is practical: can the student return to the highlighted explanation and compare it with the unmarked qualifications? If not, the extraction has not served the purpose of learning.
Frequently asked questions
How does AI find highlighted text?
It detects likely colour regions and associates them with OCR text boxes.
Is HSV a neural network?
No. HSV is a colour representation useful for classical thresholding.
Why can a yellow box fool the model?
It may be printed design rather than a user-added marker.
Can AI detect pencil underlines?
Sometimes, with geometry and local contrast, but faint marks can be missed.
Does highlighting prove mastery?
No. It indicates a mark, not the learner’s understanding.
Can different marker colours have different meanings?
Yes, but the reader’s own convention must be supplied.
Why preserve neighbouring text?
Qualifications and context can alter a highlighted sentence’s meaning.
Can the model distinguish a fraction bar from an underline?
Context and text geometry may help, but ambiguous cases require review.
What should a study-note export contain?
Exact text, annotation type, colour, page location and review status.
Primary research and related learning
- IBM Research: Detecting Highlights and Notes on Printed Text
- OpenCV: HSV Colour Segmentation Using inRange
- ERIC: Highlighting Text as a Study Strategy
- How AI Finds Faded Ink Without Inventing Missing Letters
- How AI Finds Footnotes in Scanned PDFs
- How AI Understands Arrows in Science Diagrams
- eduKateSG | How X Works Hub
Editorial note: All study pages and colours are invented teaching examples. A detected highlight is evidence of a visual annotation, not proof that a sentence is correct or understood. Keep the unaltered source and verify text associations before exporting notes.