How artificial intelligence works when matching captions to diagrams becomes clear when a student opens a scanned science article containing three figures. The caption beneath a drawing says ‘Figure 2: Water moves through the stem,’ but the nearby picture might belong to Figure 1 because the page uses two columns and a crowded layout. A computer that attaches the wrong caption can give a perfectly fluent explanation of the wrong experiment.
Did you know that recognising the words in a caption and finding the diagram it describes are separate tasks? A text recogniser can read every caption correctly and still attach it to the wrong figure. Document-layout AI attempts to locate candidate figures and caption text, then infer the correct relationship using page geometry, labels, reading order and sometimes semantic content. A strong result preserves the original page location so the relationship can be checked.
The useful pipeline is PDF or scan → detect figure, table and caption regions → recognise caption text → parse figure identifiers → propose candidate visual regions → score spatial and structural correspondence → resolve conflicts → export matched pairs with source coordinates → review difficult cases. Proximity is a clue, not proof. The correct figure may sit above its caption, beside it or on a nearby page.
The quick route: five questions
- Where are the figures and caption candidates?
- Which words identify the figure number and scope?
- How does page geometry support a particular pairing?
- Could panels, legends or page breaks change the match?
- Can the pair be checked in the original document?
1. The problem: words can be correct while relationships are wrong
Imagine a printed worksheet with a drawing of a plant and a drawing of a thermometer. If the description of the plant is attached to the thermometer, the reader may learn an incorrect association even though no individual word was misread.
Document analysis has to recover relationships that a human infers from layout. Captions supply titles, explanations, units and contextual information. Diagrams provide shapes, measurements and visual evidence. Their connection is part of the document’s meaning.
An AI summary that ignores those links can attribute the wrong temperature to an experiment or describe the wrong stage of a process. The task therefore needs explicit matching rather than an unstructured pile of extracted images and paragraphs.
2. What figure–caption matching means
Figure–caption matching identifies which textual description belongs to which visual region in a document. The visual may be a diagram, photograph, graph, chart or composite of several panels. A table and its caption create a related but distinct relationship.
The output might contain a figure image, the full caption, a figure number, page index and bounding boxes for both elements. These fields allow a later reader or search tool to inspect the exact source.
Matching is not the same as generating a caption from an image. Generative image captioning invents or infers descriptive words; extraction aims to preserve the document author’s existing caption and associate it with the correct source graphic.
3. Why ordinary OCR is not enough
Optical character recognition converts visible letters into text. It may recover ‘Figure 3. Leaf structure’ without knowing whether the relevant drawing is above, below or in the neighbouring column.
If a document contains two captions close to one another, simply sorting all text from top to bottom can attach a caption to the wrong graphic. OCR confidence measures recognition of visible characters under a model; it is not a guarantee of layout association.
A robust process stores each text block’s coordinates and label, then uses figure-region detection to establish links. The text string and its location are both important evidence.
4. How a document-layout detector sees the page
An image or PDF-layout model can classify areas such as title, body paragraph, figure, table, caption and footer. Learned detectors use examples labelled with those categories, while traditional systems can analyse PDF text objects and graphical components.
The initial output often contains rectangles or polygons marking candidate regions. A page with a large graph, an inset legend and a small caption may need several components to be grouped into one figure rather than treated as unrelated objects.
DocLayNet and scientific-document layout datasets demonstrate that figure and caption detection require explicit annotations. A model trained mainly on journal articles may not generalise to classroom worksheets or handwritten diagrams without adaptation.
5. What makes a caption look like a caption?
Captions often begin with a recognizable prefix such as ‘Figure 1’, ‘Fig. 2’ or a table number. They may use smaller text, a distinct font or a position close to a graphic. These conventions help an algorithm locate candidates.
Yet authors vary their layouts. A caption may appear above a table, below a photograph or beside a figure in a narrow column. Some documents use unnumbered captions or place the label at the end rather than the beginning.
An extractor should use the visible prefix as a clue without treating any number following the word figure as definite evidence. Body prose saying ‘see Figure 2’ is a cross-reference, not necessarily the caption itself.
6. Candidate diagrams need complete boundaries
Finding the main picture is not always straightforward. A figure may contain four labelled subpanels, a legend on the side and annotations that occupy significant space. Cropping only the central plot can lose the context needed to read it.
PDFFigures 2.0 describes extracting figures, their captions, region boxes and inferred figure names. Its implementation proposes candidate figure regions near captions and scores them, while trying to avoid selecting overlapping regions for separate captions.
This illustrates that matching depends on defining what counts as the full figure. A caption attached to a cropped figure missing its legend may be technically associated with the correct location while still delivering an incomplete educational resource.
7. Spatial proximity is useful but insufficient
Humans usually expect a caption to be near the diagram it describes. A matching system can use the distance and direction between caption and figure candidates to rank possibilities. It can also check whether they occupy the same column or section.
But two figures might appear side by side, with captions below them in a separate row. The geometrically closest caption to one image can be the wrong one if a column break or shared page design is misunderstood.
Spatial matching should therefore consider layout grouping and document conventions, not just the shortest straight-line distance between two rectangles. When geometry alone is ambiguous, a review flag is often more reliable than forced assignment.
8. A fictional two-column science article
Imagine a fictional science article with a plant cross-section in the left column and a graph of plant growth in the right column. Beneath the left image is the caption ‘Figure 1: Stem layers’. Beneath the right is ‘Figure 2: Growth over ten days’.
An OCR engine reads both captions. A layout detector returns two visual regions and two caption boxes. The matcher uses columns, proximity and figure identifiers to propose the two pairs. A verifier can show the exact highlights on the source page.
Now change the layout so Figure 2’s graph extends across the page, while its caption begins below the left column. A simple column-matching rule may fail. A richer algorithm evaluates the whole layout and can flag an uncertain case.
The worksheet and figure numbers are invented to illustrate the mechanism, not results from a benchmark.
9. Figure numbers and body references supply extra constraints
A document’s main text may say ‘Figure 3 shows the change in pressure’. That reference helps confirm that a caption labelled Figure 3 is associated with a diagram discussed in that section.
Some extraction systems parse figure labels and numbers from captions, then record links from body references. If a proposed match places the Figure 3 caption beside a graphic clearly labelled ‘A’ while another candidate fits the figure sequence, the mismatch deserves review.
References do not prove a visual pairing by themselves. Articles can discuss a figure on the previous page or cite several figures together. Identifiers are constraints supporting a hypothesis rather than infallible coordinates.
10. Multiple panels introduce a hierarchy
A scientific figure may contain panels labelled (a), (b), (c) and (d), all explained in one caption. Each panel can show a different experimental condition. Splitting that figure into four unrelated pictures would destroy the shared caption structure.
A useful extraction schema may represent the complete figure as a parent with child panel regions. It can preserve the caption’s descriptions of each letter without pretending that panel (b) has its own independent figure number.
Researchers and teachers need that hierarchy to survive export. A student reading only one detached panel might misunderstand the comparison and repeat the wrong conclusion. Correct matching includes identifying which graphical parts form one logical unit.
| Stage | What it produces | What remains uncertain |
|---|---|---|
| Layout detection | Candidate figure and caption regions | Which pair belongs together |
| OCR or PDF text extraction | Caption words and identifiers | Whether the text is the true caption |
| Figure boundary extraction | Graphic including panels or legend | Whether all parts were included |
| Pair matching | Proposed figure–caption association | Correctness of ambiguous layouts |
| Source verification | Original page and coordinates | Any unsupported content inference |
11. What if the caption is on the next page?
Page-break layouts can separate figures and captions or place the diagram near a different page region from the section text that discusses it. A system limited to same-page matching may miss those relationships.
A broader parser can consider nearby pages and explicit figure numbers, but widening the search increases the risk of matching one caption to an unrelated graphic. It should not automatically attach a caption across a page boundary simply because the labels are similar.
The original physical order, PDF page label and rendered coordinates should remain traceable. A cross-page candidate can be represented as uncertain and verified by an editor or reader.
12. Separating a table caption from a figure caption
Scientific documents often follow conventions in which figure captions appear below images while table titles or captions appear above tables. Those conventions vary, and models should not assume universal positions.
A table is structurally different from a diagram. It contains rows, columns, headers and perhaps spanning cells. A label attached to a table can include units or conditions essential for interpreting the numbers.
The extracted object type matters. A caption about ‘mean rainfall in millimetres’ must remain attached to the correct data table or graph. Confusing the two can lead a downstream AI to invent a visual relationship that was never present in the source.
13. The role of learned visual and textual similarity
Beyond layout geometry, a system may compare the caption’s vocabulary with visual features or text recognised inside candidate diagrams. If a caption mentions a microscope image while one candidate resembles a graph, semantic evidence could help select a plausible pair.
However, semantic matching can be misleading. Two nearby charts may both concern plant height, and their captions may differ only by dates or experimental groups. A model attracted to topic similarity can attach a caption to the wrong visually similar plot.
Document structure, figure labels and source coordinates often provide stronger constraints than a free-form generated description. Learned similarity is useful as a supporting clue, not as a licence to override visible numbering.
14. Where extraction research gives concrete examples
PDFFigures 2.0 provides a publicly described workflow for extracting figures, tables, captions and section titles from scholarly PDFs. Its implementation includes layout analysis, caption detection, candidate figure regions and a matching score.
PDFigCapX, described in the Bioinformatics literature, uses document layout and figure-content analysis to extract figure–caption pairs from biomedical publications. Both examples illustrate practical approaches rather than universal solutions.
The SCI-3000 dataset offers annotated figure, table and caption regions in thousands of open-access scientific publications. Such datasets make evaluation possible, but generalisation to children’s worksheets, handwritten notes and unusual page designs requires separate testing.
15. What a good evaluation must measure
Check figure-region accuracy, caption-region accuracy, text recognition and pairing accuracy separately. A model might find both the graphic and caption perfectly yet link them incorrectly; one aggregate detection score can conceal the error.
Include two-column articles, nearby figures, long captions, panels, legends, cross-page references and tables. Hold out documents from development data. A model tested on nearly identical pages can appear stronger than it is in a new publishing layout.
Inspect errors by consequence. A mislabeled axis or wrong caption attached to a medical diagram can be more serious than slightly imprecise bounding boxes around a decorative photograph. Task-based verification matters.
| Document condition | Possible error | Useful check |
|---|---|---|
| Two nearby figures | Captions swapped | Column and identifier association |
| Four-panel diagram | One panel extracted alone | Preserve parent figure and labels |
| Long caption wrapping lines | Caption truncated | Identify complete text block |
| Table title above grid | Title classified as a figure caption | Check object type and convention |
| Cross-page figure reference | Wrong nearby graphic | Review figure number and page context |
| Caption describes similar plots | Semantic overmatching | Prefer source identifiers and layout |
16. What readers should see in the result
A useful interface displays the proposed diagram and its actual extracted caption together, with a link or highlight pointing back to their original page. If several pairings are plausible, it should show the candidates and preserve their uncertainty.
Exports should include page index, visible figure label, caption text, visual region coordinates and the source document identifier. Any human correction should update the relationship without overwriting the original evidence.
An AI-generated description can be added as a separate accessibility aid if required, but it should not silently replace the author’s caption. Authored text and inferred image descriptions have different provenance and must stay distinguishable.
17. A paper experiment without any special software
Draw two different diagrams on a sheet: a growing plant and a weather map. Put the captions beneath them, then move the captions so the closest text no longer belongs to the nearest drawing.
Ask a friend to match each caption by numbering, content and layout. What happens if both captions say ‘Figure 1’? What additional evidence would resolve the ambiguity? This makes the limits of pure proximity obvious.
Add a four-panel figure with a single caption explaining panels (a) to (d). The exercise reveals why a complete figure may consist of several visual regions and why matching cannot be reduced to simply choosing one image box for one line of text.
18. Common misconceptions
First, detecting a figure is not the same as matching its caption. Second, OCR accuracy is not evidence of correct visual association. Third, the nearest text block is not always the caption. Fourth, one caption can apply to several panels.
Fifth, a body-text reference to Figure 2 is not necessarily the caption of Figure 2. Sixth, a figure can contain a legend and annotations that must stay with it. Seventh, a model-generated caption is different from the one the author wrote.
These distinctions help people assess whether an AI explanation is grounded in an actual source figure rather than being merely plausible prose about a nearby picture.
19. The larger lesson: meaning lives in relationships
Figure–caption matching is an example of AI reasoning that connects separate pieces of evidence. The words are one observation, the image is another and the document layout provides constraints about their intended relationship.
Traditional parsing, computer vision and learned models can each help. A trustworthy system checks that the chosen relation matches visible page structure and preserves enough context to make human correction practical.
The enduring habit is to ask not only ‘Did the computer read the words?’ but also ‘Did it connect them to the right evidence?’ Correct relationships make a document useful. Confidently wrong relationships make even perfect OCR dangerous.
Frequently asked questions
How does AI match a figure to its caption?
It detects candidate image and caption regions, reads labels and scores spatial and structural relationships, sometimes with learned visual-text features.
Is OCR enough to match captions?
No. OCR recognises words; matching requires page layout and relationships between text and graphics.
What does PDFFigures 2.0 extract?
Figures, tables, captions and associated metadata including bounding boxes and figure names in supported scholarly PDFs.
What is PDFigCapX?
A research system for extracting figures and their associated captions from scientific and biomedical documents.
Can a caption appear above a diagram?
Yes. Position varies with publishing style and object type, so proximity rules alone are insufficient.
What if two figures share one caption?
A composite figure can contain several labelled panels covered by one parent caption.
Can the correct caption be on another page?
Sometimes; a parser may need broader document context and careful verification.
Is a generated image description the same as an original caption?
No. It is new inferred text, not the source author’s words.
How should a figure–caption extraction be evaluated?
Check region detection, caption text, pairing accuracy and full figure boundaries on independent documents.
What is the safest output format?
Matched figure and caption with page identifiers, source coordinates, original text and reviewable uncertainty.
Primary research and connected reading
- Allen Institute for AI: PDFFigures 2.0 — figure and caption extraction
- Bioinformatics: Figure and Caption Extraction From Biomedical Documents
- SCI-3000 dataset: figures, tables and captions
- How Artificial Intelligence Works | How AI Finds the Correct Reading Order on a Scanned Page
- How Artificial Intelligence Works | How AI Reconstructs Tables From Scanned Documents
- How Artificial Intelligence Works | How AI Finds Document Edges Without Cutting Off Text
- eduKateSG: How X Works Hub
Editorial note: The two-column school article, plant figures and matching scenarios are fictional teaching cases, not model benchmark results. A caption association is an inference unless checked against the original document. No system should silently invent absent source captions or replace author text with generated descriptions.