How AI Reconstructs Scanned Tables

How artificial intelligence works when reading a table in a scanned document starts with a familiar school problem. A worksheet contains four columns: Student, Experiment, Result and Unit. A camera captures the page as a flat picture. If software extracts all the words but mixes their positions, a result can end up attached to the wrong student. The words may be perfectly recognised, yet the table’s meaning is lost.

Did you know that a table is not just text arranged near other text? Its structure tells the reader which values belong to which row, which column headers define the values and which cells may span several rows or columns. Modern document AI can attempt to recover that invisible structure by detecting the table, predicting its rows, columns and cells, and assigning recognised text to the correct positions.

The useful pipeline is scan or PDF page → locate table regions → recognise rows, columns and cell boundaries → recover spanning and blank cells → attach text and header relationships → export a structured table → verify against the original page. Computer vision can help with layout; OCR can help read the text. Neither one alone proves that the final table is correct.

The quick route: five questions

Where is the table on the page?
How are the rows, columns and cells organised?
Which words and numbers belong to each cell?
How do headings, blanks and spanning cells affect meaning?
What must be verified before exporting or calculating?

1. Why a scanned table is harder than it looks

Look at a timetable printed on paper. Human readers know that a lesson time in the left column belongs to the subject written beside it. They also understand that a heading spanning several columns labels the entire group underneath it. This relationship depends on spatial structure, not just the sequence of letters.

A scan converts the table into an image of pixels. Unless the source file already contains useful digital semantics, the computer must infer which lines, spaces, words and boxes represent table structure. A text recogniser might see ’10:30′, ‘Science’ and ‘Room 2’ but not know how those strings connect.

This distinction matters for school worksheets, financial statements, scientific results, schedules and administrative forms. A wrong cell assignment may lead to incorrect calculations even when the string inside each cell was recognised accurately.

2. Detection, structure recognition and content extraction are different tasks

Table detection first identifies where a table appears on a page. Table structure recognition estimates the rows, columns, cells, spanning cells and header regions inside it. Content extraction then connects the recognised characters or text tokens to the structure.

Imagine a page with a paragraph above a results table and a caption below. A detector may draw a box around the whole table. That is useful, but it does not say which number belongs to which column. Structure recognition goes a step further by reconstructing the layout relationships that make the box meaningful.

Microsoft Research’s PubTables-1M work discusses table detection, structure recognition and functional analysis as related yet distinct tasks. Keeping these components separate helps diagnose failures and prevents a polished-looking text export from hiding a wrong underlying arrangement.

3. Why ordinary OCR alone is insufficient

Optical character recognition estimates letters, digits and symbols from visual marks. A good OCR engine may read every temperature value correctly but return them in a stream separated by spaces or lines. If the table contains several columns, visual reading order alone may not preserve cell membership.

Suppose a table shows Experiment A at 20 degrees and Experiment B at 40 degrees. If the software associates 40 with A and 20 with B, both numerals may be recognised perfectly while the scientific conclusion becomes wrong.

A stronger workflow keeps text positions as bounding boxes, then assigns them to predicted cells. It should preserve the original page coordinates so a reviewer can click on a table value and inspect where it came from in the scan.

4. The first challenge is finding the table

Tables often have visible borders, but not always. Some use only whitespace and aligned text. Others contain faint horizontal rules, coloured header bands or cells with very different widths. A layout model can look for repeated alignment, line structures, text groups and page context.

A scanned page may be skewed, blurry, curved near a book spine or partly covered by a finger. These problems can confuse a detector before any table structure is considered. Image preparation such as deskewing or local contrast adjustment may improve the evidence, but aggressive enhancement can alter fine characters or borders.

The output of detection is a candidate table region. A model should support multiple tables on one page, nested or adjacent regions, and notes that are close to but not actually part of the grid. The next stage must determine what structure exists inside the region.

5. Recovering rows and columns without guessing

An ordinary grid can be described as horizontal bands for rows and vertical bands for columns. When visible borders are present, image processing may locate them as lines and intersections. When borders are missing, aligned text baselines, spacing and learned document-layout patterns can help infer the boundaries.

A model may propose row and column boxes, which intersect to form candidate cells. Another approach predicts cells directly or uses a transformer-based structure detector. The important output is a coherent table arrangement that permits each text token to be assigned to the correct cell.

Rows and columns must remain consistently ordered. If the software inserts a false narrow column because of a vertical line in one cell, the entire table can shift. A good system validates its own structural predictions and allows a reviewer to correct them.

6. The special problem of spanning cells

A school table might have a heading labelled ‘Term 1’ above two subcolumns, ‘Test’ and ‘Practical’. The heading occupies more than one ordinary column. This is a column-spanning cell, often represented in HTML with a colspan relationship.

Another table may place a single class name beside several experiment rows, so that one visible cell spans multiple rows. An exporter that duplicates or moves the class name carelessly can change the intended grouping. The structural representation needs explicit row-span and column-span relationships.

Spanning cells are difficult when their borders are faint or absent. A system should not simply assume every text item occupies exactly one grid position. Understanding the hierarchy is as important as locating physical lines.

7. Blank cells are real information

Imagine a table with five test results, and one student’s result is deliberately blank because the experiment was not conducted. If extraction software removes the empty cell, later values can slide one column to the left. A missing measurement may become falsely attached to a different property.

Blank, merged and intentionally omitted cells must therefore be represented consistently. A blank value is not always zero. It may mean not measured, not applicable, missing or simply unrecorded; the intended meaning comes from the source and its conventions.

PubTables-1M provides annotations that include cell locations and blank cells. Its research also discusses avoiding inconsistent structural labels. These details matter because an apparently minor annotation decision can strongly affect training and evaluation of table models.

8. A complete fictional classroom-results example

Picture a fictional science worksheet table with columns labelled ‘Trial’, ‘Water added’, ‘Plant height’ and ‘Notes’. Three rows report Trial A with 50 millilitres and 12 centimetres, Trial B with 75 millilitres and 15 centimetres, and Trial C with a blank height because the plant was accidentally damaged.

An AI document system first finds the table, then estimates its columns and rows. OCR reads the words and numbers, and a cell-assignment stage connects ’75’ to Trial B under Water added, rather than to the Plant height column. A reviewer checks the original image, especially the blank cell for Trial C.

Now imagine a faded vertical line between water and height. If the extractor merges those two columns, the text may be retained but the scientific variables become indistinguishable. That is a structural error, not a failure to recognise the characters ’75’ and ’15’.

All data and experiment details in this example are invented. They illustrate how meaning depends on cell relationships, not the apparent fluency of a final explanation.

9. What table headers actually contribute

Headers give interpretation to numbers. The value 12 means very little until it is linked with ‘Plant height’ and ‘centimetres’. A table may have column headers, row headers and multi-level headings that apply to several groups. Functional analysis estimates what roles different regions play.

A system that exports a grid of bare strings without header relationships can make the data searchable but difficult to interpret accurately. An accessible HTML or spreadsheet representation should preserve the relevant associations, not merely reproduce where text happened to sit.

Some complex tables use introductory notes, units in headings or footnote markers. These details may change how a value should be understood. For technical or scientific documents, treating the notes as decorative text can create errors even when every cell was placed correctly.

Table 1. Four steps of table understanding
Stage	Question it answers	What can still go wrong
Table detection	Where is the table?	Incorrect table region
Structure recognition	Where are rows, columns and cells?	Merged or shifted cells
OCR or text extraction	What text appears within cells?	Wrong digits or omitted symbols
Functional interpretation	Which headings and units define values?	Misattributed measurements

10. How machine learning is trained for table structure

Table recognition models can be trained using document pages annotated with table regions, rows, columns, header areas and spanning cells. The model learns visual patterns and layout relationships that predict these structures for new documents.

Microsoft’s Table Transformer research and released tooling show how object-detection-style neural models can be adapted to table detection and structure recognition. Such models make predictions about structural boxes and categories; additional processing is needed to build a coherent final table and attach its text.

The training data must reflect the documents the model will encounter. Scientific articles, school worksheets, receipts and handwritten tables have different conventions. A high score on one collection does not imply the model will work equally well with every skewed photograph or handwritten laboratory log.

11. Why oversegmentation is a serious error

Oversegmentation occurs when software divides a real structure into too many pieces. It might treat one merged header as two separate cells or split a cell containing wrapped text into two rows. Undersegmentation is the opposite: it merges distinct cells into one.

Both mistakes change the table’s logic. If a long label wraps onto two printed lines, a naïve detector can mistake the second line for the start of a new table row. Once that row is inserted, values below it may be displaced throughout the export.

The PubTables-1M work includes procedures for canonicalising annotations and addressing inconsistent oversegmentation. That research illustrates a general principle: a model can learn poor structural conventions if its labelled examples disagree about what a cell is.

12. How to export a recovered table responsibly

A table may be exported as HTML, CSV, spreadsheet cells or a structured document representation. CSV can carry simple row-and-column values but does not natively express complex merged cells or header hierarchy. HTML can use table headers, row spans and column spans when authored correctly.

The export format should match the task. If the goal is to calculate averages, a carefully checked flat data table may be appropriate. If the goal is accessible publication of a hierarchical table, a richer structural representation may be necessary.

Never convert a blank measurement into zero by default. Preserve units, signs, decimals, negative values and any explanatory notes. When the extractor is uncertain, record that uncertainty and give reviewers a direct path back to the scanned source.

13. The difference between table layout and reading order

Reading-order AI asks how content blocks should be presented in a meaningful sequence. Table-structure AI asks how entries relate across two dimensions. A row-major text reading of a table can sometimes be convenient, but it does not by itself preserve all the information required to compare columns or interpret spanning headers.

Consider a table with grouped columns for Mathematics and Science. Reading every cell left-to-right may lose the fact that ‘Average’ sits under Mathematics while another ‘Average’ sits under Science. The system needs both the heading hierarchy and the cell coordinates.

These tasks can work together in a document-processing pipeline, yet they should not be collapsed into one general claim that the AI has read the page. A correctly ordered paragraph and a correctly structured table satisfy different technical tests.

14. How to evaluate table recognition

A useful evaluation set contains scanned and digital documents with trustworthy reference structures. Include bordered and borderless tables, multi-level headers, blank cells, spanning regions, small text, skewed pages and irregular row heights. Keep near-identical pages out of both training and test splits.

Evaluate table detection separately from structure: did the system find the correct region, did it correctly form rows and columns, and did it associate text with the right cells? Examine functional accuracy too: does the recovered table allow a user to answer the same questions as the original?

Metrics may compare predicted structural boxes or grid topology, but a strong numeric score does not prove that a particular critical value was extracted correctly. Inspect consequential cells and header relationships individually when the table feeds educational grading, financial or scientific decisions.

Table 2. Table-format errors and their consequences
Source feature	Possible extraction error	Why it matters
Blank cell	Removed or converted to zero	Missing data changes meaning
Merged header	Split into independent labels	Category hierarchy lost
Wrapped text	Extra row created	Later values shift
Faded vertical rule	Columns merged	Variables become confused
Small decimal point	Wrong numerical value	Incorrect calculations
Footnote marker	Dropped from value or heading	Missing interpretation condition

15. Common failure cases in real scanned pages

A folded paper may curve grid lines. A faded photocopy may lose a border entirely. Handwritten additions can overlap neighbouring cells. A teacher might add an arrow linking two rows, or use colour shading to mark a subcategory that a monochrome scan does not preserve.

Other tables contain diagonal split headers, nested subtables or footnotes that appear inside the grid. One generic recogniser may not model all these structures. The right response is to identify the unsupported case and offer review rather than silently flatten the layout into misleading data.

Photograph quality affects the downstream task. If the page is blurry, first ask for a better scan when possible. No table-recognition model can faithfully read a digit that is not present in the recorded evidence.

16. A practical exercise with paper and a ruler

Draw a four-column table on paper with three rows of fictional data. Put a title above it and merge the top two cells to make one group heading. Leave one interior cell blank. Ask someone to write out the values as a plain comma-separated list.

Next ask that person to reconstruct the original grid from the list alone. Which information has been lost? They may struggle with the merged heading, the blank cell or the units. Then label every cell with a row and column position and see how much easier the reconstruction becomes.

For an advanced exercise, photograph the paper twice, once clearly and once with a mild tilt. Compare how a table-recognition tool handles them, keeping the original images and manually checking its cell assignments. Do not enter personal student scores or other private information simply to test the technique.

17. Privacy, traceability and human correction

Scanned tables often contain names, marks, payment amounts or other information about people. An AI extraction service may process sensitive material even if it outputs only a few numerical rows. Choose appropriate authorised tools, avoid unnecessary uploads, and apply proportionate access and retention controls.

Provenance matters too. A useful system stores the source page, page identifier, cell positions, model version and a record of any human corrections. If a later report contains a disputed number, reviewers should be able to trace it to the exact cell in the original scan.

Human review should correct the structure, not just fix individual words. If the third column is shifted or a header spans the wrong region, repairing one recognised number may leave the entire table semantically wrong.

18. The deeper lesson: intelligence requires preserving relationships

Table extraction teaches an important lesson about artificial intelligence. Recognising symbols is not the same as understanding their organisation. A row, column, header and blank cell can carry meaning without adding a single visible word. The system needs to infer those relationships before presenting an accurate structured result.

The technology is useful because it can convert unwieldy scanned material into searchable, computable information. But the risk of an elegant-looking wrong table is real. Errors in one cell or header can contaminate calculations and downstream AI summaries.

The strongest workflow therefore distinguishes observation, structural inference, text recognition and verified interpretation. A good tool makes those stages inspectable, preserves the original and knows when a page needs human correction instead of manufacturing certainty.

Frequently asked questions

How does AI turn a scanned table into a spreadsheet?

It identifies the table, predicts rows and columns, assigns recognised text to cells and exports a structured representation for checking.

Is table extraction the same as OCR?

No. OCR recognises text; table extraction must also recover the relationships between cells, rows and headers.

What is table structure recognition?

The task of estimating the layout of rows, columns, cells and spanning or header structures inside a table.

Can AI read tables without visible grid lines?

Sometimes, using text alignment, spacing and learned layout context, but borderless tables can be ambiguous.

Why do merged cells matter?

A merged cell may label several rows or columns; splitting it incorrectly changes which values fall under the heading.

What is a blank cell supposed to mean?

Its meaning depends on the source: missing, not applicable or not recorded. It should not automatically become zero.

What is PubTables-1M?

A research dataset introduced for table detection, structure recognition and functional analysis, with detailed annotations.

What is Table Transformer?

A family of transformer-based models for document table detection and structure-related tasks.

Does perfect OCR guarantee a correct table?

No. All recognised strings can be correct while cell associations and headers are wrong.

How should table extraction be checked?

Compare critical values, blank cells, headers, spans and row–column membership against the original document.

Primary research and connected reading

Editorial note: The school experiment table is invented for teaching and contains no real student results. The cited research establishes approaches to detecting and recognising tables, not flawless extraction of every page or guarantee of correct values. Verify the original whenever an extracted table informs a consequential decision.

How Artificial Intelligence Works | How AI Reconstructs Tables From Scanned Documents