How artificial intelligence works when repairing words split across lines becomes clear when a textbook page is scanned into a PDF. At the end of a narrow printed line, the word ‘evaporation’ might appear as ‘evapo-‘ and continue with ‘ration’ on the next line. A text extractor can accidentally preserve the break and return ‘evapo-ration’, or turn it into two separate words. The printed layout was readable to a person, but the digital text now contains a different spelling.
Did you know that not every hyphen at the end of a line should disappear? A printed line might end with ‘well-‘ and continue with ‘being’, where the intended compound is ‘well-being’. Other cases use a discretionary hyphen only because the line was too short. A quality document processor must distinguish layout-driven word division from meaningful hyphenation while preserving punctuation and mathematical symbols.
The practical pipeline is PDF page or scan → recover lines, words and coordinates → detect line-final hyphens and continuation candidates → classify discretionary split versus genuine compound → reconstruct the correct reading sequence → check against visual source text → retain an audit of changes. Classical rules, dictionaries and language models can all assist, but a confident-looking reconstructed word is not proof that its original letters were correctly recognised.
Five questions for reliable word repair
- Was the dash caused by a printed line break?
- Do the fragments belong to one logical paragraph?
- Would the whole word normally contain a hyphen?
- Could the dash be mathematical or part of a name?
- Can the reconstruction be checked in the original page?
1. Why printed lines divide words
Printed books have finite column widths. When a word does not fit, typesetting software may divide it across two lines at a permissible location. A visible hyphen signals the division at the line ending.
For example, the ordinary word ‘education’ might be printed as ‘edu-‘ at one line’s end followed by ‘cation’ at the next line’s start. Reading the book, a person usually reassembles the word automatically because the hyphen belongs to layout, not spelling.
A PDF or OCR exporter may preserve the artificial line break, remove the wrong character or insert a space. The resulting digital word can be corrupted even when the scanner has captured every visible printed letter.
2. Text extraction and reading order come first
One extracted PDF line may have coordinates indicating its place in a paragraph or column. Before joining words, the system needs to decide which line follows which. That is especially important in a two-column article or a page with tables and margin notes.
If the next line belongs to an unrelated column, joining the end of one line to its beginning creates a nonsense word. An apparent end-of-line hyphen is therefore only a candidate for repair after the document’s reading order has been established.
Structured extraction should retain page, column, paragraph and line boundaries. Flattening the PDF into one long string too early can remove the contextual clues needed to decide where a printed word actually continued.
3. Discretionary versus meaningful hyphens
A discretionary hyphen marks a permissible break inserted to fit the line; it may disappear when the word is displayed on one line. A hard hyphen is a meaningful spelling character that should normally remain in a compound or name.
The pdf_oxide hyphenation-aware reconstruction implementation documents this distinction using examples such as ‘Govern-‘ plus ‘ment’, which should be joined without the hyphen, and ‘content-‘ plus ‘coding’, which may need to preserve a compound hyphen.
The system must not treat every ASCII hyphen the same way. The visible character, original PDF encoding, position and linguistic structure determine whether removal is justified.
4. Soft hyphens and Unicode add another complication
Digital text can contain a soft-hyphen character that marks a possible word break. Depending on rendering and extraction, the character may be visible, invisible or represented differently from an ordinary hard hyphen.
Apache Tika’s public issue tracker documents cases where soft hyphens and wrapped PDF lines were not extracted as expected. Such bug reports show how real text pipelines can introduce errors even before a language model begins processing.
A robust parser records the original character sequence where possible and distinguishes Unicode soft-hyphen metadata from visible punctuation. It should not casually strip every hyphen character because some belong to correct compound words.
5. A basic geometry-based repair rule
A simple rule can find a hyphen at the last visible position of a text line, then inspect the first text fragment on the next line in the same paragraph. If the two fragments plausibly form one word, the system proposes a join.
That is more defensible than removing hyphens inside a sentence because the visual line boundary provides evidence that the mark arose from layout. Yet the next line can begin a new paragraph or a different column, so proximity alone is insufficient.
The algorithm may also consult a dictionary or morphological model. Such support improves the proposal but does not guarantee correctness for technical vocabulary, invented terms or proper names.
6. How dictionaries help—and where they fail
Suppose the fragments are ‘photo-‘ and ‘synthesis’. Joining them gives the recognised term ‘photosynthesis’. A lexicon can recognise that as a plausible whole word and support removing the line-end hyphen.
Now consider ‘cost-‘ followed by ‘effective’. Both ‘costeffective’ and ‘cost-effective’ have different linguistic status. A dictionary or language model should preserve the standard compound rather than remove its meaningful punctuation.
Technical writing also includes unfamiliar names, chemical terms and identifiers. Their absence from a common dictionary does not prove the printed hyphen must be retained. The system needs source-aware evidence rather than a simplistic valid-word test.
7. A complete fictional science-reading example
Imagine a worksheet that says ‘The process of evapo-‘ on one line and ‘ration can be faster in warm conditions’ on the next. A naive OCR export may produce ‘evapo- ration’, disrupting a vocabulary search.
A dehyphenation step identifies a line-final hyphen, confirms the continuation belongs to the same paragraph and proposes ‘evaporation’. A reviewer can check the source page to see that the combined word matches the intended scientific term.
Now add a heading that ends with ‘long-‘ followed by ‘term effects’ on the next line. The intended phrase is ‘long-term effects’. Deleting this hyphen would change the spelling. Both examples are invented learning cases and not benchmark outputs.
8. When a minus sign is not a hyphen
Mathematics and science documents contain negative numbers, subtraction operators, ranges and units. A minus sign at a line break can look like a typographic hyphen, especially after OCR replaces distinct Unicode characters with one ASCII dash.
If the first line contains a formula ending in a subtraction operator and the next line continues with a term, joining them as one word would corrupt the mathematics. The system needs to classify the surrounding region as a mathematical expression instead of ordinary paragraph text.
Preserve formulas and their layout. Dehyphenation should be applied only to appropriate text regions, with mathematical notation subject to a separately validated parsing process.
9. Why paragraph boundaries matter
A line break caused by page layout is not necessarily a paragraph break. The next text block may continue the same sentence, or it may begin a new heading, table cell or footnote.
Apache’s historical issue TIKA-1671 documents how hard it can be to recover intended text flow when PDF extraction has inserted line breaks without preserving whether they came from layout or author intent.
A good parser uses geometric indentation, line spacing, typography and document structure. Reconstructing a paragraph before dehyphenating words prevents many accidental joins across unrelated elements.
10. Repairing across page boundaries
A word can occasionally be split at the very end of a printed page or across column boundaries, depending on source formatting. A parser that processes each page in isolation may overlook such continuations.
However, arbitrary joining across pages is dangerous because the next page might begin a header or new section. The algorithm should track paragraph continuity, section markers and the original page reference.
For archival records, preserving the page-break position is still valuable even when the reconstructed reading text joins the word. The exported text and visual source coordinates should remain linked.
| Printed fragments | Possible continuous text | Reason |
|---|---|---|
| evapo- / ration | evaporation | Discretionary word split |
| photo- / synthesis | photosynthesis | Term broken for layout |
| well- / being | well-being | Meaningful compound hyphen |
| cost- / effective | cost-effective | Fixed compound |
| formula – / 3 | Keep mathematical operator | Not ordinary word hyphenation |
11. How learned AI can assist
A language model can score candidate reconstructions based on their surrounding sentence. An embedding model can assess whether the joined phrase makes contextual sense, and OCR confidence can provide evidence about uncertain letters.
But linguistic plausibility is not enough. A model may choose a common word even when the printed page contains an uncommon technical term. It may also silently correct the author’s spelling, turning faithful extraction into unapproved editing.
Use learned predictions to prioritise review, not to replace the source. The objective is to reconstruct what the page actually says, not to rewrite it into the most probable sentence under training data.
12. A one-character error can alter a conclusion
A hyphen or a minus sign can change how a calculation or assertion is interpreted. Even in prose, ‘re-cover’ and ‘recover’ can have different meanings in context, and names may use fixed hyphenation.
Academic documents may contain chemical nomenclature, author names or standards identifiers whose punctuation is essential. Removing every line-end dash can damage citations and searches.
An auditable tool should record each proposed join with the source fragments and reason. High-impact changes deserve a check against the original page image or a trusted text layer.
13. Multilingual documents need different rules
Languages differ in word formation, compound conventions and line-breaking practices. A dehyphenation rule tuned for ordinary English prose may not work for German compounds, mixed-language technical material or scripts without the same spacing conventions.
Some words legitimately contain more than one hyphen. Others use different punctuation or typographic dashes. An OCR engine might also misidentify a script character as a hyphen-like stroke.
A multilingual system should preserve source language labels and avoid applying one broad English lexicon to every passage. When language detection is uncertain, conservative handling can prevent irreversible text corruption.
14. What a good dehyphenation audit looks like
An extraction report can store the original fragments, line coordinates, proposed reconstructed word and whether the original hyphen was kept or removed. A user can then inspect the visual source for any disputed result.
Metadata might record that ‘evapo-‘ and ‘ration’ became ‘evaporation’ because both fragments occupied consecutive lines in one paragraph. An alternative compound example would record that the hyphen was intentionally retained.
The audit should also allow a no-change result. A parser that joins only well-supported candidates can be more reliable than one that always tries to produce seamless text.
15. Evaluate precision and recall separately
Build a test corpus with manually verified line-break and hyphen cases. Include normal words, fixed compounds, mathematics, names, footnotes, column boundaries and scanned pages with OCR errors.
Precision reflects how often proposed joins are correct; recall reflects how many legitimate joins were found. A system can improve recall while introducing numerous false word merges, so both outcomes matter.
Evaluate downstream search and reading as well. A wrong join can hide a keyword from retrieval or change a technical term, while a missed join may merely leave a visible break. The consequences depend on the actual document task.
| Error | Consequence | Safeguard |
|---|---|---|
| Every hyphen removed | Compounds corrupted | Classify hard versus soft |
| Two columns joined | Unrelated words merged | Recover reading order |
| Math minus treated as dash | Equation changed | Protect formula regions |
| Unfamiliar name normalised | Source term altered | Retain source spelling |
| Line OCR already wrong | Wrong joined word | Check image coordinates |
| Changes not logged | Hard to audit | Keep a reconstruction trail |
16. A useful classroom paper experiment
Write ‘photo-‘ at the end of one line and ‘synthesis’ at the start of the next. On another pair of lines, write ‘well-‘ and ‘being’. Ask a classmate which hyphen should disappear in the continuous reading text.
Now add a mathematical line that ends after a minus operator and a next line with a number. Does the algorithm still have grounds to treat the dash as an English word break? It does not.
The experiment illustrates the need to identify text region, layout and language convention before applying any automatic join. A correct result follows the source, not only how polished the output looks.
17. The difference between extraction and editing
An editor may deliberately improve grammar or modernise spelling. A text extractor should normally preserve the author’s wording while undoing only layout artifacts that obstruct its digital representation.
Conflating those tasks can be especially harmful in quoted documents, research papers and historical material. A language model’s improvement can erase evidence of an original typo or unusual term.
Keep a faithful transcript and optional edited version as separate outputs. The user should be able to return to the source and see exactly which line breaks and hyphens were transformed.
18. Common misconceptions
First, all line-end hyphens do not necessarily disappear. Second, removing a soft hyphen is different from deleting a meaningful hard hyphen. Third, a valid dictionary word is not always proof that the join is correct.
Fourth, OCR recognition and reading-order reconstruction must be checked before word joining. Fifth, mathematical minus signs need separate treatment. Sixth, an AI-generated correction is not authoritative evidence of the original typography.
Good processing separates the observed source, proposed reconstruction and verified result. The visual line break is a clue, not a licence to overwrite language.
19. The bigger lesson: layout and meaning must be kept separate
Line-end word repair illustrates why AI reading involves more than character recognition. A printed page contains visual layout, while a digital reader needs logical words and paragraphs. The software must translate between them without changing the underlying statement.
Classical rules, language resources and learned models can all help, but each one has assumptions and failure modes. Verification against source coordinates makes the final text more trustworthy.
The enduring lesson is to preserve meaning while normalising presentation. A correct text extractor removes layout artifacts carefully and refuses to invent letters or punctuation the page did not support.
20. A step-by-step source-aware repair example
Imagine a scanned two-column science article where the left column ends one line with ‘condensa-‘ and begins the next with ‘tion’. The extractor identifies both fragments inside the same paragraph, confirms that the next line is adjacent within the left column and proposes ‘condensation’. The source crop lets a reviewer verify the letters and the original printed hyphen.
Now change the line ending to ‘high-‘ followed by ‘speed camera’. The same geometry indicates a line continuation, but the intended text is ‘high-speed camera’. A rule that simply deletes the dash would produce a nonstandard compound. Context and a source-aware dictionary can help distinguish the candidates.
The important point is that dehyphenation has at least three choices: join without a hyphen, join while retaining it, or do not join. A trustworthy tool needs all three outcomes plus uncertainty.
21. The proof pack for a reconstructed word
A robust export can attach the original page index, two line coordinates, source fragments and proposed reconstructed token to each change. It can record why the join occurred: a line-final hyphen, compatible paragraph geometry and corroborating language evidence. A reviewer then knows exactly what was inferred.
For example, a record might show left fragment ‘evapo-‘, right fragment ‘ration’, output ‘evaporation’ and the original line boxes. If later text search finds the word incorrectly, the source-level provenance makes correction possible without reprocessing the entire book blindly.
Do not replace the original OCR stream without retaining the mapping. The transformation should improve searchability and reading while preserving a route back to the printed typography.
22. How to prevent a cascading OCR mistake
Consider a small scanned letter whose lower stroke is hidden by blur. OCR reads ‘m’ where the printed page may say ‘rn’. Dehyphenation can then combine two incorrect fragments into one fluent-looking word. A language model may make the result even more plausible, but the source ambiguity remains unresolved.
The system should therefore check recognition confidence or source image quality before aggressively joining uncertain words. For high-impact names, units, formulas and quotations, a reviewer should compare the output with the original page rather than rely on dictionary plausibility.
Keep detection, word reconstruction and editorial correction separate. The stop-loss condition is when the available image does not support a unique reading. At that point, report uncertainty instead of replacing a damaged token with a confident guess.
Frequently asked questions
What is PDF dehyphenation?
Reconstructing words divided by layout line breaks while retaining meaningful hyphens.
Should all line-end hyphens be removed?
No. Fixed compounds and other significant punctuation must remain.
What is a discretionary hyphen?
A break marker inserted to fit a line that can disappear in continuous text.
What is a soft hyphen?
A Unicode character marking a possible hyphenation or line-break point.
Why do PDFs contain split words?
Printed line layout and extraction behaviour can separate one word into fragments.
How can AI choose the right reconstruction?
It can combine line geometry, dictionaries and language context, with source verification.
Why are mathematical documents difficult?
A minus sign can resemble a hyphen but carry a different meaning.
Can the same rule work for every language?
Not reliably; word-formation and punctuation conventions vary.
Why retain page coordinates?
They allow each reconstructed word to be checked against the exact source.
What should be evaluated?
Correct joins, preserved compounds, unchanged mathematics and downstream reading fidelity.
Primary documentation and connected reading
- pdf_oxide: Hyphenation-aware PDF Text Reconstruction
- Apache Tika: Soft-hyphen Extraction Issue TIKA-2650
- Apache Tika: Wrapped Lines Extraction Issue TIKA-1671
- Research: Correcting PDF-to-Text Conversion Errors
- How AI Finds the Correct Reading Order on a Scanned Page
- How AI Keeps Footnotes Separate From the Main Text in Scanned PDFs
- eduKateSG: How X Works Hub
Editorial note: All example paragraphs, mathematics expressions and OCR fragments are invented for education. A reconstructed word is an inference about how the printed text was meant to read. Do not treat linguistic plausibility as independent evidence of the author’s exact wording; keep source scans and an audit trail.