Suppose you know the next Voynich token belongs to a certain structural class.
Does it matter which exact token it is?
That sounds like a small technical question.
It is actually one of the cleanest ways to ask where information lives in Voynichese.
If exact token identity matters strongly, then individual Voynich forms may carry distinctive sequential information beyond broad grammatical or structural category.
If class membership does most of the work, then the manuscript may be operating more like a constrained state machine in which category matters more than lexical identity.
A 2026 reproducible open-analysis project reports a striking result from this exact comparison: word identity adds little beyond word class that a meaning-free class-Markov process cannot reproduce. In the project’s Part 68 audit, the class-based null is reported to reproduce about 79% of the measured signal.
The result is not peer-reviewed consensus. It comes from an independent, openly documented analysis that explicitly publishes its null models and failed hypotheses.
Its importance is methodological rather than spectacular.
A sequence can have real grammar-like class structure without each exact visible token contributing much additional evidence of meaning.
Quick Read
- The existing Syntax Before Semantics article owns the broad observation that Voynich token classes have stable sequential preferences.
- This new owner asks a narrower question: after class is known, how much additional predictive information does the exact token identity provide?
- The 2026 Voynich Project reports that exact identity adds surprisingly little beyond class under its test.
- A deliberately meaning-free class-Markov process reportedly reproduces about 79% of the observed identity-level signal.
- This weakens any argument that sequence predictability alone demonstrates lexical semantics.
- It does not prove Voynich tokens are meaningless.
- It does not prove the induced classes are the historical grammatical categories.
- It suggests that a large share of local order may live in class transitions rather than exact-token succession.
- That is compatible with language, codebooks, notation, constrained generation and other structured systems.
- The decisive next tests must ask what exact identity predicts that class alone cannot: image association, record-specific content, long-range recurrence, external crib behaviour or held-out semantic effects.
First: What Is a Word Class Here?
Do not hear “noun”, “verb” or “adjective” yet.
Those are semantic and grammatical names from known languages.
In the 2026 analysis, classes are induced from distributional context: tokens are grouped because they tend to occur in similar sequential environments.
The project reports that clustering the 236 most frequent Voynich tokens by context yields eight coherent classes with stable transition preferences.
That means the algorithm can recover categories without being told what any token means.
This is already important.
It tells us that the token stream is not merely a sequence of independent whole forms.
Some forms occupy similar jobs in the local order.
But “same job in a sequence” is not yet “same part of speech”.
Class Structure Can Exist Without Meaning
This is the central twist.
A meaningless machine can still have states.
Imagine a generator with three rules:
- after class A, choose class B most of the time;
- after class B, choose class C;
- after class C, return to A or terminate the line.
The output will have grammar-like order.
It may even be easy to predict.
Yet none of the tokens need refer to anything in the world.
This is why class transitions are evidence of rule-governed structure, not proof of recoverable semantics.
The Identity Question
Now sharpen the test.
Suppose two Voynich forms belong to the same class.
Does replacing one with the other substantially change what follows?
If yes, exact identity carries information beyond class.
If no, local sequence may care mostly about category.
This is analogous to known language in a deliberately simplified way.
Consider:
- the dog runs;
- the cat runs;
- the child runs.
At a grammatical level, noun-class membership predicts the legal sentence pattern.
Exact noun identity contributes semantics—dog versus cat versus child—but may contribute relatively little to the immediate syntactic transition.
That is perfectly compatible with meaningful language.
The difficulty in Voynich is that we cannot independently verify the semantic layer.
What “79% of the Signal” Should Mean to a Reader
The reported 79% figure should not be read as:
79% of Voynich has no meaning.
It is not a semantic percentage.
It is a model-control result.
The project asks how much of an observed identity-level sequential effect can be reproduced when a model knows only class transition structure and has no semantic content.
The reported answer is roughly four-fifths.
That is a warning about interpretation.
If a meaning-free class machine can reproduce most of a statistic, that statistic cannot carry the full evidentiary burden of meaning.
Why This Does Not Refute Language
Natural languages often have strong class structure.
Syntax frequently operates over categories rather than exact lexical items.
Articles precede nouns as a class.
Prepositions select certain phrase types.
Verb arguments can belong to broad grammatical categories.
So a result in which class explains most immediate sequence behaviour is not anti-language.
It simply tells us that immediate sequence order is not where lexical meaning necessarily announces itself most strongly.
Why This Does Not Prove Meaning Either
A state machine can create classes.
A codebook can create classes.
A templated notation can create classes.
A copy-and-modify generator can create classes.
A cipher can create classes if its output depends on state.
Therefore the existence of class transitions cannot independently discriminate language from several non-language or transformed-language mechanisms.
This is the same reason the 2026 project narrowed its own earlier grammar-like headline.
Grammar-like order survived.
Semantic inference did not automatically follow.
Where This Article Separates From Syntax Before Semantics
The earlier Syntax Before Semantics article asks:
What kinds of Voynich tokens prefer to follow other kinds?
This article asks:
Once you know the kind, how much does knowing the exact token add?
That is a different evidentiary job.
One establishes class-sequential structure.
The other measures how much information remains at the identity level after class is accounted for.
The Right Null Model Is Everything
A context-free random generator is an easy opponent.
It destroys the very class structure already known to exist.
If Voynich beats that null, we have learned only that Voynich contains class order.
The stronger control preserves the known class transitions while stripping away meaning.
That is what the class-Markov model is designed to do.
It says:
- preserve which class tends to follow which;
- choose exact tokens without semantic knowledge;
- measure how much of the target statistic survives.
This is a much harsher test.
The fact that it reproduces much of the signal is therefore informative.
A Bad Null Can Make Semantics Appear From Nowhere
Suppose we shuffle every Voynich token completely.
The class order disappears.
The real manuscript then looks deeply structured by comparison.
Correct.
But if the research question is whether exact words matter beyond class, complete shuffling is the wrong null.
The null must preserve the lower-level explanation already known.
This is a general lesson across the entire Voynich estate.
Null models should get progressively stronger as our knowledge improves.
See Tangential Voynich | The Null Model Is Not Neutral.
What Could the Remaining 21% Contain?
The 79% control does not erase the residual.
The remaining difference deserves investigation.
Possible sources include:
- real lexical identity;
- subclasses not captured by the eight induced groups;
- line position;
- page topic;
- Currier regime;
- scribal or production-state effects;
- token morphology;
- rare local dependencies;
- model misspecification.
The residual is therefore not automatically “the semantic 21%”.
It is simply what the current class null does not explain.
The Next Test: Identity Beyond Class and Layout
Class is not the only confound.
Voynich token identity also correlates with line position, page subject, Currier regime and local burstiness.
A stronger model should therefore ask whether exact identity adds predictive value after controlling simultaneously for:
- class;
- line position;
- paragraph position;
- Currier A/B;
- page or section;
- proposed hand;
- local recurrence history.
If identity still predicts something important after those controls, its evidentiary value becomes much more interesting.
What Should Exact Identity Predict If It Carries Lexical Meaning?
A meaningful lexical item should eventually do more than sit inside a class.
It should help identify a specific referent, topic, relation or repeated function.
Possible tests include:
- Does one token recur on pages with the same distinctive visual feature more than other members of its class?
- Does it recur across multiple mentions of the same internal object?
- Does it preserve a stable relation in labels and prose?
- Does it recur in a predictable record field?
- Does it align with a secure external crib?
- Does it predict held-out image or diagram structure?
These are identity-level jobs.
Class alone should not be able to perform all of them if lexical identity is real and recoverable.
Labels Are a Good Identity Test
Labels create a particularly useful setting.
If a label names an object, exact identity should matter strongly.
Two labels belonging to the same structural class should not become interchangeable if they name different objects.
The same 2026 project reports that repeated labels carry a weak page-level signal, but that this is confounded by ordinary within-page repetition and does not establish that the label names its own illustration.
That makes labels a promising but still unresolved bridge between class and identity.
Line-Records Make the Question Even More Interesting
The new Line-Record Problem proposes that individual lines may be relatively self-contained informational units.
If so, exact identity may matter mainly inside each record rather than through long discourse.
That creates a hierarchical possibility:
- class controls local syntax or field order;
- exact identity carries record-specific content;
- page or section controls topic.
This would make Voynich look structured at three scales without requiring conventional continuous prose.
It is an attractive model.
It still needs independent evidence that identity encodes record content rather than merely finer production state.
A Word-Level Code Could Produce the Same Result
Suppose each visible Voynich token is a code value standing for a whole plaintext word.
If related plaintext words receive related code families, visible token classes may preserve grammatical category.
The local sequence can therefore look class-governed while exact token identity remains semantically opaque without the external key.
This is one reason a lost-codebook model remains compatible with the class-vs-identity result.
Compatible does not mean proven.
The model must still explain token morphology, near-twins, page topic, line records and historical implementation.
A Meaning-Free Generator Could Also Produce It
A generator can first choose a class according to a transition matrix.
Then choose one token from that class.
The resulting text has stable class grammar.
Exact identity adds little.
No semantics are required.
That is why the class-Markov control is so important: it shows that this mechanism family can generate much of the target signature.
Any semantic theory must therefore win somewhere the meaningless class machine cannot.
Where Should Meaning Win?
Not necessarily in next-token prediction.
Meaning may reveal itself through:
- stable object reference;
- cross-page recurrence of specific concepts;
- image–text grounding;
- external historical anchors;
- quantities or numerals;
- repeated labels for demonstrably repeated entities;
- record-specific co-occurrence beyond class expectations.
This suggests a broader principle.
Do not demand that semantics reveal itself in the statistic most convenient to compute. Ask which evidence semantic identity should uniquely improve.
Class Induction Itself Needs Stress Testing
The eight classes are not historical labels recovered from the manuscript.
They are model outputs.
A robust result should survive reasonable changes in:
- number of classes;
- clustering method;
- minimum token frequency;
- transcription;
- space treatment;
- Currier stratification;
- line-edge removal.
If the 79% effect depends on exactly eight classes chosen after inspecting the target statistic, confidence should fall.
If similar conclusions survive several class ontologies, confidence rises.
The Number of Classes Is Not the Meaning of the Classes
Eight statistical classes do not imply eight parts of speech.
They do not imply eight cipher states.
They do not imply eight record fields.
The number is a useful compression of the observed transition space.
Historical interpretation comes later.
Currier A/B Could Change the Identity Result
Pool Currier A and B and one class may contain tokens whose local roles differ across regimes.
This can blur identity effects.
A stronger test asks:
- How much identity information remains within Currier A?
- How much remains within Currier B?
- Do the same classes exist in both?
- Does the 79% null reproduction remain similar after stratification?
If class dominates separately in both regimes, the result becomes more general.
If identity suddenly matters strongly within one regime, the pooled result may hide an important local system.
Hand and Production Drift Matter Too
The manuscript changes across production space.
If one hand or period prefers certain tokens within a class, exact identity may be carrying scribal information rather than semantic information.
That is still real signal.
It is a different kind of signal.
This is why the identity problem has to sit beside the Drift Problem and the Scribe Problem rather than pretending every remaining token-specific effect belongs to lexical meaning.
A Better Hierarchical Model
The most informative next model may be hierarchical rather than flat.
- Level 1: manuscript regime—Currier, page type, production state.
- Level 2: token class—local sequential role.
- Level 3: exact token identity—candidate lexical or code value.
- Level 4: morphology—prefix/core/suffix structure inside the token.
Then ask how much predictive information each level contributes after the levels above it are known.
This avoids forcing one scale to explain what another already accounts for.
Why This Matters for AI
A modern language model can discover token classes quickly.
It can then produce impressive predictions.
The danger is calling that semantic understanding.
If a small class-Markov model reproduces most of the same measurable signal, the sophisticated model has not yet demonstrated a semantic advantage.
AI-based Voynich claims should therefore report an ablation hierarchy:
- class-only model;
- class + morphology;
- class + exact identity;
- class + identity + page context;
- full multimodal model.
Only the incremental gains tell us which information level the model is actually using.
A Real Semantic Model Should Beat Class-Only on External Tasks
The most important future benchmark is not another in-corpus next-token score.
It is an external task.
- Predict which image class a token refers to.
- Predict which repeated visual entity receives the token.
- Predict a held-out label relationship.
- Predict a known zodiac association.
- Predict record-specific content across repeated structures.
If exact identity adds large predictive value there while class-only does not, the semantic case becomes much stronger.
What the Class-vs-Identity Result Does Not Prove
- It does not prove Voynich tokens are meaningless.
- It does not prove the manuscript is a Markov process.
- It does not prove the eight induced classes are historical grammatical classes.
- It does not prove exact token identity carries no information.
- It does not convert the residual unexplained signal into semantics automatically.
- It does not distinguish a word-level code from a meaning-free formal system by itself.
- It does not establish the manuscript’s language, cipher, purpose or author.
What It Can Give Us
- A clean decomposition of sequence information into class-level and identity-level components.
- A strong null model for grammar-like claims.
- A warning that local ordering can be deep without being semantic.
- A target for hierarchical modelling across Currier, hand, class and token identity.
- A sharper definition of where semantic evidence must eventually outperform meaningless structure.
Primary School: The Colour Rule and the Card Name
Imagine cards in three colours.
After red, you usually play blue.
After blue, you usually play green.
The exact picture on the card may not matter to the rule.
Colour is the class.
Picture is the identity.
A sequence can be predictable from colour even when the pictures carry completely different meanings.
Secondary School: Hold Class Constant
Give students several sequences where tokens are assigned to classes.
First predict the next class.
Then ask whether exact token identity improves the prediction after class is already known.
This teaches conditional information directly.
JC and Adult Readers: Hierarchical Information Gain
- Induce classes on training data.
- Freeze the class model.
- Estimate next-token or context prediction using class only.
- Add exact token identity.
- Measure incremental information gain on held-out data.
- Repeat within Currier and hand strata.
- Compare against a class-Markov null with the same class transition matrix.
- Move the same comparison to external image or record tasks.
Reader Checklist
- How were token classes induced?
- How many classes were chosen, and why?
- Was class induction separated from final evaluation?
- What exact statistic measures identity-level information?
- What does the class-only baseline achieve?
- What does the meaning-free class-Markov null achieve?
- Does identity add information after line position and Currier are controlled?
- Does the effect survive another transcription?
- Does exact identity predict external visual or record information?
- Is grammar-like class order being confused with semantic content?
Research Foundations
- The Voynich Project — reproducible open-science investigation (2026 independent public research release; not peer reviewed).
- Syntax Before Semantics.
- The Line-Record Problem.
- Tangential Voynich | The Null Model Is Not Neutral.
The Final Idea
Voynich may care deeply about what kind of token comes next while caring much less—at least for local sequence—about which exact token occupies the slot.
That does not make the book meaningless.
It tells us where meaning has not yet distinguished itself from structure.
The exact word earns semantic status only when it predicts something its class cannot.