One reality can be represented many ways because different media are good at preserving different things.
A sentence can explain a cause. A diagram can expose structure. A photograph can preserve visible detail. A map can preserve spatial relation. A graph can make a trend visible. Audio can preserve timing and tone. Video can preserve movement across time.
Multimodal representation works by assigning different parts of a communication problem to different representational modes, then coordinating those modes so the receiver can reconstruct one coherent model rather than several disconnected fragments.
This article sits beneath How Information Works, How Communication Works and the previous How Representational Invariance Works. The key question is not “which medium is best?” but which medium should carry which structure, and how will the receiver connect them?
1. Modes Have Different Affordances
Text is good at sequence, qualification and explicit argument. Images are good at appearance and spatial relation. Diagrams are good at selected structure. Audio is good at rhythm, prosody and temporal flow. Video is good at dynamic change. Tables are good at exact comparison.
These strengths are not absolute. They are affordances: features that make one mode more suitable for one task than another.
2. Redundancy Can Help or Waste Attention
If text and diagram communicate the same relation in complementary ways, the receiver gets another route into the concept.
If a slide simply reads aloud a dense paragraph while displaying the identical paragraph, the second channel can add little and may compete for attention.
Useful multimodality is coordinated, not decorative.
3. Complementarity Is Stronger Than Duplication
A map can show where; text can explain why. A graph can show a trend; a caption can explain the measurement and uncertainty. A photograph can show a component; an exploded diagram can show how it fits into the system.
The modes become powerful when each contributes information the others do not carry efficiently.
4. Cross-Modal Alignment Is the Hidden Interface
The text must refer to the same entity the image highlights. The graph must use the same units the paragraph discusses. The video timestamp must align with the transcript. The legend must match the map symbols.
When modes disagree, the receiver must choose which representation to trust.
This makes multimodal communication an interface problem. See How Semantic Alignment Works.
5. Captions Bind Image to Claim
An image without context can preserve visible evidence while leaving the receiver unsure what to notice.
A good caption identifies the object, time, place or relevant mechanism and makes clear what the image supports. It should not claim more than the image can show.
The Photography estate owns the wider visible-evidence problem through How Photography Works.
6. Diagrams Are Selective by Design
A diagram often removes texture, colour and physical realism to make relationships clearer.
That simplification is not a flaw if the omitted detail is irrelevant to the intended mechanism. It becomes a flaw when the omitted detail affects the decision.
7. Audio Carries Social and Temporal Information
Tone, stress, hesitation, rhythm and pacing can change meaning in ways a plain transcript may not preserve.
A transcript can improve search and accessibility while losing part of the performance. Multimodal archives often need both.
8. Data Visualisation Is a Translation Layer
A chart turns numerical relationships into spatial form. That can make pattern easier to see, but scale, axis choice and aggregation determine what becomes visually salient.
The chart is not the data. It is one representation of selected relationships inside the data.
9. Accessibility Requires Alternate Routes
A visual explanation can exclude a blind reader. Audio without transcript can exclude a deaf reader. Colour-only distinction can fail for some viewers. Motion can create barriers for others.
Multimodal design should therefore include alternate representations where the information matters: alt text, captions, transcripts, labels, keyboard access and textual summaries.
The goal is not identical sensory experience. It is access to the information and task.
10. Worked Example: Science Textbook
A page explaining the heart can combine an anatomical image, labelled diagram, explanatory text and flow arrows.
The image preserves appearance. The diagram selects chambers and vessels. The arrows preserve direction. The prose explains pressure and sequence.
No single representation carries the whole mechanism efficiently.
11. Worked Example: MRT Navigation
A passenger uses line colour, station names, platform signs, directional arrows, announcements and schematic maps together.
The system is multimodal because navigation must survive noisy stations, distance, language differences and changing attention. Each mode provides another route to the same operational state.
The MRT owner remains How MRT Works | It’s Mathematics.
12. Worked Example: Education
A learner studies a fraction through symbolic notation, area model, number line, verbal explanation and physical grouping.
If the learner can connect the representations, the concept becomes more robust. If each mode is memorised separately, multimodality creates more surfaces without deeper understanding.
13. Worked Example: News
A news story can combine text, photograph, map, timeline and chart.
Each mode can strengthen understanding, but each can also frame attention differently. The media system therefore has to keep provenance and context visible across modalities.
14. Failure Mode: Decorative Multimodality
An image can make a page look richer without adding information. An animation can attract attention away from the mechanism. A chart can create the appearance of evidence while displaying no meaningful comparison.
Every added mode creates cognitive and technical cost. It should earn its place by carrying useful structure.
15. Failure Mode: Cross-Modal Contradiction
If a diagram says one thing and the text says another, the receiver has to perform reconciliation.
In high-stakes contexts, that contradiction can create operational error. Version control and editorial checking should therefore cover all modalities, not only prose.
16. A Multimodal-Representation Checklist
- Define the receiver’s task.
- Assign each information structure to the mode that carries it best.
- Use complementary modes rather than decorative duplication.
- Align entities, units, labels and timing across modes.
- Provide captions, legends and provenance where needed.
- Add alternate access routes for important information.
- Test whether the receiver can integrate the modes into one model.
- Remove modes that increase load without increasing understanding.
- Version all representations together when the underlying fact changes.
17. Read the Mechanism Forward, Backward and Sideways
Forward: one reality → several representational modes → cross-modal alignment → receiver integration → usable mental model. Backward: start from a misunderstanding and identify which mode carried the wrong or missing invariant. Sideways: compare writer, designer, accessibility specialist, teacher and reader. Each sees a different failure mode in the same multimodal artefact.
18. The Civilisation Lesson
Civilisation does not store reality in one medium. It builds overlapping representational systems so knowledge can survive different receivers, tasks, places and technologies.
Multimodal representation works when each medium carries the part of reality it can preserve best, and the receiver can still put the parts back together.
Continue through How Representational Invariance Works, How Communication Works and the master How X Works hub. Next: annotation schemes — how complex reality becomes labels a dataset can use.